stable/9 @r241776 panic: REDZONE: Buffer underflow detected...

[Mateusz Guzik]

On Sat, Oct 20, 2012 at 07:10:19AM -0700, David Wolfskill wrote:
> This seems ... fairly weird to me.
> 
> Yesterday, I built & booted:
> 
> FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #274 241726M: Fri Oct 19 05:40:05 PDT 2012     root at g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386
> 
> and used the machine all day; nothing unusual (including various
> reboots (e.g. when I disembarked the train for the final leg of my
> commute home, so I powered the laptop off).
> 
> This morning, I built:
> 
> FreeBSD g1-227.catwhisker.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #275 241776M: Sat Oct 20 04:34:45 PDT 2012     root at g1-227.catwhisker.org:/usr/obj/usr/src/sys/CANARY  i386
> 
> and on first reboot, I got a panic.
> 
[..]
> 
> ...
> Starting devd.
> REDZONE: Buffer underflow detected.  1 byte corrupted before 0xced40080 (4294966796 bytes allocated).
> Allocation backtrace:
> #0 0xc0ceac8f at redzone_setup+0xcf
> #1 0xc0a5d5c9 at malloc+0x1d9
> ...[about 20 more such lines I didn't record]...
> 
> > bt
> Tracing pid 901 tid 100106 td 0xd2b99000
> kdb_enter(...)
> panic(...)
> free(...)
> devread(ce8c2d00,f7274c0c,0,c0b1e4f0,d279e380,...) at devread+0x1a6
> giant_read(...) at giant_read+0x87
> devfs_read(...) at devfs_read+0xc6
> dofileread(...) at dofileread+0x99
> sys_read(...) at sys_read+0x98
> syscall(f7274d08) at syscall+0x387
> 

This looks a lot like issue you reported a couple of months earlier,
even affected buffer address matches.

At least part of REDZONE metadata placed directly before the buffer is
corrupted. So the idea is to set a watchpoint at a place that is known
to contain wrong data (in this case allocation size) and wait for some
code to try to modify it.

I hacked up the following (really ugly, but should do the job):
http://people.freebsd.org/~mjg/patches/watchpoint-hack.diff

Note: this assumes that address of affected buffer is always the same.

Assuming I didn't mess anything up, instructions are simple:
Just try to reproduce the issue, at some point you should be dropped to
the debugger. If that happens when dumpdevice is configured, please get a
core. Otherwise just a backtrace ("bt" command).

Note 2: this code does no clear the watchpoint, so if it fails to catch
the offending case, it may catch completely legitimate code later.