kern/155421: System can't dump corefile

[Andrew Boyer]

>Number:         155421
>Category:       kern
>Synopsis:       System can't dump corefile
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Mar 09 20:30:12 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Andrew Boyer
>Release:        8.2
>Organization:
Avere Systems
>Environment:
FreeBSD node1 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Release 8.2 (and 8.2-RC3 before it, at least) do not appear to be able to dump core.  This works reliably in previous releases.

node1 is an 8-core system with 72GB of RAM and a 250GB drive plugged into an onboard ICH10R SATA port.

node2 is an 8-core system with 64GB of RAM and a 250GB drive plugged into an onboard ESB2 SATA port.

8.2 was installed from a USB drive.  All space was allocated to the first slice and the default partition scheme was used.  swap is set to 4GB.

dumpdev="AUTO" is set in /etc/rc.conf, and /dev/dumpdev exists.

Initiating a core with 'sysctl debug.kdb.panic=1' results in one of several hangs:
 - no progress at all
 - very, very slow progress
 - a secondary panic on 'bufwrite: buffer is not busy'
 - other secondary panics

Examples:
=== node1 ===
# sysctl debug.kdb.panic=1
debug.kdb.panic: 0panic: kdb_sysctl_panic
cpuid = 4
KDB: stack backtrace:
...
Uptime: 20m50s
Physical memory: 73706 MB
Dumping 2780 MB:

=== node2 ===
# sysctl debug.kdb.panic=1
debug.kdb.panic: 0panic: kdb_sysctl_panic
cpuid = 4
KDB: stack backtrace:
...
Uptime: 5m50s
Physical memory: 65521 MB
Dumping 2437 MB:

Other things I have tried with the same results:
- Explicitly specifying the dump device e.g. 'dumpon -v /dev/ad0s1b'
- Increasing swap from 4GB to 32GB
- Changing the SATA controller mode to Compatible, Enhanced, and AHCI
- Booting from a SAS HDD connected to a SAS1068e

>How-To-Repeat:
1. Set dumpdev to 'AUTO' in /etc/rc.conf
2. Run '/etc/rc.d/dumpon restart', or just reboot
3. Run 'sysctl debug.kdb.panic=1'

>Fix:
The issue may be that the syncer is still doing filesystem updates in parallel with the attempt to dump core.  I added enough debugging output to follow the dumper's progress through the first few steps of the dump.  It doesn't seem right that other things are running in parallel with a panic/coredump.

>Release-Note:
>Audit-Trail:
>Unformatted: