ia64/149208: mksnap_ffs hang/deadlock
Karl
karl at denninger.net
Mon Aug 2 20:30:09 UTC 2010
>Number: 149208
>Category: ia64
>Synopsis: mksnap_ffs hang/deadlock
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-ia64
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Aug 02 20:30:08 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: Karl
>Release: FreeBSD 8.0-STABLE amd64
>Organization:
Cuda Systems LLC
>Environment:
System: FreeBSD tickerforum.org 8.0-STABLE FreeBSD 8.0-STABLE #3: Fri Dec 18 17:41:53 EST 2009 karl at tickerforum.org:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
Automated nightly backup hung during mksnap_ffs call from dump;
all I/O to that disk blocked in a "D" state. System was
otherwise functional. There were no errors in the system logs
or on the console and the raid adapter was fully function. Raw
I/O to the disk devices (e.g. reads of the "c" partition of the
disk that was wedged) remained fully functional during the event.
Disk adapter is a 3ware 9650; firmware FE9x 4.08.00.006, driver
3.70.05.001, BBU enabled. It has no faults recorded against it
or the drives attached to it.
Looks like a race condition/deadlock of some sort in the
mksnap_ffs code.
The only unique "feature" of this event was that the disk in
question was a DBMS storage RAID array and had a VERY large
(many thousands of files) directory on it containing the log
segments (typically ~1-2MB each, compressed, sequential log
files numbering many thousand.)
System was unable to be cleared; mksnap_ffs could not be killed
and ultimately the machine had to be forcibly rebooted without a
sync().
>How-To-Repeat:
Attempting to replicate at this time. Will update the PR if
I am able to do so. The subject machine had been up for more
than six months at the time of the fault, and it has not
reoccurred as of submission of this PR.
Migration to 8.1 on this cluster is taking place, but I am going
to put in a significant amount of time in attempting to
replicate the problem on 8.0-STABLE (same kernel and binaries)
on my test cluster to see if I can provoke the failure; I have
an isolated environment in which I can make a decent attempt at
recreating the event.
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-ia64
mailing list