Failed to flush worklist

[Eric Anderson]

On one of my servers, I'm unable to change a mount point from rw to ro 
(something I do all the time).  When I try it, I receive messages like:

Jul 19 08:57:18 snapshot1 kernel: softdep_waitidle: Failed to flush 
worklist for 0xc9f37400softdep_waitidle: Failed to flush worklist for 
0xc9f37400
Jul 19 08:57:18 snapshot1 kernel: softdep_waitidle: Failed to flush 
worklist for 0xc9f37400
Jul 19 08:57:53 snapshot1 kernel: softdep_waitidle: Failed to flush 
worklist for 0xc9f37400
Jul 19 08:58:28 snapshot1 kernel: softdep_waitidle: Failed to flush 
worklist for 0xc9f37400


# df /vol4
Filesystem      1K-blocks        Used     Avail Capacity  Mounted on
/dev/label/vol4 1891668564 1769762782 -29427702   102%    /vol4

# uname -a
FreeBSD snapshot1.centtech.com 6.1-STABLE FreeBSD 6.1-STABLE #2: Tue May 
16 08:34:43 CDT 2006 
root at snapshot1.centtech.com:/usr/obj/usr/src/sys/SNAPSHOT  i386

# mount
/dev/label/vol4 on /vol4 (ufs, local, noatime, soft-updates)

Now, here's how I think I broke it:

Filesystem was mounted rw, and all was well.
Perl program was running in the background, accessing data in a 
directory deep down inside the filesystem, maybe even its cwd was in 
there when it started.
I did massive rm's, hard links, and writing of new files (see rsync and 
the link-dest option), probably including the directory the perl program 
was in.
Later on, I tried to do:

mount -u -oro /vol4
which failed, with either permission denied or operation not permitted. 
  I cannot umount the filesystem, nor does the -f option help.

Here's a snippet from fstat - nothing is currently running on that 
filesystem that I can find, and this is the only thing that looks odd:

root     perl       99912 root /             2 drwxr-xr-x    1024  r
root     perl       99912   wd /             2 drwxr-xr-x    1024  r
root     perl       99912 text /        238402 -rwxr-xr-x   10088  r
root     perl       99912    0 -         -         bad    -
root     perl       99912    1 -         -         bad    -
root     perl       99912    2 -         -         bad    -
root     perl       99740 root /             2 drwxr-xr-x    1024  r
root     perl       99740   wd /             2 drwxr-xr-x    1024  r
root     perl       99740 text /        238402 -rwxr-xr-x   10088  r
root     perl       99740    0 -         -         bad    -
root     perl       99740    1 -         -         bad    -
root     perl       99740    2 -         -         bad    -

Any ideas??

Eric



-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
Anything that works is better than anything that doesn't.
------------------------------------------------------------------------