fsck_ufs locked in snaplk
Dmitry Morozovsky
marck at rinet.ru
Mon Apr 24 18:05:00 UTC 2006
On Mon, 24 Apr 2006, Dmitry Morozovsky wrote:
DM> kKK> > one of my servers had to be rebooted uncleanly and then I have backgrounded
DM> KK> > fsck locked for more than an our in snaplk:
DM> KK> >
DM> KK> > 742 root 1 -4 4 1320K 688K snaplk 0:02 0.00% fsck_ufs
DM> KK> >
DM> KK> > File system in question is 200G gmirror on SATA. Usually making a snapshot
DM> KK> > (e.g., for making dumps) consumes 3-4 minutes for that fs, so it seems to me
DM> KK> > that filesystem is in a deadlock.
DM> KK>
DM> KK> Is the process performing I/O? Background fsck deliberately runs at a
DM> KK> slow rate so it does not destroy I/O performance on the rest of the
DM> KK> system.
DM>
DM> Nope. For that case, 50+ smbds had been locked in 'ufs' state, so I've been
DM> urged to revive the machine and reboot, turning off bgfsck.
DM>
DM> This night, dump -L locks in the same position on the same filesystem:
DM>
DM> 0 2887 2886 0 -4 0 1260 692 snaplk D ?? 0:01.28
DM> /sbin/mksnap_ffs root 0.0 0.1 5:19AM
DM>
DM> it has been started at 5:19am, and now is 9:20 - no disk activity
DM>
DM>
DM> For the reference: it's fresh RELENG_6_1/i386.
Just rechecked it: did mksnap_ffs on an otherwise idle file system:
marck at office:/> mksnap_ffs /st /st/.snap/test_snapshot
load: 0.02 cmd: mksnap_ffs 4012 [biord] 0.00u 0.04s 0% 696k
load: 0.04 cmd: mksnap_ffs 4012 [biord] 0.00u 0.44s 0% 696k
load: 0.21 cmd: mksnap_ffs 4012 [snaprdb] 0.00u 1.17s 0% 696k
load: 0.20 cmd: mksnap_ffs 4012 [snaprdb] 0.00u 1.23s 0% 696k
load: 0.13 cmd: mksnap_ffs 4012 [snaplk] 0.00u 1.30s 0% 696k
load: 0.08 cmd: mksnap_ffs 4012 [snaplk] 0.00u 1.30s 0% 696k
load: 0.01 cmd: mksnap_ffs 4012 [snaplk] 0.00u 1.30s 0% 696k
(I hit ^T several times)
biord phase consumes about 1.5-2 mins,
snaprdb phase - about 30-40 secs, and then process died. Most disk requests
succeeds; however, accessing /st/.snap locks process in ufs state forever.
What bothers me most is that it is the only machine reproducibly hangs in
snapshots, and it did not hang before RELENG_5 -> RELENG_6 upgrade. Other
RELENG_6 machines do snapshot backups flawlessly (knock-on-wood!)
Sincerely,
D.Marck [DM5020, MCK-RIPE, DM3-RIPN]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck at rinet.ru ***
------------------------------------------------------------------------
More information about the freebsd-stable
mailing list