zfs problem with processes stuck in state zfs and zilog-

Kai Gallasch gallasch at free.de
Tue Jan 26 12:06:12 UTC 2010


Hi.

Some mail services (qmail, dovecot) on one of my freebsd jails became
unavailable. The jail is using zfs as a filesystem. After logging in I
found that I could not kill processes related to the unavailable
services running inside the jail and that the jail could not be
restartet.

top revealed the following:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 8320 root        1  44    0  4772K  1088K zilog-  1   0:58  0.00% multilog
 8336 qmaill      1  44    0  4772K  1088K zilog-  6   0:21  0.00% multilog
 8464 mysql       2  44    0   108M 70356K STOP    7   0:21  0.00% mysqld
 8292 spamd       1  44    0  4772K  1088K zilog-  7   0:18  0.00% multilog
 8295 nobody      1  44    0  4772K  1088K zilog-  2   0:10  0.00% multilog
 8298 qmaill      1  44    0  4772K  1088K zilog-  3   0:01  0.00% multilog
 8294 root        1  44    0  4772K  1088K zilog-  4   0:01  0.00% multilog
 8304 qmaill      1  44    0  4772K  1088K zilog-  1   0:01  0.00% multilog
34278 www         1  63    0 29852K 14828K zfs     4   0:00  0.00% perl
34368 www         1  76    0 29852K 14828K zfs     4   0:00  0.00% perl
 8319 root        1  44    0  4772K  1088K zilog-  2   0:00  0.00% multilog
 4311 qmaild      1  59    0  2792K  1040K zilog-  1   0:00  0.00% qmail-queue
32203 qmaild      1  60    0  2792K  1040K zilog-  0   0:00  0.00% qmail-queue
16225 qmaild      1  59    0  2792K  1040K zilog-  4   0:00  0.00% qmail-queue
32254 qmaild      1  60    0  2792K  1040K zilog-  0   0:00  0.00% qmail-queue
16545 qmaild      1  57    0  2792K  1040K zilog-  4   0:00  0.00% qmail-queue
 4521 qmaild      1  56    0  2792K  1040K zilog-  5   0:00  0.00% qmail-queue
 8327 root        1  44    0  4772K  1088K zilog-  6   0:00  0.00% multilog
36733 qmaild      1  52    0  2792K  1040K zilog-  4   0:00  0.00% qmail-queue
47903 root        1  44    0 10284K  3412K pause   0   0:00  0.00% tcsh
35517 qmaild      1  63    0  2792K  1040K zilog-  4   0:00  0.00% qmail-queue
35492 qmaild      1  63    0  2792K  1040K zilog-  0   0:00  0.00% qmail-queue
62826 qmaild      1  53    0  2792K  1040K zilog-  4   0:00  0.00% qmail-queue
69033 qmaild      1  53    0  2792K  1040K zilog-  0   0:00  0.00% qmail-queue
66010 qmaild      1  50    0  2792K  1040K zilog-  4   0:00  0.00% qmail-queue
40947 qmaild      1  50    0  2792K  1040K zilog-  0   0:00  0.00% qmail-queue
27726 qmaild      1  51    0  2792K  1040K zilog-  0   0:00  0.00% qmail-queue
35520 qmaild      1  53    0  2792K  1040K zilog-  6   0:00  0.00% qmail-queue
30639 root        1  44    0 12164K  3420K zfs     1   0:00  0.00% perl
23538 root        1  44    0 12164K  3420K zfs     1   0:00  0.00% perl
 4834 root        1  76    0 12164K  3420K zfs     1   0:00  0.00% perl
26410 root        1  47    0 12164K  3420K zfs     0   0:00  0.00% perl
97136 root        1  76    0 12164K  3420K zfs     4   0:00  0.00% perl
 9290 root        1  44    0 12164K  3420K zfs     1   0:00  0.00% perl
18127 root        1  44    0 12164K  3420K zfs     5   0:00  0.00% perl
12553 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
21574 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
 2323 root        1  44    0 12164K  3420K zfs     0   0:00  0.00% perl
68707 root        1  45    0 12164K  3420K zfs     2   0:00  0.00% perl
 9294 root        1  44    0 12164K  3420K zfs     1   0:00  0.00% perl
40393 root        1  44    0 12164K  3420K zfs     1   0:00  0.00% perl
41985 root        1  44    0 12164K  3420K zfs     0   0:00  0.00% perl
21672 root        1  44    0 12164K  3420K zfs     0   0:00  0.00% perl
10420 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
 6965 root        1  44    0 12164K  3420K zfs     6   0:00  0.00% perl
14132 root        1  44    0 12164K  3420K zfs     1   0:00  0.00% perl
14867 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
77145 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
16534 root        1  44    0 12164K  3420K zfs     0   0:00  0.00% perl
25130 root        1  44    0 12164K  3420K zfs     0   0:00  0.00% perl
63026 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
44663 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
28139 root        1  44    0 12164K  3420K zfs     0   0:00  0.00% perl
34440 root        1  44    0 12164K  3420K zfs     4   0:00  0.00% perl
35619 qmaild      1  46    0  2792K  1040K zilog-  0   0:00  0.00% qmail-queue
26146 root        1  44    0 12164K  3420K zfs     0   0:00  0.00% perl


Because the server is in active use, there was no time
for me to look into this further - and no trace of the problem in syslog.

Any suggestions how to debug this problem and find out what's wrong here?
Does the state zilog- give any hint?

BTW: FreeBSD 8.0-STABLE-amd64 / zpool version 13

 --Kai




-- 

You had mail, but the super-user read it,
laughed and deleted it!


More information about the freebsd-fs mailing list