From olivier at gid0.org  Thu Oct  1 09:21:05 2009
From: olivier at gid0.org (Olivier Smedts)
Date: Thu Oct  1 09:21:12 2009
Subject: kern/139072: [zfs] zfs marked as production ready but it used a
	deprecated checksum algorithm
In-Reply-To: <200909230920.n8N9KIJ6005528@freefall.freebsd.org>
References: <200909230920.n8N9KIJ6005528@freefall.freebsd.org>
Message-ID: <367b2c980910010221kd388f43q8243797b4eac9af7@mail.gmail.com>

Hello,

2009/9/23  <pjd@freebsd.org>:
> Synopsis: [zfs] zfs marked as production ready but it used a deprecated checksum algorithm
>
> Responsible-Changed-From-To: freebsd-fs->pjd
> Responsible-Changed-By: pjd
> Responsible-Changed-When: ?ro 23 wrz 2009 09:19:57 UTC
> Responsible-Changed-Why:
> I'll take this one.
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=139072

Now that this PR is closed, is there something to change on *existing*
zfs filesystems to make them use fletcher4 (for new data) when they
have the default property "checksum=on" ? Is there something to do
(other than dumping and restoring) to change checksums to fletcher4
for existing data and metadata ?

Thanks,
Olivier

-- 
Olivier Smedts                                                 _
                                        ASCII ribbon campaign ( )
e-mail: olivier@gid0.org        - against HTML email & vCards  X
www: http://www.gid0.org    - against proprietary attachments / \

  "Il y a seulement 10 sortes de gens dans le monde :
  ceux qui comprennent le binaire,
  et ceux qui ne le comprennent pas."
From solon at pyro.de  Thu Oct  1 09:43:31 2009
From: solon at pyro.de (Solon Lutz)
Date: Thu Oct  1 09:43:37 2009
Subject: Help needed! ZFS I/O error recovery?
Message-ID: <683849754.20091001110503@pyro.de>

Hi erverybody,

I'm faced with a 10TB ZFS pool on a 12TB RAID6 Areca controller.
And yes, I know, you shouldn't put a zpool on a RAID-device... =(

Due to problems with a sata-cable, some days ago the raid-controller
started to produce long timeouts to recover the resulting read errors.

The cable was replaced, a parity check was run on the RAID-Volume and
showed no errors, the zfs scrub however showed some 'defective' files.
After copying these files with 'dd -conv=noerror...' and comparing them
to the originals, they were error-free.

Yesterday however, three more defective cables forced the controller
to take the RAID6 volume offline. Now all cables were replaced and a parity
check was run on the RAID-Volume -> data integrity OK.

But now ZFS refuses to mount all volumes:

Solaris: WARNING: can't process intent log for temp/space1
Solaris: WARNING: can't process intent log for temp/space2
Solaris: WARNING: can't process intent log for temp/space3
Solaris: WARNING: can't process intent log for temp/space4

A scrub revealed to following:

errors: Permanent errors have been detected in the following files:

        temp:<0x0>
        temp/space1:<0x0>
        temp/space2:<0x0>
        temp/space3:<0x0>
        temp/space4:<0x0>


I tried to switch off checksums for this pool, but that didn't help in any
way. I also mounted the pool by hand and was faced with with 'empty' volumes
and 'I/O errors' when trying to list their contents...

Any suggestions? I'm offering some self-made blackberry jam and raspberry brandy
to the person who can help to restore or backup the data.

Tech specs:

FreeBSD 7.2-STABLE #21: Tue May  5 18:44:10 CEST 2009 (AMD64)
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <Areca ARC-1280-VOL#00 R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da0: Command Queueing Enabled
da0: 10490414MB (21484367872 512 byte sectors: 255H 63S/T 1337340C)
ZFS filesystem version 6
ZFS storage pool version 6




Best regards,

Solon

From pjd at FreeBSD.org  Thu Oct  1 10:31:15 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Thu Oct  1 10:31:22 2009
Subject: kern/139072: [zfs] zfs marked as production ready but it used a
	deprecated checksum algorithm
In-Reply-To: <367b2c980910010221kd388f43q8243797b4eac9af7@mail.gmail.com>
References: <200909230920.n8N9KIJ6005528@freefall.freebsd.org>
	<367b2c980910010221kd388f43q8243797b4eac9af7@mail.gmail.com>
Message-ID: <20091001103110.GB1595@garage.freebsd.pl>

On Thu, Oct 01, 2009 at 11:21:03AM +0200, Olivier Smedts wrote:
> Hello,
> 
> 2009/9/23  <pjd@freebsd.org>:
> > Synopsis: [zfs] zfs marked as production ready but it used a deprecated checksum algorithm
> >
> > Responsible-Changed-From-To: freebsd-fs->pjd
> > Responsible-Changed-By: pjd
> > Responsible-Changed-When: ?ro 23 wrz 2009 09:19:57 UTC
> > Responsible-Changed-Why:
> > I'll take this one.
> >
> > http://www.freebsd.org/cgi/query-pr.cgi?pr=139072
> 
> Now that this PR is closed, is there something to change on *existing*
> zfs filesystems to make them use fletcher4 (for new data) when they
> have the default property "checksum=on" ? Is there something to do
> (other than dumping and restoring) to change checksums to fletcher4
> for existing data and metadata ?

You have to manually change checksum to fletcher4 for existing datasets,
I think and backup/restore your data.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091001/b498a5df/attachment.pgp
From james-freebsd-fs2 at jrv.org  Thu Oct  1 13:51:27 2009
From: james-freebsd-fs2 at jrv.org (James R. Van Artsdalen)
Date: Thu Oct  1 13:51:34 2009
Subject: kern/139072: [zfs] zfs marked as production ready but it used
 a	deprecated checksum algorithm
In-Reply-To: <367b2c980910010221kd388f43q8243797b4eac9af7@mail.gmail.com>
References: <200909230920.n8N9KIJ6005528@freefall.freebsd.org>
	<367b2c980910010221kd388f43q8243797b4eac9af7@mail.gmail.com>
Message-ID: <4AC4B3DD.5050600@jrv.org>

Olivier Smedts wrote:
> Hello,
>
> Now that this PR is closed, is there something to change on *existing*
> zfs filesystems to make them use fletcher4 (for new data) when they
> have the default property "checksum=on"?

# zfs set checksum=fletcher4 pool

>  Is there something to do
> (other than dumping and restoring) to change checksums to fletcher4
> for existing data and metadata ?

No.

Even "fletcher4" has the undesirable property that the checksum of every
group of zeros, of any length, is the same as the initial value of the
accumulator.  This means that fletcher4 is insensitive to the number of
leading zeros in the checksummed data.

The ZFS team needs to revisit the checksum issue and add another
algorithm but they have other things to worry about at the moment.  Some
SHA-3 contestants claim to be very fast though it's not clear they're
fast enough to replace a true Fletcher sum in the real world, at least
not yet.
From mj at feral.com  Thu Oct  1 16:55:11 2009
From: mj at feral.com (Matthew Jacob)
Date: Thu Oct  1 16:55:18 2009
Subject: review needed for a simple fix to growfs
Message-ID: <4AC4D7AA.1080005@feral.com>

I have a simple fix to growfs that eliminates some issues Doug was 
seeing. I probably don't know enough to know some of the implications of 
the change and wonder if anyone would care to comment on it?

The symptoms would be fsck failures on the grown filesystem- my take is 
that it was because the new cylinder groups were being initialized as 
having all the inodes allocated. This is puzzling me because, like, how 
could this ever have worked?

It was trivial for me to reproduce- see 
http://people.freebsd.org/~mjacob/growfs.failure.txt.

The second change, btw, is not essential- it just adjusts maxino down if 
you had to drop the number of cylinder groups down.


Index: growfs.c
===================================================================
--- growfs.c    (revision 197658)
+++ growfs.c    (working copy)
@@ -401,7 +401,6 @@
        acg.cg_magic = CG_MAGIC;
        acg.cg_cgx = cylno;
        acg.cg_niblk = sblock.fs_ipg;
-       acg.cg_initediblk = sblock.fs_ipg;
        acg.cg_ndblk = dmax - cbase;
        if (sblock.fs_contigsumsize > 0)
                acg.cg_nclusterblks = acg.cg_ndblk / sblock.fs_frag;
@@ -2217,6 +2216,7 @@
                printf("Warning: %jd sector(s) cannot be allocated.\n",
                    (intmax_t)fsbtodb(&sblock, sblock.fs_size % 
sblock.fs_fpg));
                sblock.fs_size = sblock.fs_ncg * sblock.fs_fpg;
+               maxino -= sblock.fs_ipg;
        }

        /*

From pepelac at gmail.com  Thu Oct  1 20:48:29 2009
From: pepelac at gmail.com (Alexander Shevchenko)
Date: Thu Oct  1 20:48:35 2009
Subject: ARC & L2ARC efficiency
Message-ID: <8c9ae7950910011322j1a6b66fcp73615cc17ae20328@mail.gmail.com>

Good time of day!

How could i check the efficiency of ARC?
Are total reads from pool equal kstat.zfs.misc.arcstats.hits +
kstat.zfs.misc.arcstats.misses, or this values are just reads from cache?
By  efficiency i mean reads_from_cache/(reads_from_cache+reads_from_drives)
Are there any document where kstat values described?

zpool status
  pool: data
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          da2       ONLINE       0     0     0
          da4       ONLINE       0     0     0
        cache
          da3       ONLINE       0     0     0


#sysctl kstat
kstat.zfs.misc.arcstats.hits: 282927703
kstat.zfs.misc.arcstats.misses: 66220328
kstat.zfs.misc.arcstats.demand_data_hits: 164374119
kstat.zfs.misc.arcstats.demand_data_misses: 6615511
kstat.zfs.misc.arcstats.demand_metadata_hits: 88715021
kstat.zfs.misc.arcstats.demand_metadata_misses: 4464890
kstat.zfs.misc.arcstats.prefetch_data_hits: 28851210
kstat.zfs.misc.arcstats.prefetch_data_misses: 55109950
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 987353
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 29977
kstat.zfs.misc.arcstats.mru_hits: 44560461
kstat.zfs.misc.arcstats.mru_ghost_hits: 1493532
kstat.zfs.misc.arcstats.mfu_hits: 211027800
kstat.zfs.misc.arcstats.mfu_ghost_hits: 16337660
kstat.zfs.misc.arcstats.deleted: 49112923
kstat.zfs.misc.arcstats.recycle_miss: 9574100
kstat.zfs.misc.arcstats.mutex_miss: 252423
kstat.zfs.misc.arcstats.evict_skip: 2269320648
kstat.zfs.misc.arcstats.hash_elements: 644877
kstat.zfs.misc.arcstats.hash_elements_max: 678888
kstat.zfs.misc.arcstats.hash_collisions: 21697862
kstat.zfs.misc.arcstats.hash_chains: 182323
kstat.zfs.misc.arcstats.hash_chain_max: 9
kstat.zfs.misc.arcstats.p: 1251375616
kstat.zfs.misc.arcstats.c: 1252817408
kstat.zfs.misc.arcstats.c_min: 1252817408
kstat.zfs.misc.arcstats.c_max: 10022539264
kstat.zfs.misc.arcstats.size: 1237578176
kstat.zfs.misc.arcstats.hdr_size: 9610640
kstat.zfs.misc.arcstats.l2_hits: 12905801
kstat.zfs.misc.arcstats.l2_misses: 680
kstat.zfs.misc.arcstats.l2_feeds: 52666
kstat.zfs.misc.arcstats.l2_rw_clash: 680
kstat.zfs.misc.arcstats.l2_writes_sent: 41330
kstat.zfs.misc.arcstats.l2_writes_done: 41330
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 62
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 53
kstat.zfs.misc.arcstats.l2_evict_reading: 5
kstat.zfs.misc.arcstats.l2_free_on_write: 30044
kstat.zfs.misc.arcstats.l2_abort_lowmem: 309837
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 79319831552
kstat.zfs.misc.arcstats.l2_hdr_size: 134102528
kstat.zfs.misc.arcstats.memory_throttle_count: 112340
kstat.zfs.misc.vdev_cache_stats.delegations: 3822
kstat.zfs.misc.vdev_cache_stats.hits: 342974
kstat.zfs.misc.vdev_cache_stats.misses: 170601


WBR,
Alexander Shevchenko
From fbsdlist at src.cx  Thu Oct  1 23:20:03 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Thu Oct  1 23:20:09 2009
Subject: ARC & L2ARC efficiency
In-Reply-To: <ed91d4a80910011600x3f3e855etd24694bde69a96c7@mail.gmail.com>
References: <8c9ae7950910011322j1a6b66fcp73615cc17ae20328@mail.gmail.com>
	<ed91d4a80910011600x3f3e855etd24694bde69a96c7@mail.gmail.com>
Message-ID: <ed91d4a80910011620y73f931c9ie1e41692d6fc46e3@mail.gmail.com>

Here's another script:
http://www.solarisinternals.com/wiki/index.php/Arcstat

Attached hacked FreeBSD version.

--Artem



On Thu, Oct 1, 2009 at 4:00 PM, Artem Belevich <fbsdlist@src.cx> wrote:
> There's a pretty useful script to present ARC stats (alas, L2ARC info
> is not included) in a readable way:
> http://cuddletech.com/arc_summary/
>
> I've attaches somewhat hacked (and a bit outdated) version that runs on FreeBSD.
>
> --Artem
>
>
>
> On Thu, Oct 1, 2009 at 1:22 PM, Alexander Shevchenko <pepelac@gmail.com> wrote:
>> Good time of day!
>>
>> How could i check the efficiency of ARC?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arcstat.pl
Type: application/octet-stream
Size: 8192 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091001/295b345e/arcstat.obj
From fbsdlist at src.cx  Thu Oct  1 23:24:12 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Thu Oct  1 23:24:19 2009
Subject: ARC & L2ARC efficiency
In-Reply-To: <8c9ae7950910011322j1a6b66fcp73615cc17ae20328@mail.gmail.com>
References: <8c9ae7950910011322j1a6b66fcp73615cc17ae20328@mail.gmail.com>
Message-ID: <ed91d4a80910011600x3f3e855etd24694bde69a96c7@mail.gmail.com>

There's a pretty useful script to present ARC stats (alas, L2ARC info
is not included) in a readable way:
http://cuddletech.com/arc_summary/

I've attaches somewhat hacked (and a bit outdated) version that runs on FreeBSD.

--Artem



On Thu, Oct 1, 2009 at 1:22 PM, Alexander Shevchenko <pepelac@gmail.com> wrote:
> Good time of day!
>
> How could i check the efficiency of ARC?
> Are total reads from pool equal kstat.zfs.misc.arcstats.hits +
> kstat.zfs.misc.arcstats.misses, or this values are just reads from cache?
> By ?efficiency i mean reads_from_cache/(reads_from_cache+reads_from_drives)
> Are there any document where kstat values described?
>
> zpool status
> ?pool: data
> ?state: ONLINE
> ?scrub: none requested
> config:
>
> ? ? ? ?NAME ? ? ? ?STATE ? ? READ WRITE CKSUM
> ? ? ? ?data ? ? ? ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
> ? ? ? ? ?da2 ? ? ? ONLINE ? ? ? 0 ? ? 0 ? ? 0
> ? ? ? ? ?da4 ? ? ? ONLINE ? ? ? 0 ? ? 0 ? ? 0
> ? ? ? ?cache
> ? ? ? ? ?da3 ? ? ? ONLINE ? ? ? 0 ? ? 0 ? ? 0
>
>
> #sysctl kstat
> kstat.zfs.misc.arcstats.hits: 282927703
> kstat.zfs.misc.arcstats.misses: 66220328
> kstat.zfs.misc.arcstats.demand_data_hits: 164374119
> kstat.zfs.misc.arcstats.demand_data_misses: 6615511
> kstat.zfs.misc.arcstats.demand_metadata_hits: 88715021
> kstat.zfs.misc.arcstats.demand_metadata_misses: 4464890
> kstat.zfs.misc.arcstats.prefetch_data_hits: 28851210
> kstat.zfs.misc.arcstats.prefetch_data_misses: 55109950
> kstat.zfs.misc.arcstats.prefetch_metadata_hits: 987353
> kstat.zfs.misc.arcstats.prefetch_metadata_misses: 29977
> kstat.zfs.misc.arcstats.mru_hits: 44560461
> kstat.zfs.misc.arcstats.mru_ghost_hits: 1493532
> kstat.zfs.misc.arcstats.mfu_hits: 211027800
> kstat.zfs.misc.arcstats.mfu_ghost_hits: 16337660
> kstat.zfs.misc.arcstats.deleted: 49112923
> kstat.zfs.misc.arcstats.recycle_miss: 9574100
> kstat.zfs.misc.arcstats.mutex_miss: 252423
> kstat.zfs.misc.arcstats.evict_skip: 2269320648
> kstat.zfs.misc.arcstats.hash_elements: 644877
> kstat.zfs.misc.arcstats.hash_elements_max: 678888
> kstat.zfs.misc.arcstats.hash_collisions: 21697862
> kstat.zfs.misc.arcstats.hash_chains: 182323
> kstat.zfs.misc.arcstats.hash_chain_max: 9
> kstat.zfs.misc.arcstats.p: 1251375616
> kstat.zfs.misc.arcstats.c: 1252817408
> kstat.zfs.misc.arcstats.c_min: 1252817408
> kstat.zfs.misc.arcstats.c_max: 10022539264
> kstat.zfs.misc.arcstats.size: 1237578176
> kstat.zfs.misc.arcstats.hdr_size: 9610640
> kstat.zfs.misc.arcstats.l2_hits: 12905801
> kstat.zfs.misc.arcstats.l2_misses: 680
> kstat.zfs.misc.arcstats.l2_feeds: 52666
> kstat.zfs.misc.arcstats.l2_rw_clash: 680
> kstat.zfs.misc.arcstats.l2_writes_sent: 41330
> kstat.zfs.misc.arcstats.l2_writes_done: 41330
> kstat.zfs.misc.arcstats.l2_writes_error: 0
> kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 62
> kstat.zfs.misc.arcstats.l2_evict_lock_retry: 53
> kstat.zfs.misc.arcstats.l2_evict_reading: 5
> kstat.zfs.misc.arcstats.l2_free_on_write: 30044
> kstat.zfs.misc.arcstats.l2_abort_lowmem: 309837
> kstat.zfs.misc.arcstats.l2_cksum_bad: 0
> kstat.zfs.misc.arcstats.l2_io_error: 0
> kstat.zfs.misc.arcstats.l2_size: 79319831552
> kstat.zfs.misc.arcstats.l2_hdr_size: 134102528
> kstat.zfs.misc.arcstats.memory_throttle_count: 112340
> kstat.zfs.misc.vdev_cache_stats.delegations: 3822
> kstat.zfs.misc.vdev_cache_stats.hits: 342974
> kstat.zfs.misc.vdev_cache_stats.misses: 170601
>
>
> WBR,
> Alexander Shevchenko
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arc_summary.pl
Type: application/octet-stream
Size: 7357 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091001/9e32cd8d/arc_summary.obj
From trasz at FreeBSD.org  Fri Oct  2 07:32:44 2009
From: trasz at FreeBSD.org (trasz@FreeBSD.org)
Date: Fri Oct  2 07:32:50 2009
Subject: kern/133373: [zfs] umass attachment causes ZFS checksum errors,
	data loss
Message-ID: <200910020732.n927Wh1u036437@freefall.freebsd.org>

Synopsis: [zfs] umass attachment causes ZFS checksum errors, data loss

Responsible-Changed-From-To: freebsd-fs->trasz
Responsible-Changed-By: trasz
Responsible-Changed-When: Fri Oct 2 07:32:43 UTC 2009
Responsible-Changed-Why: 
I'll take it.

http://www.freebsd.org/cgi/query-pr.cgi?pr=133373
From bra at fsn.hu  Fri Oct  2 07:59:10 2009
From: bra at fsn.hu (Attila Nagy)
Date: Fri Oct  2 07:59:17 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <4AC1E540.9070001@fsn.hu>
References: <4AC1E540.9070001@fsn.hu>
Message-ID: <4AC5B2C7.2000200@fsn.hu>

On 09/29/09 12:45, Attila Nagy wrote:
> I'm using FreeBSD 8 (previously 7) on a machine with a lot of disks 
> and 32 GB RAM. With 7.x it ran very well for about 50 days, but 
> suddenly every operation have slowed down.
> gstat showed that the disks are working a lot more than usual the 
> zpool/zfs was pretty unusable.
>
> I've rebooted the machine then with FreeBSD 8 in the hope the new ZFS 
> fixes will correct this issue (no 50 days have passed since then, so I 
> don't know yet) and started to monitor ZFS's statistics.
>
> It seems that after a reboot, the ARC size starts to grow, then 
> something flips the switch and it changes to shrinking, instead of 
> maintaining the size.
>
> Please see the pictures here: 
> http://people.fsn.hu/~bra/freebsd/20090929-zfs-arcsize/
>
> Before the 27th, the machine ran FreeBSD 7, after that date it runs 8.
>
> As you can see, no user process tooks the memory, so I don't know why 
> the ARC size grows first and then start to decrease.
>
> Could it be that the ARC size decreases such a big amount that it 
> effectively disappears and this causes the IO activity go up and kill 
> the machine?
I've upgraded another machine from an older 8-CURRENT to 8-STABLE. It 
has low memory (1GB) and it's i386.
The above symptoms can be triggered very easily: if I do an IMAP search 
on a lot of mailboxes (which I do regularly), about 10 minutes needed 
for the IMAP server to become completely inaccessible.
The machine runs fine, but every operation of the ZFS pool take ages.
According to gstat there is only a very minimal disk activity. The 
machine can't even be rebooted, at least not in ten minutes (reboot, 
wait 10 minutes, nearly nothing happens, reboot -qn makes the machine 
disappear from the net, but it doesn't restart).

Backing out this change from the 8-STABLE kernel:
http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902

makes it survive about half and hour of IMAP searching. Of course only 
time will tell whether this helps in the long run, but so far 10/10 
tries succeeded to kill the machine with this method...

According to this, I would say that this change makes things worse even 
on low memory, i386 (1G RAM) and "there's a plenty of RAM" (32 G) amd64 
servers.
From gerrit at pmp.uni-hannover.de  Fri Oct  2 13:47:33 2009
From: gerrit at pmp.uni-hannover.de (Gerrit =?ISO-8859-1?Q?K=FChn?=)
Date: Fri Oct  2 13:47:40 2009
Subject: Fw: Linux/KDE and NFS locking on 7-stable
In-Reply-To: <Pine.GSO.4.63.0909181024360.15785@muncher.cs.uoguelph.ca>
References: <20090917094412.962e8729.gerrit@pmp.uni-hannover.de>
	<Pine.GSO.4.63.0909171056210.1169@muncher.cs.uoguelph.ca>
	<20090918091435.465bfc1e.gerrit@pmp.uni-hannover.de>
	<Pine.GSO.4.63.0909181024360.15785@muncher.cs.uoguelph.ca>
Message-ID: <20091002154729.06fbcefc.gerrit@pmp.uni-hannover.de>

On Fri, 18 Sep 2009 10:56:59 -0400 (EDT) Rick Macklem
<rmacklem@uoguelph.ca> wrote about Re: Fw: Linux/KDE and NFS locking on
7-stable:

First of all, thanks to Rick for his explatations.

RM> > RM> I believe setting the following in the server's /etc/rc.conf
RM> > RM> and rebooting the server (or just killing off lockd on the
RM> > RM> server), combined with "nolock" as you have on the above Linux
RM> > RM> mount, might work ok:
RM> > RM>  	rpc_lockd_enable="NO"
RM> > RM>  	rpc_statd_enable="NO"
RM> >
RM> > I did not try that so far, as there are some clients which seem to
RM> > work fine with locking.

Well, in the meantime I tried more or less each and every combination of
client and server setting I could think of.
My first result is that locking almost every time causes troubles, because
is does not work over NAT.
However, even putting the server in the same subnet and using it without
NAT did not make everything work with locking.

Right now it looks like after booting the client KDE logins *never* work
(all other nfs stuff is fine, though). After a failed login, I can umount
and remount the home dir and try to login again. After doing so for some
time, it suddenly works (and continues to work) obviously regardless of
mount options being changed.
Turning off lockd and statd on the server side does not influence this
behaviour.

Does anyone here have an idea how I can get a working client from the
beginning on? Also, I do not even know if this is some kind of kde problem
or rather a nfs problem. After all, it is rather annoying do have to login
and umount/remount several times before everything works. :-)


cu
  Gerrit
From gerrit at pmp.uni-hannover.de  Fri Oct  2 14:47:26 2009
From: gerrit at pmp.uni-hannover.de (Gerrit =?ISO-8859-1?Q?K=FChn?=)
Date: Fri Oct  2 14:47:38 2009
Subject: Fw: Linux/KDE and NFS locking on 7-stable
In-Reply-To: <20091002154729.06fbcefc.gerrit@pmp.uni-hannover.de>
References: <20090917094412.962e8729.gerrit@pmp.uni-hannover.de>
	<Pine.GSO.4.63.0909171056210.1169@muncher.cs.uoguelph.ca>
	<20090918091435.465bfc1e.gerrit@pmp.uni-hannover.de>
	<Pine.GSO.4.63.0909181024360.15785@muncher.cs.uoguelph.ca>
	<20091002154729.06fbcefc.gerrit@pmp.uni-hannover.de>
Message-ID: <20091002164723.5484c2d1.gerrit@pmp.uni-hannover.de>

On Fri, 2 Oct 2009 15:47:29 +0200 Gerrit K?hn <gerrit@pmp.uni-hannover.de>
wrote about Re: Fw: Linux/KDE and NFS locking on 7-stable:

GK> Does anyone here have an idea how I can get a working client from the
GK> beginning on? Also, I do not even know if this is some kind of kde
GK> problem or rather a nfs problem. After all, it is rather annoying do
GK> have to login and umount/remount several times before everything
GK> works. :-)

I have one more thing to add: as a last resort, I set the home dirs mount
back to nfsvers=2 (together with tcp and nolock). This seems to work fine
(although nfs2 is probably not a good idea these days - I guess I am
looking forward to 8.0 including a nfs4 server :-).


cu
  Gerrit
From aaron at goflexitllc.com  Fri Oct  2 16:04:09 2009
From: aaron at goflexitllc.com (Aaron Hurt)
Date: Fri Oct  2 16:04:16 2009
Subject: ZFS I/O Error - unable to import/mount pool
Message-ID: <4AC62471.1080009@goflexitllc.com>

I have a rather small ZFS pool on a 7-STABLE machine composed of 4 disks 
in raidz-1.  One of the drives was giving me fits and acting flaky so I 
got a warranty replacement for it, however that drive as well as another 
both started giving DMA timeout/LBA write errors simultaneously 
lastnight.  I was able to bring the machine back up after it locked (no 
crash/panic it just locked up) and do a scrub to get the pool online.  I 
then powered down and disconnected the drive that was giving the 
majority of the errors.  When the system came back up it says the pool 
is faulted, re-attaching the disk I removed does not resolve the fault 
and the raidz mentions corrupt metadata.  I though I might be able to 
export/import the array but that also failed.  The array did export but 
now refuses to import saying I/O error.  If anyone could give me some 
insight on how I could get this pool back online just long enough to 
snag the data I would be really appreciative.  I will also be more than 
happy to provide any other detailed information you may need just let me 
know.

Thank You,

-- 
Aaron Hurt
Managing Partner
Flex I.T., LLC
611 Commerce Street
Suite 3117
Nashville, TN  37203
Phone: 615.438.7101
E-mail: aaron@goflexitllc.com

From jamie at FreeBSD.org  Fri Oct  2 16:13:58 2009
From: jamie at FreeBSD.org (jamie@FreeBSD.org)
Date: Fri Oct  2 16:14:04 2009
Subject: kern/139198: [nfs] Page Fault out of NLM
Message-ID: <200910021613.n92GDu8h056564@freefall.freebsd.org>

Synopsis: [nfs] Page Fault out of NLM

State-Changed-From-To: open->closed
State-Changed-By: jamie
State-Changed-When: Fri Oct 2 16:13:00 UTC 2009
State-Changed-Why: 
Fixed in r197667.

http://www.freebsd.org/cgi/query-pr.cgi?pr=139198
From pjd at FreeBSD.org  Fri Oct  2 18:45:44 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Fri Oct  2 18:45:51 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <4AC5B2C7.2000200@fsn.hu>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
Message-ID: <20091002184526.GA1660@garage.freebsd.pl>

On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
> Backing out this change from the 8-STABLE kernel:
> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
> 
> makes it survive about half and hour of IMAP searching. Of course only 
> time will tell whether this helps in the long run, but so far 10/10 
> tries succeeded to kill the machine with this method...

Could you try this patch:

	http://people.freebsd.org/~pjd/patches/arc.c.4.patch

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091002/9bee6d8e/attachment.pgp
From gleb.kurtsou at gmail.com  Fri Oct  2 20:50:03 2009
From: gleb.kurtsou at gmail.com (Gleb Kurtsou)
Date: Fri Oct  2 20:50:11 2009
Subject: kern/127213: [tmpfs] sendfile on tmpfs data corruption
Message-ID: <200910022050.n92Ko3hR036603@freefall.freebsd.org>

The following reply was made to PR kern/127213; it has been noted by GNATS.

From: Gleb Kurtsou <gleb.kurtsou@gmail.com>
To: bug-followup@FreeBSD.org, citrin@citrin.ru
Cc:  
Subject: Re: kern/127213: [tmpfs] sendfile on tmpfs data corruption
Date: Fri, 2 Oct 2009 23:48:29 +0300

 --GvXjxJ+pjyke8COw
 Content-Type: text/plain; charset=utf-8
 Content-Disposition: inline
 
 [ resending it with changed subject.
 looks like it was ignored by bug-followup@ ]
 
 Try the patch attached, it fixes the bug for me.
 The same workaround is used by zfs.
 
 
 --GvXjxJ+pjyke8COw
 Content-Type: text/plain; charset=utf-8
 Content-Disposition: attachment; filename="tmpfs-sendfile.patch.txt"
 
 diff --git a/sys/fs/tmpfs/tmpfs_vnops.c b/sys/fs/tmpfs/tmpfs_vnops.c
 index db8ceea..a6e4510 100644
 --- a/sys/fs/tmpfs/tmpfs_vnops.c
 +++ b/sys/fs/tmpfs/tmpfs_vnops.c
 @@ -43,6 +43,8 @@ __FBSDID("$FreeBSD$");
  #include <sys/priv.h>
  #include <sys/proc.h>
  #include <sys/resourcevar.h>
 +#include <sys/sched.h>
 +#include <sys/sf_buf.h>
  #include <sys/stat.h>
  #include <sys/systm.h>
  #include <sys/unistd.h>
 @@ -428,15 +430,72 @@ tmpfs_setattr(struct vop_setattr_args *v)
  }
  
  /* --------------------------------------------------------------------- */
 +static int
 +tmpfs_nocacheread(vm_object_t tobj, vm_pindex_t idx,
 +    vm_offset_t offset, size_t tlen, struct uio *uio)
 +{
 +	vm_page_t	m;
 +	int		error;
 +
 +	VM_OBJECT_LOCK(tobj);
 +	vm_object_pip_add(tobj, 1);
 +	m = vm_page_grab(tobj, idx, VM_ALLOC_WIRED |
 +	    VM_ALLOC_ZERO | VM_ALLOC_NORMAL | VM_ALLOC_RETRY);
 +	if (m->valid != VM_PAGE_BITS_ALL) {
 +		if (vm_pager_has_page(tobj, idx, NULL, NULL)) {
 +			error = vm_pager_get_pages(tobj, &m, 1, 0);
 +			if (error != 0) {
 +				printf("tmpfs get pages from pager error [read]\n");
 +				goto out;
 +			}
 +		} else
 +			vm_page_zero_invalid(m, TRUE);
 +	}
 +	VM_OBJECT_UNLOCK(tobj);
 +	error = uiomove_fromphys(&m, offset, tlen, uio);
 +	VM_OBJECT_LOCK(tobj);
 +out:
 +	vm_page_lock_queues();
 +	vm_page_unwire(m, TRUE);
 +	vm_page_unlock_queues();
 +	vm_page_wakeup(m);
 +	vm_object_pip_subtract(tobj, 1);
 +	VM_OBJECT_UNLOCK(tobj);
 +
 +	return (error);
 +}
 +
 +static __inline int
 +tmpfs_nocacheread_buf(vm_object_t tobj, vm_pindex_t idx,
 +    vm_offset_t offset, size_t tlen, void *buf)
 +{
 +	struct uio uio;
 +	struct iovec iov;
 +
 +	uio.uio_iovcnt = 1;
 +	uio.uio_iov = &iov;
 +	iov.iov_base = buf;
 +	iov.iov_len = tlen;
 +
 +	uio.uio_offset = 0;
 +	uio.uio_resid = tlen;
 +	uio.uio_rw = UIO_READ;
 +	uio.uio_segflg = UIO_SYSSPACE;
 +	uio.uio_td = curthread;
 +
 +	return (tmpfs_nocacheread(tobj, idx, offset, tlen, &uio));
 +}
  
  static int
  tmpfs_mappedread(vm_object_t vobj, vm_object_t tobj, size_t len, struct uio *uio)
  {
 +	struct sf_buf	*sf;
  	vm_pindex_t	idx;
  	vm_page_t	m;
  	vm_offset_t	offset;
  	off_t		addr;
  	size_t		tlen;
 +	char		*ma;
  	int		error;
  
  	addr = uio->uio_offset;
 @@ -460,33 +519,30 @@ lookupvpg:
  		vm_page_wakeup(m);
  		VM_OBJECT_UNLOCK(vobj);
  		return	(error);
 +	} else if (m != NULL && uio->uio_segflg == UIO_NOCOPY) {
 +		if (vm_page_sleep_if_busy(m, FALSE, "tmfsmr"))
 +			goto lookupvpg;
 +		vm_page_busy(m);
 +		VM_OBJECT_UNLOCK(vobj);
 +		sched_pin();
 +		sf = sf_buf_alloc(m, SFB_CPUPRIVATE);
 +		ma = (char *)sf_buf_kva(sf);
 +		error = tmpfs_nocacheread_buf(tobj, idx, offset, tlen,
 +		    ma + offset);
 +		if (error == 0) {
 +			uio->uio_offset += tlen;
 +			uio->uio_resid -= tlen;
 +		}
 +		sf_buf_free(sf);
 +		sched_unpin();
 +		VM_OBJECT_LOCK(vobj);
 +		vm_page_wakeup(m);
 +		VM_OBJECT_UNLOCK(vobj);
 +		return	(error);
  	}
  	VM_OBJECT_UNLOCK(vobj);
  nocache:
 -	VM_OBJECT_LOCK(tobj);
 -	vm_object_pip_add(tobj, 1);
 -	m = vm_page_grab(tobj, idx, VM_ALLOC_WIRED |
 -	    VM_ALLOC_ZERO | VM_ALLOC_NORMAL | VM_ALLOC_RETRY);
 -	if (m->valid != VM_PAGE_BITS_ALL) {
 -		if (vm_pager_has_page(tobj, idx, NULL, NULL)) {
 -			error = vm_pager_get_pages(tobj, &m, 1, 0);
 -			if (error != 0) {
 -				printf("tmpfs get pages from pager error [read]\n");
 -				goto out;
 -			}
 -		} else
 -			vm_page_zero_invalid(m, TRUE);
 -	}
 -	VM_OBJECT_UNLOCK(tobj);
 -	error = uiomove_fromphys(&m, offset, tlen, uio);
 -	VM_OBJECT_LOCK(tobj);
 -out:
 -	vm_page_lock_queues();
 -	vm_page_unwire(m, TRUE);
 -	vm_page_unlock_queues();
 -	vm_page_wakeup(m);
 -	vm_object_pip_subtract(tobj, 1);
 -	VM_OBJECT_UNLOCK(tobj);
 +	error = tmpfs_nocacheread(tobj, idx, offset, tlen, uio);
  
  	return	(error);
  }
 
 --GvXjxJ+pjyke8COw--
From linimon at FreeBSD.org  Fri Oct  2 21:54:41 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Fri Oct  2 21:54:53 2009
Subject: kern/139312: [tmpfs] [patch] tmpfs mmap synchronization bug
Message-ID: <200910022154.n92LsftZ007924@freefall.freebsd.org>

Old Synopsis: [PATCH] tmpfs mmap synchronization bug
New Synopsis: [tmpfs] [patch] tmpfs mmap synchronization bug

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Fri Oct 2 21:54:26 UTC 2009
Responsible-Changed-Why: 

http://www.freebsd.org/cgi/query-pr.cgi?pr=139312
From gleb.kurtsou at gmail.com  Fri Oct  2 22:23:29 2009
From: gleb.kurtsou at gmail.com (Gleb Kurtsou)
Date: Fri Oct  2 22:23:35 2009
Subject: kern/122038: [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp:
 type 0xc7d2fab0 0
In-Reply-To: <200908121810.n7CIA8Qv058688@freefall.freebsd.org>
References: <200908121810.n7CIA8Qv058688@freefall.freebsd.org>
Message-ID: <20091002222306.GA1729@tops>

Could you test following patch. I think it should fix the issue, but it
seems locking for tn_parent field is missing in some places. It needs a
closer look and more thorough testing.

-------------- next part --------------
diff --git a/sys/fs/tmpfs/tmpfs_subr.c b/sys/fs/tmpfs/tmpfs_subr.c
index dad634e..2d28058 100644
--- a/sys/fs/tmpfs/tmpfs_subr.c
+++ b/sys/fs/tmpfs/tmpfs_subr.c
@@ -375,6 +375,7 @@ loop:
 		vp->v_op = &tmpfs_fifoop_entries;
 		break;
 	case VDIR:
+		MPASS(node->tn_dir.tn_parent != NULL);
 		if (node->tn_dir.tn_parent == node)
 			vp->v_vflag |= VV_ROOT;
 		break;
@@ -653,6 +654,9 @@ tmpfs_dir_getdotdotdent(struct tmpfs_node *node, struct uio *uio)
 	TMPFS_VALIDATE_DIR(node);
 	MPASS(uio->uio_offset == TMPFS_DIRCOOKIE_DOTDOT);
 
+	if (node->tn_dir.tn_parent == NULL)
+		return ENOENT;
+
 	dent.d_fileno = node->tn_dir.tn_parent->tn_id;
 	dent.d_type = DT_DIR;
 	dent.d_namlen = 2;
diff --git a/sys/fs/tmpfs/tmpfs_vnops.c b/sys/fs/tmpfs/tmpfs_vnops.c
index db8ceea..7caac14 100644
--- a/sys/fs/tmpfs/tmpfs_vnops.c
+++ b/sys/fs/tmpfs/tmpfs_vnops.c
@@ -88,6 +88,10 @@ tmpfs_lookup(struct vop_cachedlookup_args *v)
 	if (cnp->cn_flags & ISDOTDOT) {
 		int ltype = 0;
 
+		if (dnode->tn_dir.tn_parent == NULL) {
+			error = ENOENT;
+			goto out;
+		}
 		ltype = VOP_ISLOCKED(dvp);
 		vhold(dvp);
 		VOP_UNLOCK(dvp, 0);
@@ -98,6 +102,10 @@ tmpfs_lookup(struct vop_cachedlookup_args *v)
 		vn_lock(dvp, ltype | LK_RETRY);
 		vdrop(dvp);
 	} else if (cnp->cn_namelen == 1 && cnp->cn_nameptr[0] == '.') {
+		if (dnode->tn_dir.tn_parent == NULL) {
+			error = ENOENT;
+			goto out;
+		}
 		VREF(dvp);
 		*vpp = dvp;
 		error = 0;
@@ -959,7 +967,8 @@ tmpfs_rename(struct vop_rename_args *v)
 			 * with stale nodes. */
 			n = tdnode;
 			while (n != n->tn_dir.tn_parent) {
-				if (n == fnode) {
+				MPASS(n->tn_dir.tn_parent != NULL);
+				if (n == fnode || n->tn_dir.tn_parent == NULL) {
 					error = EINVAL;
 					if (newname != NULL)
 						    free(newname, M_TMPFSNAME);
@@ -1112,6 +1121,7 @@ tmpfs_rmdir(struct vop_rmdir_args *v)
 	node->tn_dir.tn_parent->tn_links--;
 	node->tn_dir.tn_parent->tn_status |= TMPFS_NODE_ACCESSED | \
 	    TMPFS_NODE_CHANGED | TMPFS_NODE_MODIFIED;
+	node->tn_dir.tn_parent = NULL;
 
 	cache_purge(dvp);
 	cache_purge(vp);
From gleb.kurtsou at gmail.com  Fri Oct  2 22:30:04 2009
From: gleb.kurtsou at gmail.com (Gleb Kurtsou)
Date: Fri Oct  2 22:30:10 2009
Subject: kern/122038: [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp:
 type 0xc7d2fab0 0
Message-ID: <200910022230.n92MU348038447@freefall.freebsd.org>

The following reply was made to PR kern/122038; it has been noted by GNATS.

From: Gleb Kurtsou <gleb.kurtsou@gmail.com>
To: bug-followup@FreeBSD.org, delphij@FreeBSD.org, gprspb@mail.ru
Cc: freebsd-fs@FreeBSD.org
Subject: Re: kern/122038: [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp:
 type 0xc7d2fab0 0
Date: Sat, 3 Oct 2009 01:23:06 +0300

 --5vNYLRcllDrimb99
 Content-Type: text/plain; charset=utf-8
 Content-Disposition: inline
 
 Could you test following patch. I think it should fix the issue, but it
 seems locking for tn_parent field is missing in some places. It needs a
 closer look and more thorough testing.
 
 
 --5vNYLRcllDrimb99
 Content-Type: text/plain; charset=utf-8
 Content-Disposition: attachment; filename="tmpfs-rmparent.patch.txt"
 
 diff --git a/sys/fs/tmpfs/tmpfs_subr.c b/sys/fs/tmpfs/tmpfs_subr.c
 index dad634e..2d28058 100644
 --- a/sys/fs/tmpfs/tmpfs_subr.c
 +++ b/sys/fs/tmpfs/tmpfs_subr.c
 @@ -375,6 +375,7 @@ loop:
  		vp->v_op = &tmpfs_fifoop_entries;
  		break;
  	case VDIR:
 +		MPASS(node->tn_dir.tn_parent != NULL);
  		if (node->tn_dir.tn_parent == node)
  			vp->v_vflag |= VV_ROOT;
  		break;
 @@ -653,6 +654,9 @@ tmpfs_dir_getdotdotdent(struct tmpfs_node *node, struct uio *uio)
  	TMPFS_VALIDATE_DIR(node);
  	MPASS(uio->uio_offset == TMPFS_DIRCOOKIE_DOTDOT);
  
 +	if (node->tn_dir.tn_parent == NULL)
 +		return ENOENT;
 +
  	dent.d_fileno = node->tn_dir.tn_parent->tn_id;
  	dent.d_type = DT_DIR;
  	dent.d_namlen = 2;
 diff --git a/sys/fs/tmpfs/tmpfs_vnops.c b/sys/fs/tmpfs/tmpfs_vnops.c
 index db8ceea..7caac14 100644
 --- a/sys/fs/tmpfs/tmpfs_vnops.c
 +++ b/sys/fs/tmpfs/tmpfs_vnops.c
 @@ -88,6 +88,10 @@ tmpfs_lookup(struct vop_cachedlookup_args *v)
  	if (cnp->cn_flags & ISDOTDOT) {
  		int ltype = 0;
  
 +		if (dnode->tn_dir.tn_parent == NULL) {
 +			error = ENOENT;
 +			goto out;
 +		}
  		ltype = VOP_ISLOCKED(dvp);
  		vhold(dvp);
  		VOP_UNLOCK(dvp, 0);
 @@ -98,6 +102,10 @@ tmpfs_lookup(struct vop_cachedlookup_args *v)
  		vn_lock(dvp, ltype | LK_RETRY);
  		vdrop(dvp);
  	} else if (cnp->cn_namelen == 1 && cnp->cn_nameptr[0] == '.') {
 +		if (dnode->tn_dir.tn_parent == NULL) {
 +			error = ENOENT;
 +			goto out;
 +		}
  		VREF(dvp);
  		*vpp = dvp;
  		error = 0;
 @@ -959,7 +967,8 @@ tmpfs_rename(struct vop_rename_args *v)
  			 * with stale nodes. */
  			n = tdnode;
  			while (n != n->tn_dir.tn_parent) {
 -				if (n == fnode) {
 +				MPASS(n->tn_dir.tn_parent != NULL);
 +				if (n == fnode || n->tn_dir.tn_parent == NULL) {
  					error = EINVAL;
  					if (newname != NULL)
  						    free(newname, M_TMPFSNAME);
 @@ -1112,6 +1121,7 @@ tmpfs_rmdir(struct vop_rmdir_args *v)
  	node->tn_dir.tn_parent->tn_links--;
  	node->tn_dir.tn_parent->tn_status |= TMPFS_NODE_ACCESSED | \
  	    TMPFS_NODE_CHANGED | TMPFS_NODE_MODIFIED;
 +	node->tn_dir.tn_parent = NULL;
  
  	cache_purge(dvp);
  	cache_purge(vp);
 
 --5vNYLRcllDrimb99--
From fbsdlist at src.cx  Fri Oct  2 23:38:26 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Fri Oct  2 23:38:32 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <20091002184526.GA1660@garage.freebsd.pl>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
Message-ID: <ed91d4a80910021638r72213389uf0767b9bb523b7d9@mail.gmail.com>

With the patch, if vfs.zfs.arc_min is set high enough, the system locks up.

On a box with 8G or RAM I had arc_min=6G and arc_max=7G. Once ARC grew
to ~5.8G as reported by kstat.zfs.misc.arcstats.size, number of wired
pages grew to ~7400MB and the processes got stuck in 'vmwait' state. I
had to reboot in order to recover.

On one hand setting arc_min can be considered a pilot error. On the
other, it may be a good idea to allow system to reclaim memory from
ARC even if ARC is smaller than arc_min if the system really really
needs it. The question is how to define "really needs it".

On a side note, it appears that wired page count tends to be
substantially larger than ARC size. I.e. in my case if ARC size grows
to 6G, wired page count is about 1.5G bigger. Perhaps we should allow
reclaiming memory

--Artem



On Fri, Oct 2, 2009 at 11:45 AM, Pawel Jakub Dawidek <pjd@freebsd.org> wrote:
> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>> Backing out this change from the 8-STABLE kernel:
>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
>>
>> makes it survive about half and hour of IMAP searching. Of course only
>> time will tell whether this helps in the long run, but so far 10/10
>> tries succeeded to kill the machine with this method...
>
> Could you try this patch:
>
> ? ? ? ?http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>
> --
> Pawel Jakub Dawidek ? ? ? ? ? ? ? ? ? ? ? http://www.wheel.pl
> pjd@FreeBSD.org ? ? ? ? ? ? ? ? ? ? ? ? ? http://www.FreeBSD.org
> FreeBSD committer ? ? ? ? ? ? ? ? ? ? ? ? Am I Evil? Yes, I Am!
>
From pjd at FreeBSD.org  Sat Oct  3 00:09:19 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Sat Oct  3 00:09:32 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <ed91d4a80910021638r72213389uf0767b9bb523b7d9@mail.gmail.com>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<ed91d4a80910021638r72213389uf0767b9bb523b7d9@mail.gmail.com>
Message-ID: <20091003000909.GD1660@garage.freebsd.pl>

On Fri, Oct 02, 2009 at 04:38:24PM -0700, Artem Belevich wrote:
> With the patch, if vfs.zfs.arc_min is set high enough, the system locks up.
> 
> On a box with 8G or RAM I had arc_min=6G and arc_max=7G. Once ARC grew
> to ~5.8G as reported by kstat.zfs.misc.arcstats.size, number of wired
> pages grew to ~7400MB and the processes got stuck in 'vmwait' state. I
> had to reboot in order to recover.
> 
> On one hand setting arc_min can be considered a pilot error. On the
> other, it may be a good idea to allow system to reclaim memory from
> ARC even if ARC is smaller than arc_min if the system really really
> needs it. The question is how to define "really needs it".
> 
> On a side note, it appears that wired page count tends to be
> substantially larger than ARC size. I.e. in my case if ARC size grows
> to 6G, wired page count is about 1.5G bigger. Perhaps we should allow
> reclaiming memory

Before we start debuging pathological cases, could you try the patch
with defaul settings? Eventually with vm.kmem_size set to the amount of
RAM you have.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091003/2f48565b/attachment.pgp
From fbsdlist at src.cx  Sat Oct  3 00:59:37 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Sat Oct  3 00:59:43 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <20091003000909.GD1660@garage.freebsd.pl>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<ed91d4a80910021638r72213389uf0767b9bb523b7d9@mail.gmail.com>
	<20091003000909.GD1660@garage.freebsd.pl>
Message-ID: <ed91d4a80910021759h4a3d3b9csde86c59075795eeb@mail.gmail.com>

> Before we start debuging pathological cases, could you try the patch
> with defaul settings? Eventually with vm.kmem_size set to the amount of
> RAM you have.

System runs stable/8 r197716 on 4-core amd64 with 8G of RAM.

With default /boot/loader.conf. Kernel comes up with following parameters:

vm.kmem_size: 2764533760
vfs.zfs.arc_min: 215979200
vfs.zfs.arc_max: 1727833600

Under load ARC size reaches ~1.7G. At that time top reports:
Mem: 47M Active, 11M Inact, 2158M Wired, 268K Cache, 21M Buf, 5693M Free

However, as the FS load continues, ARC size, stays at 1.7G for couple
of minutes, then shrinks down to 1.2G, then slowly grows to 1.7G,
stays there for a little and then the shrink/grow cycle repeats.
Throughout the test there's always ~5G of *free* memory.

===============================================================
Now, the same experiment, with vm.kmem_size=8G
vm.kmem_size: 8589934592
vfs.zfs.arc_min: 939524096
vfs.zfs.arc_max: 7516192768

ARC grows to 6.2G:
Mem: 47M Active, 13M Inact, 7376M Wired, 31M Buf, 473M Free

Then it quickly shrinks to 4.6G and grows to 6.2G again, shrinks again, etc..

What's different from the previous case is that after a while ZFS
adjusts target size (kstat.zfs.misc.arcstats.c) down to ~5.8G and
after that ZFS size oscillates between 4.2G and 5.6G. Another
observation -- ARC shrinking happens when system is left with ~512M of
free memory. Yet another observation is that even with ARC peak of
~5.8G, system has about 7.5G wired. Where did almost 2G of difference
go? Fragmentation?

I've tried both experiments with and without L2ARC -- behavior seems
to be the same.

--Artem
From gleb.kurtsou at gmail.com  Sat Oct  3 09:30:04 2009
From: gleb.kurtsou at gmail.com (Gleb Kurtsou)
Date: Sat Oct  3 09:30:11 2009
Subject: kern/138367: [tmpfs] [panic] 'panic: Assertion pages > 0
 failed' when running regression/tmpfs
Message-ID: <200910030930.n939U3kc077250@freefall.freebsd.org>

The following reply was made to PR kern/138367; it has been noted by GNATS.

From: Gleb Kurtsou <gleb.kurtsou@gmail.com>
To: bug-followup@FreeBSD.org, bruce@cran.org.uk
Cc:  
Subject: Re: kern/138367: [tmpfs] [panic] 'panic: Assertion pages > 0
 failed' when running regression/tmpfs
Date: Sat, 3 Oct 2009 12:21:08 +0300

 --7AUc2qLy4jB3hD7Z
 Content-Type: text/plain; charset=utf-8
 Content-Disposition: inline
 
 I wasn't able to trigger it on amd64, but there were several integer
 overflow bugs.
 
 Besides there was inconsistency in setting max values. Max pages was set
 to SIZE_MAX (if no value provided by user), but max file size depended
 on available swap/memory at the moment of mounting filesystem. I've set
 max file size to 4GB (of memory limit set by user). It can be changed to
 uint64_t max, but using 4GB seems to be sufficient limit to prevent
 resource exhaustion.
 
 Would you try this patch, I have no i386 system running to test it.
 
 
 --7AUc2qLy4jB3hD7Z
 Content-Type: text/plain; charset=utf-8
 Content-Disposition: attachment; filename="tmpfs-intoverflow.patch.txt"
 
 diff --git a/sys/fs/tmpfs/tmpfs.h b/sys/fs/tmpfs/tmpfs.h
 index ffd705f..42a0e5d 100644
 --- a/sys/fs/tmpfs/tmpfs.h
 +++ b/sys/fs/tmpfs/tmpfs.h
 @@ -470,6 +470,11 @@ int	tmpfs_truncate(struct vnode *, off_t);
  #define TMPFS_PAGES_RESERVED (4 * 1024 * 1024 / PAGE_SIZE)
  
  /*
 + * Set maximum file size to 4 GB.
 + */
 +#define TMPFS_MAXFILESIZE UINT_MAX
 +
 +/*
   * Returns information about the number of available memory pages,
   * including physical and virtual ones.
   *
 diff --git a/sys/fs/tmpfs/tmpfs_vfsops.c b/sys/fs/tmpfs/tmpfs_vfsops.c
 index f0ae6be..cdefaba 100644
 --- a/sys/fs/tmpfs/tmpfs_vfsops.c
 +++ b/sys/fs/tmpfs/tmpfs_vfsops.c
 @@ -82,57 +82,6 @@ static const char *tmpfs_opts[] = {
  };
  
  /* --------------------------------------------------------------------- */
 -
 -#define SWI_MAXMIB	3
 -
 -static u_int
 -get_swpgtotal(void)
 -{
 -	struct xswdev xsd;
 -	char *sname = "vm.swap_info";
 -	int soid[SWI_MAXMIB], oid[2];
 -	u_int unswdev, total, dmmax, nswapdev;
 -	size_t mibi, len;
 -
 -	total = 0;
 -
 -	len = sizeof(dmmax);
 -	if (kernel_sysctlbyname(curthread, "vm.dmmax", &dmmax, &len,
 -				NULL, 0, NULL, 0) != 0)
 -		return total;
 -
 -	len = sizeof(nswapdev);
 -	if (kernel_sysctlbyname(curthread, "vm.nswapdev",
 -				&nswapdev, &len,
 -				NULL, 0, NULL, 0) != 0)
 -		return total;
 -
 -	mibi = (SWI_MAXMIB - 1) * sizeof(int);
 -	oid[0] = 0;
 -	oid[1] = 3;
 -
 -	if (kernel_sysctl(curthread, oid, 2,
 -			soid, &mibi, (void *)sname, strlen(sname),
 -			NULL, 0) != 0)
 -		return total;
 -
 -	mibi = (SWI_MAXMIB - 1);
 -	for (unswdev = 0; unswdev < nswapdev; ++unswdev) {
 -		soid[mibi] = unswdev;
 -		len = sizeof(struct xswdev);
 -		if (kernel_sysctl(curthread,
 -				soid, mibi + 1, &xsd, &len, NULL, 0,
 -				NULL, 0) != 0)
 -			return total;
 -		if (len == sizeof(struct xswdev))
 -			total += (xsd.xsw_nblks - dmmax);
 -	}
 -
 -	/* Not Reached */
 -	return total;
 -}
 -
 -/* --------------------------------------------------------------------- */
  static int
  tmpfs_node_ctor(void *mem, int size, void *arg, int flags)
  {
 @@ -186,7 +135,7 @@ tmpfs_mount(struct mount *mp)
  	int error;
  	/* Size counters. */
  	ino_t	nodes_max;
 -	size_t	size_max;
 +	off_t	size_max;
  
  	/* Root node attributes. */
  	uid_t	root_uid;
 @@ -230,8 +179,7 @@ tmpfs_mount(struct mount *mp)
  
  	/* Do not allow mounts if we do not have enough memory to preserve
  	 * the minimum reserved pages. */
 -	mem_size = cnt.v_free_count + cnt.v_inactive_count + get_swpgtotal();
 -	mem_size -= mem_size > cnt.v_wire_count ? cnt.v_wire_count : mem_size;
 +	mem_size = tmpfs_mem_info();
  	if (mem_size < TMPFS_PAGES_RESERVED)
  		return ENOSPC;
  
 @@ -239,14 +187,17 @@ tmpfs_mount(struct mount *mp)
  	 * allowed to use, based on the maximum size the user passed in
  	 * the mount structure.  A value of zero is treated as if the
  	 * maximum available space was requested. */
 -	if (size_max < PAGE_SIZE || size_max >= SIZE_MAX)
 -		pages = SIZE_MAX;
 +	/* XXX Choose maximum values to prevent integer overflow */
 +	if (size_max < PAGE_SIZE || size_max > SSIZE_MAX - PAGE_SIZE)
 +		pages = SSIZE_MAX - PAGE_SIZE;
  	else
  		pages = howmany(size_max, PAGE_SIZE);
  	MPASS(pages > 0);
  
 +	CTASSERT(sizeof(ino_t) == sizeof(uint32_t));
 +	/* Set maximum node number to 2GB to prevent integer overflow. */
  	if (nodes_max <= 3)
 -		nodes = 3 + pages * PAGE_SIZE / 1024;
 +		nodes = qmin(pages + 3, INT_MAX);
  	else
  		nodes = nodes_max;
  	MPASS(nodes >= 3);
 @@ -258,7 +209,11 @@ tmpfs_mount(struct mount *mp)
  	mtx_init(&tmp->allnode_lock, "tmpfs allnode lock", NULL, MTX_DEF);
  	tmp->tm_nodes_max = nodes;
  	tmp->tm_nodes_inuse = 0;
 -	tmp->tm_maxfilesize = (u_int64_t)(cnt.v_page_count + get_swpgtotal()) * PAGE_SIZE;
 +	if ((u_int64_t)pages < (OFF_MAX >> PAGE_SHIFT))
 +		tmp->tm_maxfilesize = qmin((u_int64_t)(pages) * PAGE_SIZE,
 +		    TMPFS_MAXFILESIZE);
 +	else
 +		tmp->tm_maxfilesize = TMPFS_MAXFILESIZE;
  	LIST_INIT(&tmp->tm_nodes_used);
  
  	tmp->tm_pages_max = pages;
 
 --7AUc2qLy4jB3hD7Z--
From alex at trull.org  Sat Oct  3 13:10:51 2009
From: alex at trull.org (Alex Trull)
Date: Sat Oct  3 13:10:58 2009
Subject: ARC & L2ARC efficiency
In-Reply-To: <ed91d4a80910011620y73f931c9ie1e41692d6fc46e3@mail.gmail.com>
References: <8c9ae7950910011322j1a6b66fcp73615cc17ae20328@mail.gmail.com>
	<ed91d4a80910011600x3f3e855etd24694bde69a96c7@mail.gmail.com>
	<ed91d4a80910011620y73f931c9ie1e41692d6fc46e3@mail.gmail.com>
Message-ID: <1254574435.7247.15.camel@porksoda.turandot.home>

I went straight for munin since I like to track these stats long term
along with everything else going on - there are some older scripts I put
on muninexchange but since zfs13 came to 7. I split and updated them to
handle the newer counters including l2arc:

scripts zfs-* :
http://trull.org/~alex/src/FreeBSD/muninplugins/

example graphs (forgive the amazingly redundant path) :
http://web.internationalconspiracy.org/munin/internationalconspiracy.org/potjie.internationalconspiracy.org.html#Filesystem

If you want to see more short term performance stats with a while
running a filesystem benchmark test (or just under normal load) I
recommend 'zpool iostat -v poolname N' where poolname is obvious and N
is the number of seconds for refresh. This not only shows IO read and
write but also the throughput per disk, including the l2arc cache.

--
Alex

On Thu, 2009-10-01 at 16:20 -0700, Artem Belevich wrote:
> Here's another script:
> http://www.solarisinternals.com/wiki/index.php/Arcstat
> 
> Attached hacked FreeBSD version.
> 
> --Artem
> 
> 
> 
> On Thu, Oct 1, 2009 at 4:00 PM, Artem Belevich <fbsdlist@src.cx> wrote:
> > There's a pretty useful script to present ARC stats (alas, L2ARC info
> > is not included) in a readable way:
> > http://cuddletech.com/arc_summary/
> >
> > I've attaches somewhat hacked (and a bit outdated) version that runs on FreeBSD.
> >
> > --Artem
> >
> >
> >
> > On Thu, Oct 1, 2009 at 1:22 PM, Alexander Shevchenko <pepelac@gmail.com> wrote:
> >> Good time of day!
> >>
> >> How could i check the efficiency of ARC?
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091003/784e79c0/attachment.pgp
From patpro at patpro.net  Sun Oct  4 08:45:15 2009
From: patpro at patpro.net (Patrick Proniewski)
Date: Sun Oct  4 08:45:21 2009
Subject: Offline uncorrectable sectors on hard drive
Message-ID: <C09E0617-F326-4AB1-BFF5-6814CD084940@patpro.net>

Hi all,

smartctl on my server returned an error yesterday :

> The following warning/error was logged by the smartd daemon:
>
> Device: /dev/ad6, 1 Offline uncorrectable sectors

/var/log/messages :

> Oct  3 04:04:19 rack smartd[739]: Device: /dev/ad6, 1 Offline  
> uncorrectable sectors
> Oct  3 04:04:19 rack smartd[739]: Device: /dev/ad6, Self-Test Log  
> error count increased from 0 to 1
../..
> Oct  4 01:34:19 rack smartd[739]: Device: /dev/ad6, 1 Offline  
> uncorrectable sectors
> Oct  4 02:04:19 rack smartd[739]: Device: /dev/ad6, 1 Offline  
> uncorrectable sectors

first error flagged Oct 3 04:04:19, last error Oct 4 02:04:19. Since  
then, no more error in the logs.

So my questions are:

- is that "Offline uncorrectable sectors" in fact correctable? I've  
found howto's for extfs and reiserfs to correct this kind of error by  
remapping bad sectors, but nothing for UFS.
- this error appears to have disappeared, does it mean the harddrive  
(or the fs) made the remapping by it self?

Here is the smartctl -a output for this device:

> # smartctl -a /dev/ad6
> smartctl version 5.38 [i386-portbld-freebsd6.4] Copyright (C) 2002-8  
> Bruce Allen
> Home page is http://smartmontools.sourceforge.net/
>
> === START OF INFORMATION SECTION ===
> Model Family:     Maxtor DiamondMax 10 family (ATA/133 and SATA/150)
> Device Model:     Maxtor 6L200M0
> Serial Number:    L404EDDH
> Firmware Version: BANC1E00
> User Capacity:    203,928,109,056 bytes
> Device is:        In smartctl database [for details use: -P show]
> ATA Version is:   7
> ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 0
> Local Time is:    Sun Oct  4 10:23:02 2009 CEST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> Warning! SMART Attribute Thresholds Structure error: invalid SMART  
> checksum.
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status:  (0x02)	Offline data collection  
> activity
> 					was completed without error.
> 					Auto Offline Data Collection: Disabled.
> Self-test execution status:      (   0)	The previous self-test  
> routine completed
> 					without error or no self-test has ever
> 					been run.
> Total time to complete Offline
> data collection: 		 (1562) seconds.
> Offline data collection
> capabilities: 			 (0x5b) SMART execute Offline immediate.
> 					Auto Offline data collection on/off support.
> 					Suspend Offline collection upon new
> 					command.
> 					Offline surface scan supported.
> 					Self-test supported.
> 					No Conveyance Self-test supported.
> 					Selective Self-test supported.
> SMART capabilities:            (0x0003)	Saves SMART data before  
> entering
> 					power-saving mode.
> 					Supports SMART auto save timer.
> Error logging capability:        (0x01)	Error logging supported.
> 					General Purpose Logging supported.
> Short self-test routine
> recommended polling time: 	 (   2) minutes.
> Extended self-test routine
> recommended polling time: 	 (  81) minutes.
> SCT capabilities: 	       (0x0021)	SCT Status supported.
> 					SCT Data Table supported.
>
> SMART Attributes Data Structure revision number: 16
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE       
> UPDATED  WHEN_FAILED RAW_VALUE
>   3 Spin_Up_Time            0x0027   252   252   063    Pre-fail   
> Always       -       3148
>   4 Start_Stop_Count        0x0032   253   253   000    Old_age    
> Always       -       17
>   5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail   
> Always       -       0
>   6 Read_Channel_Margin     0x0001   253   253   100    Pre-fail   
> Offline      -       0
>   7 Seek_Error_Rate         0x000a   253   252   000    Old_age    
> Always       -       0
>   8 Seek_Time_Performance   0x0027   250   234   187    Pre-fail   
> Always       -       51522
>   9 Power_On_Minutes        0x0032   154   154   000    Old_age    
> Always       -       487h+21m
>  10 Spin_Retry_Count        0x002b   252   252   157    Pre-fail   
> Always       -       0
>  11 Calibration_Retry_Count 0x002b   253   252   223    Pre-fail   
> Always       -       0
>  12 Power_Cycle_Count       0x0032   253   253   000    Old_age    
> Always       -       20
> 192 Power-Off_Retract_Count 0x0032   253   253   000    Old_age    
> Always       -       0
> 193 Load_Cycle_Count        0x0032   253   253   000    Old_age    
> Always       -       0
> 194 Temperature_Celsius     0x0032   023   253   000    Old_age    
> Always       -       26
> 195 Hardware_ECC_Recovered  0x000a   253   252   000    Old_age    
> Always       -       7879
> 196 Reallocated_Event_Count 0x0008   251   251   000    Old_age    
> Offline      -       2
> 197 Current_Pending_Sector  0x0008   253   253   000    Old_age    
> Offline      -       0
> 198 Offline_Uncorrectable   0x0008   253   252   000    Old_age    
> Offline      -       0
> 199 UDMA_CRC_Error_Count    0x0008   199   199   000    Old_age    
> Offline      -       0
> 200 Multi_Zone_Error_Rate   0x000a   253   252   000    Old_age    
> Always       -       0
> 201 Soft_Read_Error_Rate    0x000a   253   252   000    Old_age    
> Always       -       0
> 202 TA_Increase_Count       0x000a   253   252   000    Old_age    
> Always       -       0
> 203 Run_Out_Cancel          0x000b   253   252   180    Pre-fail   
> Always       -       0
> 204 Shock_Count_Write_Opern 0x000a   253   252   000    Old_age    
> Always       -       0
> 205 Shock_Rate_Write_Opern  0x000a   253   252   000    Old_age    
> Always       -       0
> 207 Spin_High_Current       0x002a   252   252   000    Old_age    
> Always       -       0
> 208 Spin_Buzz               0x002a   252   252   000    Old_age    
> Always       -       0
> 209 Offline_Seek_Performnce 0x0024   239   239   000    Old_age    
> Offline      -       170
> 210 Unknown_Attribute       0x0032   253   252   000    Old_age    
> Always       -       0
> 211 Unknown_Attribute       0x0032   253   252   000    Old_age    
> Always       -       0
> 212 Unknown_Attribute       0x0032   253   252   000    Old_age    
> Always       -       0
>
> Warning! SMART ATA Error Log Structure error: invalid SMART checksum.
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num  Test_Description    Status                  Remaining   
> LifeTime(hours)  LBA_of_first_error
> # 1  Short offline       Completed without error       00%      
> 34339         -
> # 2  Extended offline    Completed: read failure       20%      
> 34317         201626851
> # 3  Short offline       Completed without error       00%      
> 34315         -
> # 4  Short offline       Completed without error       00%      
> 34291         -
> # 5  Short offline       Completed without error       00%      
> 34267         -
> # 6  Short offline       Completed without error       00%      
> 34243         -
> # 7  Short offline       Completed without error       00%      
> 34220         -
> # 8  Short offline       Completed without error       00%      
> 34196         -
> # 9  Short offline       Completed without error       00%      
> 34172         -
> #10  Extended offline    Completed without error       00%      
> 34150         -
> #11  Short offline       Completed without error       00%      
> 34148         -
> #12  Short offline       Completed without error       00%      
> 34124         -
> #13  Short offline       Completed without error       00%      
> 34100         -
> #14  Short offline       Completed without error       00%      
> 34076         -
> #15  Short offline       Completed without error       00%      
> 34053         -
> #16  Short offline       Completed without error       00%      
> 34029         -
> #17  Short offline       Completed without error       00%      
> 34005         -
> #18  Extended offline    Completed without error       00%      
> 33983         -
> #19  Short offline       Completed without error       00%      
> 33981         -
> #20  Short offline       Completed without error       00%      
> 33957         -
> #21  Short offline       Completed without error       00%      
> 33933         -
>
> SMART Selective self-test log data structure revision number 1
>  SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
>     1        0        0  Not_testing
>     2        0        0  Not_testing
>     3        0        0  Not_testing
>     4        0        0  Not_testing
>     5        0        0  Not_testing
> Selective self-test flags (0x0):
>   After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute  
> delay.


thanks,

patpro

From michael at fuckner.net  Sun Oct  4 08:50:20 2009
From: michael at fuckner.net (Michael Fuckner)
Date: Sun Oct  4 08:50:27 2009
Subject: Offline uncorrectable sectors on hard drive
In-Reply-To: <C09E0617-F326-4AB1-BFF5-6814CD084940@patpro.net>
References: <C09E0617-F326-4AB1-BFF5-6814CD084940@patpro.net>
Message-ID: <4AC861BF.20109@fuckner.net>

Patrick Proniewski wrote:
> Hi all,
Hi Patrick,

> ../..
>> Oct  4 01:34:19 rack smartd[739]: Device: /dev/ad6, 1 Offline 
>> uncorrectable sectors
>> Oct  4 02:04:19 rack smartd[739]: Device: /dev/ad6, 1 Offline 
>> uncorrectable sectors
> 
> first error flagged Oct 3 04:04:19, last error Oct 4 02:04:19. Since 
> then, no more error in the logs.
This looks like a bad drive


> So my questions are:
> 
> - is that "Offline uncorrectable sectors" in fact correctable? I've 
> found howto's for extfs and reiserfs to correct this kind of error by 
> remapping bad sectors, but nothing for UFS.
> - this error appears to have disappeared, does it mean the harddrive (or 
> the fs) made the remapping by it self?
sector errors are on the hardware, they don't deal with the filesystem.

Perhaps it was a once in a lifetime event, perhaps the drive is going to 
die really soon.

Please do a smartctl -t long and see if the drive has to be replaced.

Regards,
  Michael!
From delphij at FreeBSD.org  Sun Oct  4 10:34:20 2009
From: delphij at FreeBSD.org (delphij@FreeBSD.org)
Date: Sun Oct  4 10:34:27 2009
Subject: kern/139312: [tmpfs] [patch] tmpfs mmap synchronization bug
Message-ID: <200910041034.n94AYJJw080881@freefall.freebsd.org>

Synopsis: [tmpfs] [patch] tmpfs mmap synchronization bug

Responsible-Changed-From-To: freebsd-fs->delphij
Responsible-Changed-By: delphij
Responsible-Changed-When: Sun Oct 4 10:34:04 UTC 2009
Responsible-Changed-Why: 
Take.

http://www.freebsd.org/cgi/query-pr.cgi?pr=139312
From patpro at patpro.net  Sun Oct  4 11:22:26 2009
From: patpro at patpro.net (Patrick Proniewski)
Date: Sun Oct  4 11:22:32 2009
Subject: Offline uncorrectable sectors on hard drive
In-Reply-To: <4AC861BF.20109@fuckner.net>
References: <C09E0617-F326-4AB1-BFF5-6814CD084940@patpro.net>
	<4AC861BF.20109@fuckner.net>
Message-ID: <5AFF10D8-2B03-4DE8-83E8-20B6A832BBA8@patpro.net>

Hi Michael,

On 4 oct. 09, at 10:50, Michael Fuckner wrote:

>> - is that "Offline uncorrectable sectors" in fact correctable? I've  
>> found howto's for extfs and reiserfs to correct this kind of error  
>> by remapping bad sectors, but nothing for UFS.
>> - this error appears to have disappeared, does it mean the  
>> harddrive (or the fs) made the remapping by it self?
> sector errors are on the hardware, they don't deal with the  
> filesystem.
>
> Perhaps it was a once in a lifetime event, perhaps the drive is  
> going to die really soon.
>
> Please do a smartctl -t long and see if the drive has to be replaced.


here is the result (from `smartctl -l selftest /dev/ad6` after the  
`smartctl -t long /dev/ad6` ):

Warning! SMART Attribute Thresholds Structure error: invalid SMART  
checksum.
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining   
LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      
34350         -
# 2  Short offline       Completed without error       00%      
34339         -
# 3  Extended offline    Completed: read failure       20%      
34317         201626851
# 4  Short offline       Completed without error       00%      
34315         -
# 5  Short offline       Completed without error       00%      
34291         -
# 6  Short offline       Completed without error       00%      
34267         -
# 7  Short offline       Completed without error       00%      
34243         -
# 8  Short offline       Completed without error       00%      
34220         -
# 9  Short offline       Completed without error       00%      
34196         -
#10  Short offline       Completed without error       00%      
34172         -
#11  Extended offline    Completed without error       00%      
34150         -
#12  Short offline       Completed without error       00%      
34148         -
#13  Short offline       Completed without error       00%      
34124         -
#14  Short offline       Completed without error       00%      
34100         -
#15  Short offline       Completed without error       00%      
34076         -
#16  Short offline       Completed without error       00%      
34053         -
#17  Short offline       Completed without error       00%      
34029         -
#18  Short offline       Completed without error       00%      
34005         -
#19  Extended offline    Completed without error       00%      
33983         -
#20  Short offline       Completed without error       00%      
33981         -
#21  Short offline       Completed without error       00%      
33957         -

It looks like the error is gone, but may be I'm wrong.

regards,
patpro

From 000.fbsd at quip.cz  Sun Oct  4 11:43:00 2009
From: 000.fbsd at quip.cz (Miroslav Lachman)
Date: Sun Oct  4 11:43:08 2009
Subject: Offline uncorrectable sectors on hard drive
In-Reply-To: <5AFF10D8-2B03-4DE8-83E8-20B6A832BBA8@patpro.net>
References: <C09E0617-F326-4AB1-BFF5-6814CD084940@patpro.net>	<4AC861BF.20109@fuckner.net>
	<5AFF10D8-2B03-4DE8-83E8-20B6A832BBA8@patpro.net>
Message-ID: <4AC88A56.6080806@quip.cz>

Patrick Proniewski wrote:

> Hi Michael,
> 
> On 4 oct. 09, at 10:50, Michael Fuckner wrote:
> 
>>> - is that "Offline uncorrectable sectors" in fact correctable? I've  
>>> found howto's for extfs and reiserfs to correct this kind of error  
>>> by remapping bad sectors, but nothing for UFS.
>>> - this error appears to have disappeared, does it mean the  harddrive 
>>> (or the fs) made the remapping by it self?
>>
>> sector errors are on the hardware, they don't deal with the  filesystem.
>>
>> Perhaps it was a once in a lifetime event, perhaps the drive is  going 
>> to die really soon.
>>
>> Please do a smartctl -t long and see if the drive has to be replaced.
> 
> 
> 
> here is the result (from `smartctl -l selftest /dev/ad6` after the  
> `smartctl -t long /dev/ad6` ):
> 
> Warning! SMART Attribute Thresholds Structure error: invalid SMART  
> checksum.
> === START OF READ SMART DATA SECTION ===
> SMART Self-test log structure revision number 1
> Num  Test_Description    Status                  Remaining   
> LifeTime(hours)  LBA_of_first_error
> # 1  Extended offline    Completed without error       00%      
> 34350         -
> # 2  Short offline       Completed without error       00%      
> 34339         -
> # 3  Extended offline    Completed: read failure       20%      
> 34317         201626851
> # 4  Short offline       Completed without error       00%      
[...]
> It looks like the error is gone, but may be I'm wrong.

It is better to look at diff of Vendor Specific Attributes (smartctl -A)

You have Reallocated_Event_Count  2, so maybe bad sectors were reallocated.

Miroslav Lachman
From patpro at patpro.net  Sun Oct  4 14:50:21 2009
From: patpro at patpro.net (Patrick Proniewski)
Date: Sun Oct  4 14:50:27 2009
Subject: Offline uncorrectable sectors on hard drive
In-Reply-To: <4AC88A56.6080806@quip.cz>
References: <C09E0617-F326-4AB1-BFF5-6814CD084940@patpro.net>	<4AC861BF.20109@fuckner.net>
	<5AFF10D8-2B03-4DE8-83E8-20B6A832BBA8@patpro.net>
	<4AC88A56.6080806@quip.cz>
Message-ID: <8FF3DBC3-FB26-4C02-9135-79117659FCE7@patpro.net>

Miroslav,

On 4 oct. 09, at 13:43, Miroslav Lachman wrote:

>> It looks like the error is gone, but may be I'm wrong.
>
> It is better to look at diff of Vendor Specific Attributes (smartctl  
> -A)

here we are:

# smartctl -A /dev/ad6
smartctl version 5.38 [i386-portbld-freebsd6.4] Copyright (C) 2002-8  
Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Warning! SMART Attribute Thresholds Structure error: invalid SMART  
checksum.
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE       
UPDATED  WHEN_FAILED RAW_VALUE
   3 Spin_Up_Time            0x0027   252   252   063    Pre-fail   
Always       -       3148
   4 Start_Stop_Count        0x0032   253   253   000    Old_age    
Always       -       17
   5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail   
Always       -       0
   6 Read_Channel_Margin     0x0001   253   253   100    Pre-fail   
Offline      -       0
   7 Seek_Error_Rate         0x000a   253   252   000    Old_age    
Always       -       0
   8 Seek_Time_Performance   0x0027   252   234   187    Pre-fail   
Always       -       54404
   9 Power_On_Minutes        0x0032   154   154   000    Old_age    
Always       -       493h+40m
  10 Spin_Retry_Count        0x002b   252   252   157    Pre-fail   
Always       -       0
  11 Calibration_Retry_Count 0x002b   253   252   223    Pre-fail   
Always       -       0
  12 Power_Cycle_Count       0x0032   253   253   000    Old_age    
Always       -       20
192 Power-Off_Retract_Count 0x0032   253   253   000    Old_age    
Always       -       0
193 Load_Cycle_Count        0x0032   253   253   000    Old_age    
Always       -       0
194 Temperature_Celsius     0x0032   023   253   000    Old_age    
Always       -       25
195 Hardware_ECC_Recovered  0x000a   253   252   000    Old_age    
Always       -       5695
196 Reallocated_Event_Count 0x0008   251   251   000    Old_age    
Offline      -       2
197 Current_Pending_Sector  0x0008   253   253   000    Old_age    
Offline      -       0
198 Offline_Uncorrectable   0x0008   253   252   000    Old_age    
Offline      -       0
199 UDMA_CRC_Error_Count    0x0008   199   199   000    Old_age    
Offline      -       0
200 Multi_Zone_Error_Rate   0x000a   253   252   000    Old_age    
Always       -       0
201 Soft_Read_Error_Rate    0x000a   253   252   000    Old_age    
Always       -       0
202 TA_Increase_Count       0x000a   253   252   000    Old_age    
Always       -       0
203 Run_Out_Cancel          0x000b   253   252   180    Pre-fail   
Always       -       1
204 Shock_Count_Write_Opern 0x000a   253   252   000    Old_age    
Always       -       0
205 Shock_Rate_Write_Opern  0x000a   253   252   000    Old_age    
Always       -       0
207 Spin_High_Current       0x002a   252   252   000    Old_age    
Always       -       0
208 Spin_Buzz               0x002a   252   252   000    Old_age    
Always       -       0
209 Offline_Seek_Performnce 0x0024   239   239   000    Old_age    
Offline      -       170
210 Unknown_Attribute       0x0032   253   252   000    Old_age    
Always       -       0
211 Unknown_Attribute       0x0032   253   252   000    Old_age    
Always       -       0
212 Unknown_Attribute       0x0032   253   252   000    Old_age    
Always       -       0


thanks,
patpro

From pjd at FreeBSD.org  Sun Oct  4 17:48:00 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Sun Oct  4 17:48:07 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <683849754.20091001110503@pyro.de>
References: <683849754.20091001110503@pyro.de>
Message-ID: <20091004174746.GF1660@garage.freebsd.pl>

On Thu, Oct 01, 2009 at 11:05:03AM +0200, Solon Lutz wrote:
> Hi erverybody,
> 
> I'm faced with a 10TB ZFS pool on a 12TB RAID6 Areca controller.
> And yes, I know, you shouldn't put a zpool on a RAID-device... =(

Just to be sure: you have no redundancy on ZFS level at all? That's
very, very bad idea for important data (you know that already, but to
warn others)...

> The cable was replaced, a parity check was run on the RAID-Volume and
> showed no errors, the zfs scrub however showed some 'defective' files.
> After copying these files with 'dd -conv=noerror...' and comparing them
> to the originals, they were error-free.
> 
> Yesterday however, three more defective cables forced the controller
> to take the RAID6 volume offline. Now all cables were replaced and a parity
> check was run on the RAID-Volume -> data integrity OK.

This means absolutely nothing. It just means that parity match the
actual data, it doesn't mean the data is fine from file system or
application perspective.

> But now ZFS refuses to mount all volumes:
> 
> Solaris: WARNING: can't process intent log for temp/space1
> Solaris: WARNING: can't process intent log for temp/space2
> Solaris: WARNING: can't process intent log for temp/space3
> Solaris: WARNING: can't process intent log for temp/space4
> 
> A scrub revealed to following:
> 
> errors: Permanent errors have been detected in the following files:
> 
>         temp:<0x0>
>         temp/space1:<0x0>
>         temp/space2:<0x0>
>         temp/space3:<0x0>
>         temp/space4:<0x0>
> 
> 
> I tried to switch off checksums for this pool, but that didn't help in any
> way. I also mounted the pool by hand and was faced with with 'empty' volumes
> and 'I/O errors' when trying to list their contents...
> 
> Any suggestions? I'm offering some self-made blackberry jam and raspberry brandy
> to the person who can help to restore or backup the data.
> 
> Tech specs:
> 
> FreeBSD 7.2-STABLE #21: Tue May  5 18:44:10 CEST 2009 (AMD64)
> da0 at arcmsr0 bus 0 target 0 lun 0
> da0: <Areca ARC-1280-VOL#00 R001> Fixed Direct Access SCSI-5 device
> da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
> da0: Command Queueing Enabled
> da0: 10490414MB (21484367872 512 byte sectors: 255H 63S/T 1337340C)
> ZFS filesystem version 6
> ZFS storage pool version 6

If you are able to backup your disks, do it before we go further. I've
some ideas, but they can mess up your data even further.

First of all I'd start with upgrading system to stable/8, there could be
better error recovery.

Do not write anything new to the pool, actually do not even read from it
as it may trigger writting as well.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091004/7f98cd66/attachment.pgp
From aaron at goflexitllc.com  Sun Oct  4 20:05:09 2009
From: aaron at goflexitllc.com (Aaron Hurt)
Date: Sun Oct  4 20:05:16 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <20091004174746.GF1660@garage.freebsd.pl>
References: <683849754.20091001110503@pyro.de>
	<20091004174746.GF1660@garage.freebsd.pl>
Message-ID: <4AC8FFE9.90606@goflexitllc.com>

Pawel Jakub Dawidek wrote:
> On Thu, Oct 01, 2009 at 11:05:03AM +0200, Solon Lutz wrote:
>   
>> Hi erverybody,
>>
>> I'm faced with a 10TB ZFS pool on a 12TB RAID6 Areca controller.
>> And yes, I know, you shouldn't put a zpool on a RAID-device... =(
>>     
>
> Just to be sure: you have no redundancy on ZFS level at all? That's
> very, very bad idea for important data (you know that already, but to
> warn others)...
>
>   
>> The cable was replaced, a parity check was run on the RAID-Volume and
>> showed no errors, the zfs scrub however showed some 'defective' files.
>> After copying these files with 'dd -conv=noerror...' and comparing them
>> to the originals, they were error-free.
>>
>> Yesterday however, three more defective cables forced the controller
>> to take the RAID6 volume offline. Now all cables were replaced and a parity
>> check was run on the RAID-Volume -> data integrity OK.
>>     
>
> This means absolutely nothing. It just means that parity match the
> actual data, it doesn't mean the data is fine from file system or
> application perspective.
>
>   
>> But now ZFS refuses to mount all volumes:
>>
>> Solaris: WARNING: can't process intent log for temp/space1
>> Solaris: WARNING: can't process intent log for temp/space2
>> Solaris: WARNING: can't process intent log for temp/space3
>> Solaris: WARNING: can't process intent log for temp/space4
>>
>> A scrub revealed to following:
>>
>> errors: Permanent errors have been detected in the following files:
>>
>>         temp:<0x0>
>>         temp/space1:<0x0>
>>         temp/space2:<0x0>
>>         temp/space3:<0x0>
>>         temp/space4:<0x0>
>>
>>
>> I tried to switch off checksums for this pool, but that didn't help in any
>> way. I also mounted the pool by hand and was faced with with 'empty' volumes
>> and 'I/O errors' when trying to list their contents...
>>
>> Any suggestions? I'm offering some self-made blackberry jam and raspberry brandy
>> to the person who can help to restore or backup the data.
>>
>> Tech specs:
>>
>> FreeBSD 7.2-STABLE #21: Tue May  5 18:44:10 CEST 2009 (AMD64)
>> da0 at arcmsr0 bus 0 target 0 lun 0
>> da0: <Areca ARC-1280-VOL#00 R001> Fixed Direct Access SCSI-5 device
>> da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
>> da0: Command Queueing Enabled
>> da0: 10490414MB (21484367872 512 byte sectors: 255H 63S/T 1337340C)
>> ZFS filesystem version 6
>> ZFS storage pool version 6
>>     
>
> If you are able to backup your disks, do it before we go further. I've
> some ideas, but they can mess up your data even further.
>
> First of all I'd start with upgrading system to stable/8, there could be
> better error recovery.
>
> Do not write anything new to the pool, actually do not even read from it
> as it may trigger writting as well.
>
>   
I am experiencing a similar issue with a small box here at the house.  
It is not on a raid controller, just a standard 4 port non-raid.  It is 
also giving an I/O error unable to import message.  This started 
happening after a situation similar to the above where I had a drive 
going bad that started giving dma read/write errors and causing the 
machine to lock up...not panic or crash just freeze...so I just turned 
the machine off until I had time to backup the data and move it to a new 
array.  However, when I went to do that to this particular raidz1 pool 
showed faulted and had a status message about corrupt meta data.  I 
hoped I could export/import that pool to get it back in a readable 
state.  That didn't work, the array exported fine without error but now 
refuses to import saying I/O error unable to import.  Long story made 
short, I would also be very appreciative of any ZFS related data 
recovery information or processes.

-- 
Aaron Hurt
Managing Partner
Flex I.T., LLC
611 Commerce Street
Suite 3117
Nashville, TN  37203
Phone: 615.438.7101
E-mail: aaron@goflexitllc.com

From fbsdlist at src.cx  Sun Oct  4 20:35:22 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Sun Oct  4 20:35:29 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <4AC8FFE9.90606@goflexitllc.com>
References: <683849754.20091001110503@pyro.de>
	<20091004174746.GF1660@garage.freebsd.pl>
	<4AC8FFE9.90606@goflexitllc.com>
Message-ID: <ed91d4a80910041335o5b23019du39faa96ba3590a2d@mail.gmail.com>

Similar issue has been discussed on various zfs-related lists.

Following post contains number of useful links on the issue itself and
the ways to recover.
http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg26704.html

--Artem
From aaron at goflexitllc.com  Sun Oct  4 21:22:44 2009
From: aaron at goflexitllc.com (Aaron Hurt)
Date: Sun Oct  4 21:22:53 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <ed91d4a80910041335o5b23019du39faa96ba3590a2d@mail.gmail.com>
References: <683849754.20091001110503@pyro.de>	
	<20091004174746.GF1660@garage.freebsd.pl>	
	<4AC8FFE9.90606@goflexitllc.com>
	<ed91d4a80910041335o5b23019du39faa96ba3590a2d@mail.gmail.com>
Message-ID: <4AC91216.9070200@goflexitllc.com>

Artem Belevich wrote:
> Similar issue has been discussed on various zfs-related lists.
>
> Following post contains number of useful links on the issue itself and
> the ways to recover.
> http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg26704.html
>
> --Artem
>   
Thank you for the links, I read through them and read victor's 
post...however, I am not even able to get the command he mentioned to 
start on my particular machine:

schroeder# zdb -bbccsv 5666150306755460943
zdb: can't open 5666150306755460943: Input/output error

I do have some more hope than before after reading those threads, but I 
am unsure of how to proceed past this point.  Not sure if it will help 
but here is the zdb -ll for all the devices in this pool.

schroeder# zdb -ll /dev/ad4s1
--------------------------------------------
LABEL 0
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14725838678957352699
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 1
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14725838678957352699
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 2
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14725838678957352699
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 3
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14725838678957352699
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
schroeder# zdb -ll /dev/ad6s1
--------------------------------------------
LABEL 0
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478204
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14258118189710079071
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 1
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478204
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14258118189710079071
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 2
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478204
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14258118189710079071
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 3
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478204
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=14258118189710079071
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
schroeder# zdb -ll /dev/ad8s1
--------------------------------------------
LABEL 0
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=3285817555281532863
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 1
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=3285817555281532863
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 2
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=3285817555281532863
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 3
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=3285817555281532863
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
schroeder# zdb -ll /dev/ad10s1
--------------------------------------------
LABEL 0
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=4760880554491391388
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 1
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=4760880554491391388
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 2
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=4760880554491391388
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
--------------------------------------------
LABEL 3
--------------------------------------------
    version=13
    name='tank0'
    state=0
    txg=478240
    pool_guid=5666150306755460943
    hostid=2440235366
    hostname='unset'
    top_guid=2037719367177844310
    guid=4760880554491391388
    vdev_tree
        type='raidz'
        id=0
        guid=2037719367177844310
        nparity=1
        metaslab_array=23
        metaslab_shift=32
        ashift=9
        asize=1600332496896
        is_log=0
        children[0]
                type='disk'
                id=0
                guid=14725838678957352699
                path='/dev/ad4s1'
                whole_disk=0
                DTL=323
        children[1]
                type='disk'
                id=1
                guid=14258118189710079071
                path='/dev/ad6s1'
                whole_disk=0
                DTL=327
                removed=1
        children[2]
                type='disk'
                id=2
                guid=3285817555281532863
                path='/dev/ad8s1'
                whole_disk=0
                DTL=326
        children[3]
                type='disk'
                id=3
                guid=4760880554491391388
                path='/dev/ad10s1'
                whole_disk=0
                DTL=325
schroeder#


Thank You,
Aaron
From pjd at FreeBSD.org  Sun Oct  4 21:23:49 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Sun Oct  4 21:23:56 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <4AC8FFE9.90606@goflexitllc.com>
References: <683849754.20091001110503@pyro.de>
	<20091004174746.GF1660@garage.freebsd.pl>
	<4AC8FFE9.90606@goflexitllc.com>
Message-ID: <20091004212339.GK1660@garage.freebsd.pl>

On Sun, Oct 04, 2009 at 03:04:57PM -0500, Aaron Hurt wrote:
> I am experiencing a similar issue with a small box here at the house.  
> It is not on a raid controller, just a standard 4 port non-raid.  It is 
> also giving an I/O error unable to import message.  This started 
> happening after a situation similar to the above where I had a drive 
> going bad that started giving dma read/write errors and causing the 
> machine to lock up...not panic or crash just freeze...so I just turned 
> the machine off until I had time to backup the data and move it to a new 
> array.  However, when I went to do that to this particular raidz1 pool 
> showed faulted and had a status message about corrupt meta data.  I 
> hoped I could export/import that pool to get it back in a readable 
> state.  That didn't work, the array exported fine without error but now 
> refuses to import saying I/O error unable to import.  Long story made 
> short, I would also be very appreciative of any ZFS related data 
> recovery information or processes.

For starters I'd need FreeBSD version you use, 'zpool import' output,
'zdb -l /dev/<disk>' output for each pool component.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091004/835604c1/attachment.pgp
From pjd at FreeBSD.org  Sun Oct  4 21:36:13 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Sun Oct  4 21:36:19 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <4AC91216.9070200@goflexitllc.com>
References: <683849754.20091001110503@pyro.de>
	<20091004174746.GF1660@garage.freebsd.pl>
	<4AC8FFE9.90606@goflexitllc.com>
	<ed91d4a80910041335o5b23019du39faa96ba3590a2d@mail.gmail.com>
	<4AC91216.9070200@goflexitllc.com>
Message-ID: <20091004213604.GL1660@garage.freebsd.pl>

On Sun, Oct 04, 2009 at 04:22:30PM -0500, Aaron Hurt wrote:
> schroeder# zdb -ll /dev/ad4s1
[...]
>    txg=478240
[...]
> schroeder# zdb -ll /dev/ad6s1
[...]
>    txg=478204
[...]
> schroeder# zdb -ll /dev/ad8s1
[...]
>    txg=478240
[...]
> schroeder# zdb -ll /dev/ad10s1
[...]
>    txg=478240

As you can see one of your vdevs (ad6s1) has lower transaction group
number than the others. The difference is quite big (36 uberblocks), so
we may not be able to go that far in the past. It might be that ZFS
doesn't want to use ad6s1, because it is not up-to-date and there are
some errors on one of the other slices.

If you have you data backed up you may try this patch:

	http://people.freebsd.org/~pjd/patches/vdev_label.c.patch

Once you run ZFS with the patch you can try setting sysctl
vfs.zfs.maxtxg to 478204 and try importing your pool. It will try to
import the pool on txg from ad6s1. It won't work if the other slices
don't have this uberlock anymore or some earlier data is already
overwritten. Do it on your own risk, as it might mess up your data even
further.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091004/3b78987e/attachment.pgp
From bra at fsn.hu  Mon Oct  5 07:24:19 2009
From: bra at fsn.hu (Attila Nagy)
Date: Mon Oct  5 07:24:26 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <20091002184526.GA1660@garage.freebsd.pl>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
Message-ID: <4AC99F1D.3040300@fsn.hu>

On 10/02/09 20:45, Pawel Jakub Dawidek wrote:
> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>   
>> Backing out this change from the 8-STABLE kernel:
>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
>>
>> makes it survive about half and hour of IMAP searching. Of course only 
>> time will tell whether this helps in the long run, but so far 10/10 
>> tries succeeded to kill the machine with this method...
>>     
>
> Could you try this patch:
>
> 	http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>   
Sure. But before that, a report with the above modification: the machine 
has survived some days, then started to behave strangely. Meaning I 
could ping it, I could log in to the IMAP service (running from ZFS), 
read some mails, but not all.
I could not access it via ssh (which runs from UFS), but an already 
running top from a different session was alive. It showed:
last pid: 11272;  load averages:  0.00,  0.00,  0.00    up 3+15:21:13  
09:11:43
149 processes: 1 running, 143 sleeping, 1 zombie, 4 waiting
CPU:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
Mem: 234M Active, 197M Inact, 559M Wired, 111M Buf, 440K Free
Swap: 4096M Total, 976K Used, 4095M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
78492 root        1  44    0  4700K  2156K CPU1    1   5:37  0.00% top
92343 root        1  44    0  4132K  1576K nanslp  1   4:12  0.00% gstat
13401 root        1  44    0  1528K   456K piperd  0   2:19  0.00% 
readproctitl
12679 root        1  44    0  3932K  1236K vmwait  1   2:12  0.00% zpool
35988    125      4  45    0 16892K  5968K sigwai  0   1:53  0.00% 
milter-greyl
25656 root        1  45    0  1536K   564K getblk  0   1:45  0.00% supervise
25798 root        1  44    0  1536K   564K vmwait  0   1:44  0.00% supervise
28406 root        1  44    0  1536K   544K vmwait  0   1:43  0.00% supervise
30226 root        1  44    0  1536K   544K vmwait  0   1:43  0.00% supervise
35401 root        1  44    0  1536K   544K vmwait  0   1:42  0.00% supervise
29203 root        1  44    0  1536K   544K vmwait  0   1:42  0.00% supervise
21629    389      6  44    0 91664K 41892K ucond   0   1:02  0.00% slapd
72283     60      1  44    0 80972K  1948K select  1   0:34  0.00% idled
98960 root        1  44    0  9396K  2544K select  1   0:32  0.00% sshd
 1550 root        1  44    0  3340K   940K vmwait  1   0:32  0.00% syslogd
 5463    125      1  44    0  6924K  2036K vmwait  0   0:27  0.00% qmgr
54193 root        1  44    0  9396K  2516K select  0   0:22  0.00% sshd

I could not log into the console, it didn't even gave a "user name" 
filed after hitting enter. Strange.

I will try the patch.


From bugmaster at FreeBSD.org  Mon Oct  5 11:06:51 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Oct  5 11:07:47 2009
Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org
Message-ID: <200910051106.n95B6oeR088652@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o kern/138790  fs         [zfs] ZFS ceases caching when mem demand is high
o kern/138524  fs         [msdosfs] disks and usb flashes/cards with Russian lab
o kern/138421  fs         [ufs] [patch] remove UFS label limitations
o kern/138367  fs         [tmpfs] [panic] 'panic: Assertion pages > 0 failed' wh
o kern/138202  fs         mount_msdosfs(1) see only 2Gb
o kern/138109  fs         [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2f
f kern/137037  fs         [zfs] [hang] zfs rollback on root causes FreeBSD to fr
o kern/136968  fs         [ufs] [lor] ufs/bufwait/ufs (open)
o kern/136945  fs         [ufs] [lor] filedesc structure/ufs (poll)
o kern/136944  fs         [ffs] [lor] bufwait/snaplk (fsync)
o kern/136873  fs         [ntfs] Missing directories/files on NTFS volume
o kern/136865  fs         [nfs] [patch] NFS exports atomic and on-the-fly atomic
o kern/136470  fs         [nfs] Cannot mount / in read-only, over NFS
o kern/135594  fs         [zfs] Single dataset unresponsive with Samba
o kern/135546  fs         [zfs] zfs.ko module doesn't ignore zpool.cache filenam
o kern/135469  fs         [ufs] [panic] kernel crash on md operation in ufs_dirb
o kern/135050  fs         [zfs] ZFS clears/hides disk errors on reboot
o kern/134491  fs         [zfs] Hot spares are rather cold...
o kern/133980  fs         [panic] [ffs] panic: ffs_valloc: dup alloc
o kern/133676  fs         [smbfs] [panic] umount -f'ing a vnode-based memory dis
o kern/133614  fs         [smbfs] [panic] panic: ffs_truncate: read-only filesys
o kern/133174  fs         [msdosfs] [patch] msdosfs must support utf-encoded int
f kern/133150  fs         [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w
o kern/132960  fs         [ufs] [panic] panic:ffs_blkfree: freeing free frag
o kern/132597  fs         [tmpfs] [panic] tmpfs-related panic while interrupting
o kern/132397  fs         reboot causes filesystem corruption (failure to sync b
o kern/132331  fs         [ufs] [lor] LOR ufs and syncer
o kern/132237  fs         [msdosfs] msdosfs has problems to read MSDOS Floppy
o kern/132145  fs         [panic] File System Hard Crashes
o kern/131995  fs         [nfs] Failure to mount NFSv4 server
o kern/131441  fs         [unionfs] [nullfs] unionfs and/or nullfs not combineab
o kern/131360  fs         [nfs] poor scaling behavior of the NFS server under lo
o kern/131342  fs         [nfs] mounting/unmounting of disks causes NFS to fail
o bin/131341   fs         makefs: error "Bad file descriptor"  on the mount poin
o kern/131086  fs         [ext2fs] [patch] mkfs.ext2 creates rotten partition
o kern/130979  fs         [smbfs] [panic] boot/kernel/smbfs.ko
o kern/130920  fs         [msdosfs] cp(1) takes 100% CPU time while copying file
o kern/130229  fs         [iconv] usermount fails on fs that need iconv
o kern/130210  fs         [nullfs] Error by check nullfs
o kern/129760  fs         [nfs] after 'umount -f' of a stale NFS share FreeBSD l
o kern/129488  fs         [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: 
o kern/129231  fs         [ufs] [patch] New UFS mount (norandom) option - mostly
o kern/129152  fs         [panic] non-userfriendly panic when trying to mount(8)
o kern/129059  fs         [zfs] [patch] ZFS bootloader whitelistable via WITHOUT
f kern/128829  fs         smbd(8) causes periodic panic on 7-RELEASE
f kern/128173  fs         [ext2fs] ls gives "Input/output error" on mounted ext3
o kern/127659  fs         [tmpfs] tmpfs memory leak
o kern/127420  fs         [gjournal] [panic] Journal overflow on gmirrored gjour
o kern/127213  fs         [tmpfs] [patch] sendfile on tmpfs data corruption
o kern/127029  fs         [panic] mount(8): trying to mount a write protected zi
o kern/126287  fs         [ufs] [panic] Kernel panics while mounting an UFS file
s kern/125738  fs         [zfs] [request] SHA256 acceleration in ZFS
f kern/125536  fs         [ext2fs] ext 2 mounts cleanly but fails on commands li
f kern/124621  fs         [ext3] [patch] Cannot mount ext2fs partition
f bin/124424   fs         [zfs] zfs(8): zfs list -r shows strange snapshots' siz
o kern/123939  fs         [msdosfs] corrupts new files
o kern/122380  fs         [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash
o bin/122172   fs         [fs]: amd(8) automount daemon dies on 6.3-STABLE i386,
o kern/122047  fs         [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / 
o kern/122038  fs         [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc
o bin/121898   fs         [nullfs] pwd(1)/getcwd(2) fails with Permission denied
o bin/121779   fs         [ufs] snapinfo(8) (and related tools?) only work for t
o bin/121366   fs         [zfs] [patch] Automatic disk scrubbing from periodic(8
o bin/121072   fs         [smbfs] mount_smbfs(8) cannot normally convert the cha
f kern/120991  fs         [panic] [fs] [snapshot] System crashes when manipulati
o kern/120483  fs         [ntfs] [patch] NTFS filesystem locking changes
o kern/120482  fs         [ntfs] [patch] Sync style changes between NetBSD and F
f kern/119735  fs         [zfs] geli + ZFS + samba starting on boot panics 7.0-B
o kern/118912  fs         [2tb] disk sizing/geometry problem with large array
o kern/118713  fs         [minidump] [patch] Display media size required for a k
o bin/118249   fs         mv(1): moving a directory changes its mtime
o kern/118107  fs         [ntfs] [panic] Kernel panic when accessing a file at N
o bin/117315   fs         [smbfs] mount_smbfs(8) and related options can't mount
o kern/117314  fs         [ntfs] Long-filename only NTFS fs'es cause kernel pani
o kern/117158  fs         [zfs] zpool scrub causes panic if geli vdevs detach on
o bin/116980   fs         [msdosfs] [patch] mount_msdosfs(8) resets some flags f
o kern/116913  fs         [ffs] [panic] ffs_blkfree: freeing free block
p kern/116608  fs         [msdosfs] [patch] msdosfs fails to check mount options
o kern/116583  fs         [ffs] [hang] System freezes for short time when using 
o kern/116170  fs         [panic] Kernel panic when mounting /tmp
o kern/115645  fs         [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex
o bin/115361   fs         [zfs] mount(8) gets into a state where it won't set/un
o kern/114955  fs         [cd9660] [patch] [request] support for mask,dirmask,ui
o kern/114847  fs         [ntfs] [patch] [request] dirmask support for NTFS ala 
o kern/114676  fs         [ufs] snapshot creation panics: snapacct_ufs2: bad blo
o bin/114468   fs         [patch] [request] add -d option to umount(8) to detach
o kern/113852  fs         [smbfs] smbfs does not properly implement DFS referral
o bin/113838   fs         [patch] [request] mount(8): add support for relative p
o bin/113049   fs         [patch] [request] make quot(8) use getopt(3) and show 
o kern/112658  fs         [smbfs] [patch] smbfs and caching problems (resolves b
f usb/112640   fs         [ext2fs] [hang] Kernel freezes when writing a file to 
o kern/111843  fs         [msdosfs] Long Names of files are incorrectly created 
o kern/111782  fs         [ufs] dump(8) fails horribly for large filesystems
s bin/111146   fs         [2tb] fsck(8) fails on 6T filesystem
o kern/109024  fs         [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not 
o kern/109010  fs         [msdosfs] can't mv directory within fat32 file system
o bin/107829   fs         [2TB] fdisk(8): invalid boundary checking in fdisk / w
o kern/106030  fs         [ufs] [panic] panic in ufs from geom when a dead disk 
o kern/105093  fs         [ext2fs] [patch] ext2fs on read-only media cannot be m
o kern/104406  fs         [ufs] Processes get stuck in "ufs" state under persist
o kern/104133  fs         [ext2fs] EXT2FS module corrupts EXT2/3 filesystems
o kern/103035  fs         [ntfs] Directories in NTFS mounted disc images appear 
o kern/101324  fs         [smbfs] smbfs sometimes not case sensitive when it's s
o kern/99290   fs         [ntfs] mount_ntfs ignorant of cluster sizes
o kern/97377   fs         [ntfs] [patch] syntax cleanup for ntfs_ihash.c
o kern/95222   fs         [iso9660] File sections on ISO9660 level 3 CDs ignored
o kern/94849   fs         [ufs] rename on UFS filesystem is not atomic
o kern/94769   fs         [ufs] Multiple file deletions on multi-snapshotted fil
o kern/94733   fs         [smbfs] smbfs may cause double unlock
o kern/93942   fs         [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D
o kern/92272   fs         [ffs] [hang] Filling a filesystem while creating a sna
f kern/91568   fs         [ufs] [panic] writing to UFS/softupdates DVD media in 
o kern/91134   fs         [smbfs] [patch] Preserve access and modification time 
a kern/90815   fs         [smbfs] [patch] SMBFS with character conversions somet
o kern/89991   fs         [ufs] softupdates with mount -ur causes fs UNREFS
o kern/88657   fs         [smbfs] windows client hang when browsing a samba shar
o kern/88266   fs         [smbfs] smbfs does not implement UIO_NOCOPY and sendfi
o kern/87859   fs         [smbfs] System reboot while umount smbfs.
o kern/86587   fs         [msdosfs] rm -r /PATH fails with lots of small files
o kern/85326   fs         [smbfs] [panic] saving a file via samba to an overquot
o kern/84589   fs         [2TB] 5.4-STABLE unresponsive during background fsck 2
o kern/80088   fs         [smbfs] Incorrect file time setting on NTFS mounted vi
o kern/77826   fs         [ext2fs] ext2fs usb filesystem will not mount RW
o kern/73484   fs         [ntfs] Kernel panic when doing `ls` from the client si
o bin/73019    fs         [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino
o kern/71774   fs         [ntfs] NTFS cannot "see" files on a WinXP filesystem
o kern/68978   fs         [panic] [ufs] crashes with failing hard disk, loose po
o kern/65920   fs         [nwfs] Mounted Netware filesystem behaves strange
o kern/65901   fs         [smbfs] [patch] smbfs fails fsx write/truncate-down/tr
o kern/61503   fs         [smbfs] mount_smbfs does not work as non-root
o kern/55617   fs         [smbfs] Accessing an nsmb-mounted drive via a smb expo
o kern/51685   fs         [hang] Unbounded inode allocation causes kernel to loc
o kern/51583   fs         [nullfs] [patch] allow to work with devices and socket
o kern/36566   fs         [smbfs] System reboot with dead smb mount and umount
o kern/18874   fs         [2TB] 32bit NFS servers export wrong negative values t

135 problems total.

From cowens at greatbaysoftware.com  Mon Oct  5 15:00:18 2009
From: cowens at greatbaysoftware.com (Charles Owens)
Date: Mon Oct  5 15:00:25 2009
Subject: gjournal crash: "error while writing data (error=6)"
Message-ID: <4ACA015D.3090800@greatbaysoftware.com>

Hello folks,

We've had a system crash, apparently related to GEOM_JOURNAL, on an i386
system running 7.0-RELEASE-p11.   Here's what we could see on the screen
(formatted for readability):

GEOM_JOURNAL: [copy] Error while writting data (error=6) \
    ad4s1a[WRITE(offset=43561402368, length=16384)]
GEOM_JOURNAL: [copy] Error while writting data (error=6) \
    ad4s1a[WRITE(offset=48868164096, length=16896)]
GEOM_JOURNAL: Error while reading data from ad4s1a (error=6).
mode=0134172, inum=5323776, fs = / 
panic: ffs_valloc: dup alloc
cpuid = 3
Uptime: 119d10h5m43s
Cannot dump. No dump device defined.
GEOM_JOURNAL: Flush of cache of ad4s1a: error=6.
GEOM_JOURNAL: [flush] Error while writting data (error=6) \
    ad4s1a[WRITE(offset=48868197888, length=98816)]
(4 more lines like last one)
Rebooting...
cpu_reset: Stopping other CPUs


The system didn't actually reboot.. just got stuck there. When it was
eventually manually rebooted, it booted just fine.  Any thoughts as to
what could be the real problem?  What does "error=6" indicate?

I've done some scouring of the net and found something that may not
directly relate to this crash... but does relate, at least, to my
filesystem configuration.   One of the threads: 
http://markmail.org/message/tamo4r2jho3zdv3z

In the described crash, similar error messages were seen, but with
"error=1".  Ultimately Pawel Dawidek (gjournal author) gave the
diagnosis that the crash was related to the first filesystem in the
slice being set up with an offset of zero, not the correct offset of
16.  Either in this thread or elsewhere I also learned that sysinstall
always uses the zero offset... even though it is not best practice.  Not
a happy discovery.

Looking at our system that crashed... sure enough, zero offset (see
label below below -- both 'a' and 'd' are journaled).  So this then
prompts two questions:

    * Can our crash be explained by the zero offset filesystem
      configuration?
    * If not, separate from the crash, how much should we be worried
      about running a system with gjournal like this?


Thanks very much for any and all assistance,

Charles

# bsdlabel ad4s1

# /dev/ad4s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
 a: 77594624        0    4.2BSD     2048 16384 28552 
 b: 24113088 77594624      swap                    
 c: 156296322        0    unused        0     0         # "raw" part, don't edit
 d: 54588610 101707712    4.2BSD     2048 16384 28552 


-- 

**Charles Owens**
*Great Bay Software**|** ** e: *cowens@GreatBaySoftware.com****

From fbsdlist at src.cx  Mon Oct  5 15:51:36 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Mon Oct  5 15:51:43 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <4AC99F1D.3040300@fsn.hu>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl> <4AC99F1D.3040300@fsn.hu>
Message-ID: <ed91d4a80910050851m3d599f7ai67a57ef17a9a61e7@mail.gmail.com>

Your lockup is very similar (processes stuck sleeping on vmwait) to
what I had when arc_min was set too high. With Pawel's patch ZFS would
not give up any memory above arc_min.
Try bringing vfs.zfs.arc_min down.

--Artem



2009/10/5 Attila Nagy <bra@fsn.hu>:
> On 10/02/09 20:45, Pawel Jakub Dawidek wrote:
>>
>> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>>
>>>
>>> Backing out this change from the 8-STABLE kernel:
>>>
>>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
>>>
>>> makes it survive about half and hour of IMAP searching. Of course only
>>> time will tell whether this helps in the long run, but so far 10/10 tries
>>> succeeded to kill the machine with this method...
>>>
>>
>> Could you try this patch:
>>
>> ? ? ? ?http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>>
>
> Sure. But before that, a report with the above modification: the machine has
> survived some days, then started to behave strangely. Meaning I could ping
> it, I could log in to the IMAP service (running from ZFS), read some mails,
> but not all.
> I could not access it via ssh (which runs from UFS), but an already running
> top from a different session was alive. It showed:
> last pid: 11272; ?load averages: ?0.00, ?0.00, ?0.00 ? ?up 3+15:21:13
> ?09:11:43
> 149 processes: 1 running, 143 sleeping, 1 zombie, 4 waiting
> CPU: ?0.0% user, ?0.0% nice, ?0.2% system, ?0.0% interrupt, 99.8% idle
> Mem: 234M Active, 197M Inact, 559M Wired, 111M Buf, 440K Free
> Swap: 4096M Total, 976K Used, 4095M Free
>
> ?PID USERNAME ?THR PRI NICE ? SIZE ? ?RES STATE ? C ? TIME ? WCPU COMMAND
> 78492 root ? ? ? ?1 ?44 ? ?0 ?4700K ?2156K CPU1 ? ?1 ? 5:37 ?0.00% top
> 92343 root ? ? ? ?1 ?44 ? ?0 ?4132K ?1576K nanslp ?1 ? 4:12 ?0.00% gstat
> 13401 root ? ? ? ?1 ?44 ? ?0 ?1528K ? 456K piperd ?0 ? 2:19 ?0.00%
> readproctitl
> 12679 root ? ? ? ?1 ?44 ? ?0 ?3932K ?1236K vmwait ?1 ? 2:12 ?0.00% zpool
> 35988 ? ?125 ? ? ?4 ?45 ? ?0 16892K ?5968K sigwai ?0 ? 1:53 ?0.00%
> milter-greyl
> 25656 root ? ? ? ?1 ?45 ? ?0 ?1536K ? 564K getblk ?0 ? 1:45 ?0.00% supervise
> 25798 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 564K vmwait ?0 ? 1:44 ?0.00% supervise
> 28406 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 544K vmwait ?0 ? 1:43 ?0.00% supervise
> 30226 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 544K vmwait ?0 ? 1:43 ?0.00% supervise
> 35401 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 544K vmwait ?0 ? 1:42 ?0.00% supervise
> 29203 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 544K vmwait ?0 ? 1:42 ?0.00% supervise
> 21629 ? ?389 ? ? ?6 ?44 ? ?0 91664K 41892K ucond ? 0 ? 1:02 ?0.00% slapd
> 72283 ? ? 60 ? ? ?1 ?44 ? ?0 80972K ?1948K select ?1 ? 0:34 ?0.00% idled
> 98960 root ? ? ? ?1 ?44 ? ?0 ?9396K ?2544K select ?1 ? 0:32 ?0.00% sshd
> 1550 root ? ? ? ?1 ?44 ? ?0 ?3340K ? 940K vmwait ?1 ? 0:32 ?0.00% syslogd
> 5463 ? ?125 ? ? ?1 ?44 ? ?0 ?6924K ?2036K vmwait ?0 ? 0:27 ?0.00% qmgr
> 54193 root ? ? ? ?1 ?44 ? ?0 ?9396K ?2516K select ?0 ? 0:22 ?0.00% sshd
>
> I could not log into the console, it didn't even gave a "user name" filed
> after hitting enter. Strange.
>
> I will try the patch.
>
>
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
From pjd at FreeBSD.org  Mon Oct  5 16:13:06 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Mon Oct  5 16:13:12 2009
Subject: gjournal crash: "error while writing data (error=6)"
In-Reply-To: <4ACA015D.3090800@greatbaysoftware.com>
References: <4ACA015D.3090800@greatbaysoftware.com>
Message-ID: <20091005161258.GE1702@garage.freebsd.pl>

On Mon, Oct 05, 2009 at 10:23:25AM -0400, Charles Owens wrote:
> Hello folks,
> 
> We've had a system crash, apparently related to GEOM_JOURNAL, on an i386
> system running 7.0-RELEASE-p11.   Here's what we could see on the screen
> (formatted for readability):
> 
> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>     ad4s1a[WRITE(offset=43561402368, length=16384)]
> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>     ad4s1a[WRITE(offset=48868164096, length=16896)]
> GEOM_JOURNAL: Error while reading data from ad4s1a (error=6).

Error 6 (ENXIO, man errno(2)) might mean that ad4s1a disappeared. There
were no any errors earlier indicating that ad4 was disconnected or
similar?

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091005/cfd00d71/attachment.pgp
From cowens at greatbaysoftware.com  Mon Oct  5 16:26:23 2009
From: cowens at greatbaysoftware.com (Charles Owens)
Date: Mon Oct  5 16:26:29 2009
Subject: gjournal crash: "error while writing data (error=6)"
In-Reply-To: <20091005161258.GE1702@garage.freebsd.pl>
References: <4ACA015D.3090800@greatbaysoftware.com>
	<20091005161258.GE1702@garage.freebsd.pl>
Message-ID: <4ACA1EEA.3070204@greatbaysoftware.com>

Pawel Jakub Dawidek wrote:
> On Mon, Oct 05, 2009 at 10:23:25AM -0400, Charles Owens wrote:
>   
>> Hello folks,
>>
>> We've had a system crash, apparently related to GEOM_JOURNAL, on an i386
>> system running 7.0-RELEASE-p11.   Here's what we could see on the screen
>> (formatted for readability):
>>
>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>>     ad4s1a[WRITE(offset=43561402368, length=16384)]
>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>>     ad4s1a[WRITE(offset=48868164096, length=16896)]
>> GEOM_JOURNAL: Error while reading data from ad4s1a (error=6).
>>     
>
> Error 6 (ENXIO, man errno(2)) might mean that ad4s1a disappeared. There
> were no any errors earlier indicating that ad4 was disconnected or
> similar?
>   

Such as if someone had come by and temporarily pulled out the drive?  
Nothing in the logs of the sort, no.

Do you think this is unrelated to the offset-zero layout question?  In
any case, should we be worried about that?

Thanks for the reply.
From pjd at FreeBSD.org  Mon Oct  5 16:59:00 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Mon Oct  5 16:59:07 2009
Subject: gjournal crash: "error while writing data (error=6)"
In-Reply-To: <4ACA1EEA.3070204@greatbaysoftware.com>
References: <4ACA015D.3090800@greatbaysoftware.com>
	<20091005161258.GE1702@garage.freebsd.pl>
	<4ACA1EEA.3070204@greatbaysoftware.com>
Message-ID: <20091005165852.GF1702@garage.freebsd.pl>

On Mon, Oct 05, 2009 at 12:29:30PM -0400, Charles Owens wrote:
> Pawel Jakub Dawidek wrote:
> > On Mon, Oct 05, 2009 at 10:23:25AM -0400, Charles Owens wrote:
> >   
> >> Hello folks,
> >>
> >> We've had a system crash, apparently related to GEOM_JOURNAL, on an i386
> >> system running 7.0-RELEASE-p11.   Here's what we could see on the screen
> >> (formatted for readability):
> >>
> >> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
> >>     ad4s1a[WRITE(offset=43561402368, length=16384)]
> >> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
> >>     ad4s1a[WRITE(offset=48868164096, length=16896)]
> >> GEOM_JOURNAL: Error while reading data from ad4s1a (error=6).
> >>     
> >
> > Error 6 (ENXIO, man errno(2)) might mean that ad4s1a disappeared. There
> > were no any errors earlier indicating that ad4 was disconnected or
> > similar?
> >   
> 
> Such as if someone had come by and temporarily pulled out the drive?  

It can be buggy ata controller or failing disk.

> Nothing in the logs of the sort, no.

What is exact partition size in bytes?


-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091005/b91da485/attachment.pgp
From cowens at greatbaysoftware.com  Mon Oct  5 17:53:46 2009
From: cowens at greatbaysoftware.com (Charles Owens)
Date: Mon Oct  5 17:53:52 2009
Subject: gjournal crash: "error while writing data (error=6)"
In-Reply-To: <20091005165852.GF1702@garage.freebsd.pl>
References: <4ACA015D.3090800@greatbaysoftware.com>
	<20091005161258.GE1702@garage.freebsd.pl>
	<4ACA1EEA.3070204@greatbaysoftware.com>
	<20091005165852.GF1702@garage.freebsd.pl>
Message-ID: <4ACA3363.1090309@greatbaysoftware.com>

Pawel Jakub Dawidek wrote:
> On Mon, Oct 05, 2009 at 12:29:30PM -0400, Charles Owens wrote:
>   
>> Pawel Jakub Dawidek wrote:
>>     
>>> On Mon, Oct 05, 2009 at 10:23:25AM -0400, Charles Owens wrote:
>>>   
>>>       
>>>> Hello folks,
>>>>
>>>> We've had a system crash, apparently related to GEOM_JOURNAL, on an i386
>>>> system running 7.0-RELEASE-p11.   Here's what we could see on the screen
>>>> (formatted for readability):
>>>>
>>>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>>>>     ad4s1a[WRITE(offset=43561402368, length=16384)]
>>>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>>>>     ad4s1a[WRITE(offset=48868164096, length=16896)]
>>>> GEOM_JOURNAL: Error while reading data from ad4s1a (error=6).
>>>>     
>>>>         
>>> Error 6 (ENXIO, man errno(2)) might mean that ad4s1a disappeared. There
>>> were no any errors earlier indicating that ad4 was disconnected or
>>> similar?
>>>   
>>>       
>> Such as if someone had come by and temporarily pulled out the drive?  
>>     
>
> It can be buggy ata controller or failing disk.
>
>   
>> Nothing in the logs of the sort, no.
>>     
>
> What is exact partition size in bytes?
>   

>From 'bsdlabel ad4s1':

  a: 77594624        0    4.2BSD     2048 16384 28552 


77594624 * 512 = 39728447488 (37 GB)


**Charles Owens**
*Great Bay Software*****


From cowens at greatbaysoftware.com  Mon Oct  5 18:27:08 2009
From: cowens at greatbaysoftware.com (Charles Owens)
Date: Mon Oct  5 18:27:16 2009
Subject: gjournal crash: "error while writing data (error=6)"
In-Reply-To: <20091005165852.GF1702@garage.freebsd.pl>
References: <4ACA015D.3090800@greatbaysoftware.com>
	<20091005161258.GE1702@garage.freebsd.pl>
	<4ACA1EEA.3070204@greatbaysoftware.com>
	<20091005165852.GF1702@garage.freebsd.pl>
Message-ID: <4ACA3B37.6070501@greatbaysoftware.com>

Pawel Jakub Dawidek wrote:
> On Mon, Oct 05, 2009 at 12:29:30PM -0400, Charles Owens wrote:
>   
>> Pawel Jakub Dawidek wrote:
>>     
>>> On Mon, Oct 05, 2009 at 10:23:25AM -0400, Charles Owens wrote:
>>>   
>>>       
>>>> Hello folks,
>>>>
>>>> We've had a system crash, apparently related to GEOM_JOURNAL, on an i386
>>>> system running 7.0-RELEASE-p11.   Here's what we could see on the screen
>>>> (formatted for readability):
>>>>
>>>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>>>>     ad4s1a[WRITE(offset=43561402368, length=16384)]
>>>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
>>>>     ad4s1a[WRITE(offset=48868164096, length=16896)]
>>>> GEOM_JOURNAL: Error while reading data from ad4s1a (error=6).
>>>>     
>>>>         
>>> Error 6 (ENXIO, man errno(2)) might mean that ad4s1a disappeared. There
>>> were no any errors earlier indicating that ad4 was disconnected or
>>> similar?
>>>   
>>>       
>> Such as if someone had come by and temporarily pulled out the drive?  
>>     
>
> It can be buggy ata controller or failing disk.
>
>   
>> Nothing in the logs of the sort, no.
>>     
>
> What is exact partition size in bytes

Sorry, I'd grabbed the size from a similar system which I realized has a
smaller drive.

Here's the partition size from the real system in question:  59055800320
(55 GB)

While I'm at it, here's some additional detail on gjournal:

# gjournal list
Geom name: gjournal 2048257491
ID: 2048257491
Providers:
1. Name: ad4s1a.journal
   Mediasize: 45182090752 (42G)
   Sectorsize: 512
   Mode: r1w1e1
Consumers:
1. Name: ad4s1a
   Mediasize: 59055800320 (55G)
   Sectorsize: 512
   Mode: r1w1e1
   Jend: 59055799808
   Jstart: 45182090752
   Role: Data,Journal
 
Geom name: gjournal 3790277183
ID: 3790277183
Providers:
1. Name: ad4s1d.journal
   Mediasize: 168868626944 (157G)
   Sectorsize: 512
   Mode: r1w1e1
Consumers:
1. Name: ad4s1d
   Mediasize: 182742336512 (170G)
   Sectorsize: 512
   Mode: r1w1e1
   Jend: 182742336000
   Jstart: 168868626944
   Role: Data,Journal

# bsdlabel ad4s1
# /dev/ad4s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a: 115343360        0    4.2BSD     2048 16384 28552
  b: 16130016 115343360      swap                  
  c: 488392002        0    unused        0     0         # "raw" part,
don't edit
  d: 356918626 131473376    4.2BSD     2048 16384 28552



**Charles Owens**
*Great Bay Software*



From fbsdlist at src.cx  Mon Oct  5 18:28:34 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Mon Oct  5 18:28:40 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <ed91d4a80910021759h4a3d3b9csde86c59075795eeb@mail.gmail.com>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<ed91d4a80910021638r72213389uf0767b9bb523b7d9@mail.gmail.com>
	<20091003000909.GD1660@garage.freebsd.pl>
	<ed91d4a80910021759h4a3d3b9csde86c59075795eeb@mail.gmail.com>
Message-ID: <ed91d4a80910051128r568c7b0fj278452ec2c4d6bd5@mail.gmail.com>

I left my box running over the weekend (vm.kmem_size=8G) in a loop
doing a build and then deleting build results. Each loop cycle takes
about 5 hours. All in all it touches about 2GB of sources and produces
about 30GB of object files and stuff.

This morning ARC size is around 2.5G. Now and then it dips down to 1G.
I've attached graph with memory stats and ARC size.

--Artem

> ===============================================================
> Now, the same experiment, with vm.kmem_size=8G
> vm.kmem_size: 8589934592
> vfs.zfs.arc_min: 939524096
> vfs.zfs.arc_max: 7516192768
>
> ARC grows to 6.2G:
> Mem: 47M Active, 13M Inact, 7376M Wired, 31M Buf, 473M Free
>
> Then it quickly shrinks to 4.6G and grows to 6.2G again, shrinks again, etc..
>
> What's different from the previous case is that after a while ZFS
> adjusts target size (kstat.zfs.misc.arcstats.c) down to ~5.8G and
> after that ZFS size oscillates between 4.2G and 5.6G. Another
> observation -- ARC shrinking happens when system is left with ~512M of
> free memory. Yet another observation is that even with ARC peak of
> ~5.8G, system has about 7.5G wired. Where did almost 2G of difference
> go? Fragmentation?
>
> I've tried both experiments with and without L2ARC -- behavior seems
> to be the same.
>
> --Artem
>
From linimon at FreeBSD.org  Tue Oct  6 04:30:20 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Tue Oct  6 04:30:25 2009
Subject: kern/139363: [nfs] diskless root nfs mount from non FreeBSD
	server broken
Message-ID: <200910060430.n964UJtK090006@freefall.freebsd.org>

Synopsis: [nfs] diskless root nfs mount from non FreeBSD server broken

Responsible-Changed-From-To: freebsd-s->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Tue Oct 6 04:30:06 UTC 2009
Responsible-Changed-Why: 
gonna have to replace this keyboard soon ...

http://www.freebsd.org/cgi/query-pr.cgi?pr=139363
From pjd at FreeBSD.org  Tue Oct  6 18:39:47 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Tue Oct  6 18:40:52 2009
Subject: gjournal crash: "error while writing data (error=6)"
In-Reply-To: <4ACA3B37.6070501@greatbaysoftware.com>
References: <4ACA015D.3090800@greatbaysoftware.com>
	<20091005161258.GE1702@garage.freebsd.pl>
	<4ACA1EEA.3070204@greatbaysoftware.com>
	<20091005165852.GF1702@garage.freebsd.pl>
	<4ACA3B37.6070501@greatbaysoftware.com>
Message-ID: <20091006183937.GA1639@garage.freebsd.pl>

On Mon, Oct 05, 2009 at 02:30:15PM -0400, Charles Owens wrote:
> >>>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
> >>>>     ad4s1a[WRITE(offset=43561402368, length=16384)]
> >>>> GEOM_JOURNAL: [copy] Error while writting data (error=6) \
> >>>>     ad4s1a[WRITE(offset=48868164096, length=16896)]
[...]
> Here's the partition size from the real system in question:  59055800320
> (55 GB)

That's better, now we know that gjournal was using valid offset.

This still doesn't solve your mistery, but whatever it is, it doesn't
look like gjournal, because the error you are seeing was received from
provider below (ad4s1a). The error value suggest that this partition
disappeared or something else bad happen. The logs you pasted don't tell
us what, so I can only guess.

As I suggested earlier it could be that disk was detached on an error.
It happends sometimes in my home file server - it controller doesn't
report errors, but detached the disk instead.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091006/217e9a91/attachment.pgp
From linimon at FreeBSD.org  Wed Oct  7 18:10:41 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Wed Oct  7 18:10:47 2009
Subject: kern/139407: [smbfs] [panic] smb mount causes system crash if
	remote share no longer accessible
Message-ID: <200910071810.n97IAfGW029270@freefall.freebsd.org>

Old Synopsis: smb mount causes system crash if remote share no longer accessible
New Synopsis: [smbfs] [panic] smb mount causes system crash if remote share no longer accessible

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Wed Oct 7 18:10:01 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139407
From delphij at FreeBSD.org  Wed Oct  7 23:18:36 2009
From: delphij at FreeBSD.org (delphij@FreeBSD.org)
Date: Wed Oct  7 23:18:42 2009
Subject: kern/127213: [tmpfs] [patch] sendfile on tmpfs data corruption
Message-ID: <200910072318.n97NIZ7S090128@freefall.freebsd.org>

Synopsis: [tmpfs] [patch] sendfile on tmpfs data corruption

State-Changed-From-To: open->patched
State-Changed-By: delphij
State-Changed-When: Wed Oct 7 23:17:33 UTC 2009
State-Changed-Why: 
Patch from gk@ has been applied against -HEAD, MFC reminder.


Responsible-Changed-From-To: freebsd-fs->delphij
Responsible-Changed-By: delphij
Responsible-Changed-When: Wed Oct 7 23:17:33 UTC 2009
Responsible-Changed-Why: 
Take.

http://www.freebsd.org/cgi/query-pr.cgi?pr=127213
From oliver.pntr at gmail.com  Thu Oct  8 00:22:16 2009
From: oliver.pntr at gmail.com (Oliver Pinter)
Date: Thu Oct  8 00:22:22 2009
Subject: kern/139407: [smbfs] [panic] smb mount causes system crash if 
	remote share no longer accessible
In-Reply-To: <200910071810.n97IAfGW029270@freefall.freebsd.org>
References: <200910071810.n97IAfGW029270@freefall.freebsd.org>
Message-ID: <6101e8c40910071700q62982a0aqba04e8cd84d4f4b8@mail.gmail.com>

http://lists.freebsd.org/pipermail/freebsd-stable/2009-July/051099.html

On 10/7/09, linimon@freebsd.org <linimon@freebsd.org> wrote:
> Old Synopsis: smb mount causes system crash if remote share no longer
> accessible
> New Synopsis: [smbfs] [panic] smb mount causes system crash if remote share
> no longer accessible
>
> Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
> Responsible-Changed-By: linimon
> Responsible-Changed-When: Wed Oct 7 18:10:01 UTC 2009
> Responsible-Changed-Why:
> Over to maintainer(s).
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=139407
> _______________________________________________
> freebsd-bugs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
> To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@freebsd.org"
>
From matt at ixsystems.com  Thu Oct  8 05:07:32 2009
From: matt at ixsystems.com (Matt Olander)
Date: Thu Oct  8 05:07:39 2009
Subject: glusterfs/zfs
Message-ID: <9740caf0910072140s6c12ebbm4aa6e1b6019b2d50@mail.gmail.com>

We've got an opportunity to work with a popular open source project
building a server farm for their bug reporting database. They are
running out of inodes on some expensive NetApp equipment so FreeBSD
and ZFS look like a good potential solution. Does anyone have
experience running glusterfs on FreeBSD 8 yet? They'd like to use
glusterfs for this setup.

best,
-matt
From bra at fsn.hu  Thu Oct  8 08:42:23 2009
From: bra at fsn.hu (Attila Nagy)
Date: Thu Oct  8 08:42:30 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <20091002184526.GA1660@garage.freebsd.pl>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
Message-ID: <4ACDA5EA.2010600@fsn.hu>

Hello,

Pawel Jakub Dawidek wrote:
> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>   
>> Backing out this change from the 8-STABLE kernel:
>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
>>
>> makes it survive about half and hour of IMAP searching. Of course only 
>> time will tell whether this helps in the long run, but so far 10/10 
>> tries succeeded to kill the machine with this method...
>>     
>
> Could you try this patch:
>
> 	http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>   
It seems (after running for two days) that this fixes my problem. And I 
see that Kip has came out with a similar version (which I couldn't yet 
test, but hope that will also do).

Thanks!
From solon at pyro.de  Thu Oct  8 09:37:23 2009
From: solon at pyro.de (Solon Lutz)
Date: Thu Oct  8 09:37:30 2009
Subject: raidz slowing down
Message-ID: <886802879.20091008113716@pyro.de>

I built a 9x hdd 11TB raidz for some rescue purposes and started
copying an image from another partition via "dd if=/dev/da0..." to it.
It consists of: ad4 da1 da2 da3 da4 da5 da6 da7 da8, da1 to da8 are
connected via two highpoint controllers.

In the beginning write speeds were quite fair:

dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0    424      0      0    0.0    424  52483   33.9   84.6| ad4
    0      0      0      0    0.0      0      0    0.0    0.0| da0
   35    356      0      0    0.0    356  44584   76.4  124.5| da1
   35    296      0      0    0.0    296  36919   84.5  121.0| da2
   34    361      0      0    0.0    361  45111   75.5  124.7| da3
   35    346      0      0    0.0    346  43196   78.6  123.2| da4
   35    344      0      0    0.0    344  42940   80.0  124.7| da5
   35    343      0      0    0.0    343  42812   80.7  124.5| da6
   35    344      0      0    0.0    344  43051   79.8  123.9| da7
   34    342      0      0    0.0    342  42796   80.6  124.4| da8

Now, some 10 hours and 2.5TB later, it look like that most of the time:

dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0     10      0      0    0.0     10      6    0.8    0.2| ad4
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    4     13      0      0    0.0     13      8  550.4  178.5| da1
    0     12      0      0    0.0     12      7    0.7    0.2| da2
    0     11      0      0    0.0     11      7    0.7    0.2| da3
    0     10      0      0    0.0     10      5    0.6    0.2| da4
    0     11      0      0    0.0     11      6    0.9    0.3| da5
    0     12      0      0    0.0     12      7    0.7    0.2| da6
    0     11      0      0    0.0     11      7    0.7    0.2| da7
    0      9      0      0    0.0      9      6    0.8    0.2| da8


da1 seems to be busy most of time and every few seconds all the other
devices write some data with nearly normal speed:

dT: 1.003s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0    254      0      0    0.0    254  31331   34.9   35.4| ad4
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    4      0      0      0    0.0      0      0    0.0    0.0| da1
    0    254      0      0    0.0    254  31346  107.4  104.5| da2
    0    256      0      0    0.0    256  31345  108.1  104.0| da3
    0    255      0      0    0.0    255  31345  110.2  105.1| da4
   35    200      0      0    0.0    200  24912  143.3  115.0| da5
   35    211      0      0    0.0    211  26303  137.8  114.9| da6
   35    210      0      0    0.0    210  26079  139.3  114.9| da7
   35    209      0      0    0.0    209  25952  135.2  113.7| da8

Sometimes it even gets back to 'normal' behaviour, but never reaches
the speeds it once had:

dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
   35    274      0      0    0.0    274  34334   44.2   66.6| ad4
    0   1166   1166 149243    0.1      0      0    0.0   14.3| da0
   35    120      0      0    0.0    120  14717   94.4   64.5| da1
   35     96      0      0    0.0     96  11665  113.9   64.3| da2
   35    100      0      0    0.0    100  12288   98.7   63.9| da3
   35    103      0      0    0.0    103  12496   93.4   59.4| da4
   34    112      0      0    0.0    112  13694  106.1   67.4| da5
   35     71      0      0    0.0     71   8596  115.3   66.8| da6
   35    116      0      0    0.0    116  14205  101.7   67.3| da7
   35     83      0      0    0.0     83  10066  112.2   65.9| da8

Syslog reports the following:

Oct  8 09:53:40 radium kernel: hptrr: start channel [0,0]
Oct  8 09:53:40 radium kernel: hptrr: channel [0,0] started successfully
Oct  8 09:57:44 radium kernel: hptrr: start channel [0,0]
Oct  8 09:57:45 radium kernel: hptrr: channel [0,0] started successfully
Oct  8 10:54:26 radium kernel: hptrr: start channel [0,0]
Oct  8 10:54:27 radium kernel: hptrr: channel [0,0] started successfully
Oct  8 11:10:29 radium kernel: hptrr: start channel [0,0]
Oct  8 11:10:30 radium kernel: hptrr: channel [0,0] started successfully
Oct  8 11:17:27 radium kernel: hptrr: start channel [0,0]
Oct  8 11:17:27 radium kernel: hptrr: channel [0,0] started successfully

Is this a problem of the hptrr device or is da1 failing?




































Mit freundlichen Gr??en
Best regards,

Solon Lutz


+-----------------------------------------------+
| Pyro.Labs Berlin -  Creativity for tomorrow   |
| Wasgenstrasse 75/13 - 14129 Berlin, Germany   |
| www.pyro.de - phone + 49 - 30 - 48 48 58 58   |
| info@pyro.de - fax + 49 - 30 - 80 94 03 52    |
+-----------------------------------------------+

From bra at fsn.hu  Thu Oct  8 12:45:10 2009
From: bra at fsn.hu (Attila Nagy)
Date: Thu Oct  8 12:45:16 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <4ACDA5EA.2010600@fsn.hu>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl> <4ACDA5EA.2010600@fsn.hu>
Message-ID: <4ACDDED0.2070707@fsn.hu>

Attila Nagy wrote:
> Hello,
>
> Pawel Jakub Dawidek wrote:
>> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>>  
>>> Backing out this change from the 8-STABLE kernel:
>>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902 
>>>
>>>
>>> makes it survive about half and hour of IMAP searching. Of course 
>>> only time will tell whether this helps in the long run, but so far 
>>> 10/10 tries succeeded to kill the machine with this method...
>>>     
>>
>> Could you try this patch:
>>
>>     http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>>   
> It seems (after running for two days) that this fixes my problem. And 
> I see that Kip has came out with a similar version (which I couldn't 
> yet test, but hope that will also do).
It seems that I was a little bit quick regarding this.
The machine just stopped with this:
last pid: 32358;  load averages:  0.01,  0.04,  0.12    up 2+06:33:56  
14:36:25
114 processes: 1 running, 112 sleeping, 1 zombie
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
Swap: 4096M Total, 15M Used, 4081M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
24025 root        1  44    0  3932K   992K vmwait  0   6:06  0.00% zpool
84190 root        1  44    0  4700K  1592K CPU1    1   4:17  0.00% top
99029 root        1  44    0  4132K  1212K nanslp  1   3:53  0.00% gstat
26317 root        1  44    0  1528K   352K piperd  1   3:38  0.00% 
readproctitl
49143    125      4  45    0 12248K  3788K sigwai  0   2:50  0.00% 
milter-greyl
39969 root        1  44    0  1536K   516K vmwait  0   2:50  0.00% supervise
40241 root        1  44    0  1536K   516K vmwait  0   2:47  0.00% supervise
44633 root        1  44    0  1536K   512K vmwait  0   2:43  0.00% supervise
43434 root        1  44    0  1536K   516K vmwait  0   2:43  0.00% supervise
50575 root        1  44    0  1536K   516K vmwait  0   2:42  0.00% supervise
45510 root        1  44    0  1536K   512K vmwait  0   2:42  0.00% supervise
58146     60      1  44    0   264M  8828K pfault  0   2:32  0.00% imapd
47526    389      6  44    0 92688K  2296K ucond   1   1:29  0.00% slapd
 5417 root        1  44    0  9396K  1680K pfault  1   1:26  0.00% sshd
13147 root        1  44    0  3340K   860K vmwait  1   0:45  0.00% syslogd
92597 root        1  44    0  9396K  1676K pfault  1   0:39  0.00% sshd
26437    125      1  44    0  6924K  1700K vmwait  0   0:33  0.00% qmgr

The above top was refreshing, but every other stuff on different ssh 
consoles (like a running zpool iostat and gstat) was frozen.
Even top stopped when I have resized the window.

From pjd at FreeBSD.org  Thu Oct  8 16:07:26 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Thu Oct  8 16:07:33 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <4ACDDED0.2070707@fsn.hu>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu>
Message-ID: <20091008160718.GB2134@garage.freebsd.pl>

On Thu, Oct 08, 2009 at 02:45:04PM +0200, Attila Nagy wrote:
> Attila Nagy wrote:
> >Hello,
> >
> >Pawel Jakub Dawidek wrote:
> >>On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
> >> 
> >>>Backing out this change from the 8-STABLE kernel:
> >>>http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902 
> >>>
> >>>
> >>>makes it survive about half and hour of IMAP searching. Of course 
> >>>only time will tell whether this helps in the long run, but so far 
> >>>10/10 tries succeeded to kill the machine with this method...
> >>>    
> >>
> >>Could you try this patch:
> >>
> >>    http://people.freebsd.org/~pjd/patches/arc.c.4.patch
> >>  
> >It seems (after running for two days) that this fixes my problem. And 
> >I see that Kip has came out with a similar version (which I couldn't 
> >yet test, but hope that will also do).
> It seems that I was a little bit quick regarding this.
> The machine just stopped with this:
> last pid: 32358;  load averages:  0.01,  0.04,  0.12    up 2+06:33:56  
> 14:36:25
> 114 processes: 1 running, 112 sleeping, 1 zombie
> CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
> Swap: 4096M Total, 15M Used, 4081M Free
> 
>  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
> 24025 root        1  44    0  3932K   992K vmwait  0   6:06  0.00% zpool
> 84190 root        1  44    0  4700K  1592K CPU1    1   4:17  0.00% top
> 99029 root        1  44    0  4132K  1212K nanslp  1   3:53  0.00% gstat
> 26317 root        1  44    0  1528K   352K piperd  1   3:38  0.00% 
> readproctitl
> 49143    125      4  45    0 12248K  3788K sigwai  0   2:50  0.00% 
> milter-greyl
> 39969 root        1  44    0  1536K   516K vmwait  0   2:50  0.00% supervise
> 40241 root        1  44    0  1536K   516K vmwait  0   2:47  0.00% supervise
> 44633 root        1  44    0  1536K   512K vmwait  0   2:43  0.00% supervise
> 43434 root        1  44    0  1536K   516K vmwait  0   2:43  0.00% supervise
> 50575 root        1  44    0  1536K   516K vmwait  0   2:42  0.00% supervise
> 45510 root        1  44    0  1536K   512K vmwait  0   2:42  0.00% supervise
> 58146     60      1  44    0   264M  8828K pfault  0   2:32  0.00% imapd
> 47526    389      6  44    0 92688K  2296K ucond   1   1:29  0.00% slapd
> 5417 root        1  44    0  9396K  1680K pfault  1   1:26  0.00% sshd
> 13147 root        1  44    0  3340K   860K vmwait  1   0:45  0.00% syslogd
> 92597 root        1  44    0  9396K  1676K pfault  1   0:39  0.00% sshd
> 26437    125      1  44    0  6924K  1700K vmwait  0   0:33  0.00% qmgr
> 
> The above top was refreshing, but every other stuff on different ssh 
> consoles (like a running zpool iostat and gstat) was frozen.
> Even top stopped when I have resized the window.

Please try Kip's patch that was committed, it changes priorities a bit,
which should help.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091008/1375ce64/attachment.pgp
From bsdgroup.md at gmail.com  Thu Oct  8 16:52:45 2009
From: bsdgroup.md at gmail.com (Rusu Silviu)
Date: Thu Oct  8 16:52:54 2009
Subject: Panic on writing new files to fusefs devices, mounted with ntfs-3g
Message-ID: <ee705d4a0910080920x25b7939fua8ce7859dd829c2e@mail.gmail.com>

Panic happens when bt client starting a download or copying a file > 4Gb.
If the file to be downloaded already exists, there are no panic, and the bt
client says file can not be found, however it is there.

Repeated on 2 machines, an iCore workstation and an pentium mobile notebook
both with 8.0 RC1.

Thank you for any suggestions.

Here are the dump from notebook:

---------------------------------------------------------------
localhost dumped core - see /var/crash/vmcore.0

Thu Oct  8 19:29:41 UTC 2009

FreeBSD localhost 8.0-RC1 FreeBSD 8.0-RC1 #0: Thu Sep 17 20:45:19 UTC
2009     root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address    = 0x0
fault code        = supervisor read, page not present
instruction pointer    = 0x20:0x0
stack pointer            = 0x28:0xe672bc44
frame pointer            = 0x28:0xe672bc68
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, def32 1, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 1128 (rtorrent)
trap number        = 12
panic: page fault
cpuid = 0
Uptime: 1m2s
Physical memory: 1002 MB
Dumping 56 MB: 41 25 9

Reading symbols from /usr/local/modules/fuse.ko...done.
Loaded symbols for /usr/local/modules/fuse.ko
#0  doadump () at pcpu.h:246
246    pcpu.h: No such file or directory.
    in pcpu.h
(kgdb) #0  doadump () at pcpu.h:246
#1  0xc08823c7 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
#2  0xc08826b9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:579
#3  0xc0bb346c in trap_fatal (frame=0xe672bc04, eva=0)
    at /usr/src/sys/i386/i386/trap.c:933
#4  0xc0bb36f0 in trap_pfault (frame=0xe672bc04, usermode=0, eva=0)
    at /usr/src/sys/i386/i386/trap.c:846
#5  0xc0bb40d5 in trap (frame=0xe672bc04) at
/usr/src/sys/i386/i386/trap.c:528
#6  0xc0b96a4b in calltrap () at /usr/src/sys/i386/i386/exception.s:165
#7  0x00000000 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

------------------------------------------------------------------------
ps -axl

Segmentation fault (core dumped)

------------------------------------------------------------------------
vmstat -s

        0 cpu context switches
        0 device interrupts
        0 software interrupts
        0 traps
        0 system calls
        0 kernel threads created
        0  fork() calls
        0 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
        0 vnode pager pageins
        0 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
       56 pages reactivated
        0 copy-on-write faults
        0 copy-on-write optimized faults
        0 zero fill pages zeroed
        0 zero fill pages prezeroed
        0 intransit blocking page faults
        0 total VM faults taken
        0 pages affected by kernel thread creation
        0 pages affected by  fork()
        0 pages affected by vfork()
        0 pages affected by rfork()
       66 pages cached
        0 pages freed
        0 pages freed by daemon
    38835 pages freed by exiting processes
     2980 pages active
     1820 pages inactive
       10 pages in VM cache
     5745 pages wired down
   241343 pages free
     4096 bytes per page
    16186 total name lookups
          cache hits (83% pos + 10% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
    pfs_nodes    20     3K       -       20  128
         GEOM   101    18K       -      633  16,32,64,128,512,1024,2048
    acpi_perf     1     1K       -        1  256
       isadev     6     1K       -        6  64
          sbp   104     9K       -      104  32,128
         cdev    11     2K       -       11  128
        sigio     1     1K       -        1  32
     filedesc    46    12K       -     1131  16,256,512
         kenv    76     7K       -       80  16,32,64,128,4096
       kqueue     0     0K       -        4  128,1024
    proc-args    21     2K       -      309  32,64,128
      ithread    64     5K       -       64  16,64,128
          agp     1     1K       -        1  16
       acpica  2120   107K       -    80694  16,32,64,128,256,512,1024
       KTRACE   100    13K       -      100  128
       linker   116    76K       -      149  16,32,256,1024,4096
        lockf    17     1K       -       17  32,64
       ip6ndp     6     1K       -        6  64,128
         temp    25   229K       -     4956
16,32,64,128,256,512,1024,2048,4096
       devbuf  2485  3683K       -     2516
16,32,64,128,256,512,1024,2048,4096
     acpitask     1     1K       -        1  1024
       module   465    30K       -      465  64,128
      CAM SIM     2     1K       -        2  128
     mtx_pool     2     8K       -        2  4096
       kbdmux     6    10K       -        6  16,256,1024,2048,4096
      subproc    93   189K       -     1178  256,4096
         proc     2     8K       -        2  4096
      session    17     2K       -       17  64
         pgrp    19     2K       -       20  64
         cred    26     3K       -     2670  64,128
      uidinfo     2     2K       -        2  64,1024
       plimit    11     3K       -      148  256
    sysctltmp     0     0K       -      228  16,32,64,128
    sysctloid  3188    96K       -     3277  16,32,64,128
       sysctl     0     0K       -      301  16,32,64
         umtx    98     7K       -       98  64
     p1003.1b     1     1K       -        1  16
         SWAP     2   277K       -        2  64
       bus-sc    76   158K       -     3769
16,32,64,128,256,512,1024,2048,4096
          bus  1184    54K       -     6575  16,32,64,128,512,1024
      devstat     8    17K       -        8  16,4096
 eventhandler    69     4K       -       69  32,64,128
         kobj   326   652K       -      396  2048
      Per-cpu     1     1K       -        1  16
         rman   194    12K       -      684  16,32,64
      acpisem    17     2K       -       17  64,128
         sbuf     0     0K       -      368
16,32,64,128,256,512,1024,2048,4096
      CAM XPT    17     7K       -       89  16,32,64,1024,2048
        stack     0     0K       -        2  128
    taskqueue    13     1K       -       13  16,64
       Unitno    11     1K       -       29  16,64
          iov     0     0K       -     4520  64,128,256
       select     8     1K       -        8  64
     ioctlops     0     0K       -      803  16,32,64,128,256,512,1024
          msg     4    25K       -        4  1024,4096
          sem     4     6K       -        4  256,512,1024,4096
          shm     1    12K       -        1
          tty    20    10K       -       22  512,2048
     mbuf_tag     0     0K       -        1  32
         ksem     1     4K       -        1  4096
        shmfd     1     4K       -        1  4096
   CAM periph     2     1K       -       18  16,32,64,128
          pcb    36    79K       -       43  16,64,512,1024,2048,4096
       soname     3     1K       -       76  16,32,128
       biobuf     4     8K       -        6  2048
     vfscache     1   512K       -        1
     vfs_hash     1   256K       -        1
       vnodes     2     1K       -        2  128
  vnodemarker     0     0K       -       26  512
        mount    94     3K       -      138  16,32,64,128,256
          BPF     5     1K       -        5  64
  ether_multi    13     1K       -       14  16,32,64
       ifaddr    48    10K       -       48  16,32,64,128,256,512,2048
        ifnet     6     6K       -        6  64,1024
        clone     5    20K       -        5  4096
       arpcom     2     1K       -        2  16
       fw_com     1     1K       -        1  64
      lltable    14     4K       -       14  128,256
  ata_generic     2     2K       -        2  1024
    ad_driver     1     1K       -        1  32
    ar_driver     0     0K       -        6  512,2048
   acd_driver     1     2K       -        1  2048
     routetbl    53   259K       -       99  16,32,64,128,256,512
         igmp     5     1K       -        5  128
      acpidev    81     3K       -       81  32
     in_multi     2     1K       -        2  128
    sctp_iter     0     0K       -        3  128
     sctp_ifn     2     1K       -        2  128
     sctp_ifa     4     1K       -        4  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -        3  16
    hostcache     1    16K       -        1
     syncache     1    72K       -        1
     ppbusdev     3     1K       -        3  128
    in6_multi     9     1K       -        9  16,256
          mld     5     1K       -        5  128
      NFS FHA     1     1K       -        1  1024
          rpc     2     5K       -        2  128,4096
audit_evclass   172     3K       -      211  16
     savedino     0     0K       -       18  256
       dirrem     4     1K       -       21  32
        mkdir     0     0K       -       12  32
       diradd    18     2K       -       25  64
     freefile    14     1K       -       19  32
     freeblks    12     3K       -       15  256
     freefrag     0     0K       -        2  32
  allocdirect     0     0K       -       25  128
    bmsafemap     0     0K       -        7  64
       newblk     1     1K       -       26  64,256
     inodedep    34   261K       -       43  128
      pagedep     3    33K       -       11  64
  ufs_dirhash    24     5K       -       24  16,32,64,128,512
    ufs_mount    12    25K       -       12  256,2048,4096
      entropy  1024    64K       -     1024  64
CAM dev queue     2     1K       -        2  64
    vm_pgdata     2    65K       -        2  64
      fw_xfer     0     0K       -        1  128
     atkbddev     2     1K       -        2  32
     firewire    11    23K       -       14  32,64,512,1024,2048,4096
         UART     3     2K       -        3  16,256,1024
       apmdev     1     1K       -        1  64
    CAM queue     6     1K       -       64  16,32
       USBdev    16     5K       -       16  32,128,1024
          USB    28     5K       -       28  16,32,64,1024
     pci_link    16     2K       -       16  32,128
       DEVFS1   100    25K       -      110  256
       DEVFS3   120    15K       -      130  128
      memdesc     1     4K       -        1  4096
     nexusdev     3     1K       -        3  16
        DEVFS    16     1K       -       17  16,64
fuse_messaging     5     2K       -       16  128,256,512
fuse_filehandles     1     1K       -        8  64
   fuse_vnode     4     1K       -        4  256

------------------------------------------------------------------------
vmstat -z

ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS
FAILURES

UMA Kegs:                 128,        0,       88,        2,
88,        0
UMA Zones:                888,        0,       88,        0,
88,        0
UMA Slabs:                284,        0,      646,       12,
1381,        0
UMA RCntSlabs:            544,        0,      195,        1,
195,        0
UMA Hash:                 128,        0,        4,       26,
4,        0
16 Bucket:                 76,        0,       27,       23,
48,        0
32 Bucket:                140,        0,       24,        4,
45,        0
64 Bucket:                268,        0,       15,       13,       61,
13
128 Bucket:               524,        0,       25,        3,     1026,
112
VM OBJECT:                136,        0,      716,       38,
12772,        0
MAP:                      140,        0,        7,       21,
7,        0
KMAP ENTRY:                72,    56180,       29,      130,
3546,        0
MAP ENTRY:                 72,        0,      370,       54,
23404,        0
DP fakepg:                 72,        0,        0,        0,
0,        0
SG fakepg:                 72,        0,        0,        0,
0,        0
mt_zone:                 2056,        0,      267,      124,
267,        0
16:                        16,        0,     2809,      236,
42604,        0
32:                        32,        0,     2454,       32,
39391,        0
64:                        64,        0,     4336,      148,
11346,        0
128:                      128,        0,     2025,       45,
10703,        0
256:                      256,        0,      588,       42,
3023,        0
512:                      512,        0,       67,        5,
3754,        0
1024:                    1024,        0,       35,      157,
1260,        0
2048:                    2048,        0,      364,       20,
625,        0
4096:                    4096,        0,      290,       15,
5429,        0
Files:                     56,        0,       74,      127,
4349,        0
TURNSTILE:                 72,        0,       99,       51,
99,        0
umtx pi:                   52,        0,        0,        0,
0,        0
MAC labels:                20,        0,        0,        0,
0,        0
PROC:                     680,        0,       43,        5,
1128,        0
THREAD:                   572,        0,       81,       17,
81,        0
SLEEPQUEUE:                32,        0,       99,       78,
99,        0
VMSPACE:                  232,        0,       21,       30,
1106,        0
cpuset:                    40,        0,        2,      182,
2,        0
audit_record:             816,        0,        0,        0,
0,        0
mbuf_packet:              256,        0,      260,      137,
306,        0
mbuf:                     256,        0,        6,      135,
126,        0
mbuf_cluster:            2048,    25600,      384,        6,
384,        0
mbuf_jumbo_page:         4096,    12800,        0,        0,
0,        0
mbuf_jumbo_9k:           9216,    19200,        0,        0,
0,        0
mbuf_jumbo_16k:         16384,    12800,        0,        0,
0,        0
mbuf_ext_refcnt:            4,        0,        0,        0,
0,        0
g_bio:                    140,        0,        0,      168,
4616,        0
ttyinq:                   152,        0,      120,       36,
480,        0
ttyoutq:                  256,        0,       64,       11,
256,        0
ata_request:              200,        0,        1,       56,
2187,        0
ata_composite:            180,        0,        0,        0,
0,        0
VNODE:                    268,        0,      461,       29,
481,        0
VNODEPOLL:                 60,        0,        0,        0,
0,        0
S VFS Cache:               72,        0,      435,       42,
903,        0
L VFS Cache:              292,        0,        0,        0,
0,        0
NAMEI:                   1024,        0,        0,        8,
6749,        0
DIRHASH:                 1024,        0,       35,        1,
35,        0
NFSMOUNT:                 520,        0,        0,        0,
0,        0
NFSNODE:                  464,        0,        0,        0,
0,        0
pipe:                     392,        0,        1,       19,
738,        0
ksiginfo:                  80,        0,       32,     1024,
32,        0
itimer:                   220,        0,        0,        0,
0,        0
KNOTE:                     68,        0,        0,      112,
4,        0
socket:                   412,    25605,       32,       22,
182,        0
unpcb:                    172,    25622,        6,       40,
17,        0
ipq:                       32,      904,        0,        0,
0,        0
udp_inpcb:                220,    25614,        2,       34,
136,        0
udpcb:                      8,    25781,        2,      201,
136,        0
tcp_inpcb:                220,    25614,       25,       29,
28,        0
tcpcb:                    632,    25602,       24,       12,
28,        0
tcptw:                     52,     5184,        1,      143,
1,        0
syncache:                 112,    15365,        0,        0,
0,        0
hostcache:                 76,    15400,        0,        0,
0,        0
tcpreass:                  20,     1690,        0,        0,
0,        0
sackhole:                  20,        0,        0,        0,
0,        0
sctp_ep:                  848,    25600,        0,        0,
0,        0
sctp_asoc:               1460,    40000,        0,        0,
0,        0
sctp_laddr:                24,    80040,        0,      145,
3,        0
sctp_raddr:               420,    80001,        0,        0,
0,        0
sctp_chunk:                96,   400000,        0,        0,
0,        0
sctp_readq:                76,   400000,        0,        0,
0,        0
sctp_stream_msg_out:       64,   400020,        0,        0,
0,        0
sctp_asconf:               24,   400055,        0,        0,
0,        0
sctp_asconf_ack:           24,   400055,        0,        0,
0,        0
ripcb:                    220,    25614,        0,        0,
0,        0
rtentry:                  108,        0,        9,       63,
9,        0
selfd:                     28,        0,       18,      236,
1793,        0
ip4flow:                   40,     4140,       23,      161,
24,        0
ip6flow:                   64,     4118,        0,        0,
0,        0
Mountpoints:              644,        0,        6,        6,
6,        0
FFS inode:                116,        0,      415,       14,
434,        0
FFS1 dinode:              128,        0,        0,        0,
0,        0
FFS2 dinode:              256,        0,      415,       20,
434,        0
SWAPMETA:                 276,   121576,        0,        0,
0,        0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq0: clk                          62481       2403
irq1: atkbd0                         143          5
irq8: rtc                           7997        307
irq11: cbb0 cbb1+*                    92          3
irq14: ata0                         1415         54
irq15: ata1                          109          4
Total                              72237       2778

------------------------------------------------------------------------
pstat -T

 74/12328 files
0M/1622M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/ad0s2b        3322440        0  3322440     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
iostat: kvm_getcptime: invalid address (0x0)
iostat: disabling CPU time statistics
             ad0
  KB/t tps  MB/s
 16.88  44  0.72

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR
CGROUP                 CBYTES                 QNUM
QBYTES        LSPID        LRPID STIME    RTIME    CTIME

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR
CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME
DTIME    CTIME

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR
CGROUP          NSEMS OTIME    CTIME


------------------------------------------------------------------------
ipcs -T

msginfo:
    msgmax:        16384    (max characters in a message)
    msgmni:           40    (# of message queues)
    msgmnb:         2048    (max characters in a message queue)
    msgtql:           40    (max # of messages in system)
    msgssz:            8    (size of a message segment)
    msgseg:         2048    (# of message segments in system)

shminfo:
    shmmax:     33554432    (max shared memory segment size)
    shmmin:            1    (min shared memory segment size)
    shmmni:          192    (max number of shared memory identifiers)
    shmseg:          128    (max shared memory segments per process)
    shmall:         8192    (max amount of shared memory in pages)

seminfo:
    semmap:           30    (# of entries in semaphore map)
    semmni:           10    (# of semaphore identifiers)
    semmns:           60    (# of semaphores in system)
    semmnu:           30    (# of undo structures in system)
    semmsl:           60    (max # of semaphores per id)
    semopm:          100    (max # of operations per semop call)
    semume:           10    (max # of undo entries per process)
    semusz:          136    (size in bytes of undo structure)
    semvmx:        32767    (semaphore maximum value)
    semaem:        16384    (adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create
Remove
        0         0         0         0         0         0
0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus
Access
        0         0         0         0         0         0
0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits
Misses
        0         0         0         0         0         0
0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses
        0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create
Remove
        0         0         0         0         0         0
0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus
Access
        0         0         0         0         0         0
0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
    69 packets sent
        32 data packets (1637 bytes)
        0 data packets (0 bytes) retransmitted
        0 data packets unnecessarily retransmitted
        0 resends initiated by MTU discovery
        14 ack-only packets (1 delayed)
        0 URG only packets
        0 window probe packets
        0 window update packets
        23 control packets
    45 packets received
        38 acks (for 1521 bytes)
        0 duplicate acks
        0 acks for unsent data
        21 packets (12121 bytes) received in-sequence
        0 completely duplicate packets (0 bytes)
        0 old duplicate packets
        0 packets with some dup. data (0 bytes duped)
        0 out-of-order packets (0 bytes)
        0 packets (0 bytes) of data after window
        0 window probes
        1 window update packet
        0 packets received after close
        0 discarded for bad checksums
        0 discarded for bad header offset fields
        0 discarded because packet too short
        0 discarded due to memory problems
    22 connection requests
    0 connection accepts
    0 bad connection attempts
    0 listen queue overflows
    0 ignored RSTs in the windows
    10 connections established (including accepts)
    3 connections closed (including 0 drops)
        0 connections updated cached RTT on close
        0 connections updated cached RTT variance on close
        0 connections updated cached ssthresh on close
    2 embryonic connections dropped
    38 segments updated rtt (of 55 attempts)
    0 retransmit timeouts
        0 connections dropped by rexmit timeout
    0 persist timeouts
        0 connections dropped by persist timeout
    0 Connections (fin_wait_2) dropped because of timeout
    0 keepalive timeouts
        0 keepalive probes sent
        0 connections dropped by keepalive
    0 correct ACK header predictions
    4 correct data packet header predictions
    0 syncache entries added
        0 retransmitted
        0 dupsyn
        0 dropped
        0 completed
        0 bucket overflow
        0 cache overflow
        0 reset
        0 stale
        0 aborted
        0 badack
        0 unreach
        0 zone failures
    0 cookies sent
    0 cookies received
    0 SACK recovery episodes
    0 segment rexmits in SACK recovery episodes
    0 byte rexmits in SACK recovery episodes
    0 SACK options (SACK blocks) received
    0 SACK options (SACK blocks) sent
    0 SACK scoreboard overflow
    0 packets with ECN CE bit set
    0 packets with ECN ECT(0) bit set
    0 packets with ECN ECT(1) bit set
    0 successful ECN handshakes
    0 times ECN reduced the congestion window
udp:
    2 datagrams received
    0 with incomplete header
    0 with bad data length field
    0 with bad checksum
    0 with no checksum
    0 dropped due to no socket
    0 broadcast/multicast datagrams undelivered
    0 dropped due to full socket buffers
    0 not for hashed pcb
    2 delivered
    2 datagrams output
    0 times multicast source filter matched
ip:
    47 total packets received
    0 bad header checksums
    0 with size smaller than minimum
    0 with data size < data length
    0 with ip length > max ip packet size
    0 with header length < data size
    0 with data length < header length
    0 with bad options
    0 with incorrect version number
    0 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    0 packets reassembled ok
    47 packets for this host
    0 packets for unknown/unsupported protocol
    0 packets forwarded (0 packets fast forwarded)
    0 packets not forwardable
    0 packets received for unknown multicast group
    0 redirects sent
    71 packets sent from this host
    0 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    0 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 tunneling packets that can't find gif
    0 datagrams with bad address in header
icmp:
    0 calls to icmp_error
    0 errors not generated in response to an icmp message
    0 messages with bad code fields
    0 messages less than the minimum length
    0 messages with bad checksum
    0 messages with bad length
    0 multicast echo requests ignored
    0 multicast timestamp requests ignored
    0 message responses generated
    0 invalid return addresses
    0 no return routes
igmp:
    0 messages received
    0 messages received with too few bytes
    0 messages received with wrong TTL
    0 messages received with bad checksum
    0 V1/V2 membership queries received
    0 V3 membership queries received
    0 membership queries received with invalid field(s)
    0 general queries received
    0 group queries received
    0 group-source queries received
    0 group-source queries dropped
    0 membership reports received
    0 membership reports received with invalid field(s)
    0 membership reports received for groups to which we belong
    0 V3 reports received without Router Alert
    0 membership reports sent
ip6:
    0 total packets received
    0 with size smaller than minimum
    0 with data size < data length
    0 with bad options
    0 with incorrect version number
    0 fragments received
    0 fragments dropped (dup or out of space)
    0 fragments dropped after timeout
    0 fragments that exceeded limit
    0 packets reassembled ok
    0 packets for this host
    0 packets forwarded
    0 packets not forwardable
    0 redirects sent
    0 packets sent from this host
    0 packets sent with fabricated ip header
    0 output packets dropped due to no bufs, etc.
    0 output packets discarded due to no route
    0 output datagrams fragmented
    0 fragments created
    0 datagrams that can't be fragmented
    0 packets that violated scope rules
    0 multicast packets which we don't join
    Mbuf statistics:
        0 one mbuf
        0 one ext mbuf
        0 two or more ext mbuf
    0 packets whose headers are not continuous
    0 tunneling packets that can't find gif
    0 packets discarded because of too many headers
    0 failures of source address selection
    Source addresses selection rule applied:
        1 first candidate
        1 same address
icmp6:
    0 calls to icmp6_error
    0 errors not generated in response to an icmp6 message
    0 errors not generated because of rate limitation
    0 messages with bad code fields
    0 messages < minimum length
    0 bad checksums
    0 messages with bad length
    Histogram of error messages to be generated:
        0 no route
        0 administratively prohibited
        0 beyond scope
        0 address unreachable
        0 port unreachable
        0 packet too big
        0 time exceed transit
        0 time exceed reassembly
        0 erroneous header field
        0 unrecognized next header
        0 unrecognized option
        0 redirect
        0 unknown
    0 message responses generated
    0 messages with too many ND options
    0 messages with bad ND options
    0 bad neighbor solicitation messages
    0 bad neighbor advertisement messages
    0 bad router solicitation messages
    0 bad router advertisement messages
    0 bad redirect messages
    0 path MTU changes
rip6:
    0 messages received
    0 checksum calculations on inbound
    0 messages with bad checksum
    0 messages dropped due to no socket
    0 multicast messages dropped due to no socket
    0 messages dropped due to full socket buffers
    0 delivered
    0 datagrams output

------------------------------------------------------------------------
netstat -m

266/272/538 mbufs in use (current/cache/total)
247/143/390/25600 mbuf clusters in use (current/cache/total/max)
260/137 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/19200 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
560K/354K/914K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs    Opkts Oerrs
Coll Drop
bge0   1500 <Link#1>      00:0b:5d:4c:4e:3f       48     0       73
0     0    0
bge0   1500 192.168.2.0   localhost               47     -       71
-     -    -
fwe0*  1500 <Link#2>      02:00:0e:90:a7:1d        0     0        0
0     0    0
fwip0  1500 <Link#3>
00:00:0e:10:02:90:a7:1d:0a:02:ff:fe:00:00:00:00        0     0        0
0     0    0
plip0  1500 <Link#4>                               0     0        0
0     0    0
lo0   16384 <Link#5>                               0     0        0
0     0    0
lo0   16384 fe80:5::1     fe80:5::1                0     -        0
-     -    -
lo0   16384 localhost     ::1                      0     -        0
-     -    -
lo0   16384 your-net      localhost                0     -        0
-     -    -

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.2.2        UGS        23       71   bge0
127.0.0.1          link#5             UH          0        0    lo0
192.168.2.0/24     link#1             U           0        0   bge0
192.168.2.5        link#5             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags
Netif Expire
::1                               ::1                           UH
lo0
fe80::%lo0/64                     link#5                        U
lo0
fe80::1%lo0                       link#5                        UHS
lo0
ff01:5::/32                       fe80::1%lo0                   U
lo0
ff02::%lo0/32                     fe80::1%lo0                   U
lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb    Proto Recv-Q Send-Q  Local Address      Foreign Address   (state)
c4a63278 tcp4       0      0 192.168.2.5.31858  95.220.50.57.62773 SYN_SENT
c4a634f0 tcp4       0      0 192.168.2.5.64367  89.185.88.245.5084 SYN_SENT
c4a63768 tcp4       0      0 192.168.2.5.45874  193.53.83.21.44156 SYN_SENT
c4a5d768 tcp4       0      0 192.168.2.5.34819  91.77.41.60.38168  SYN_SENT
c4a5d9e0 tcp4       0      5 192.168.2.5.21376  85.202.230.199.615
ESTABLISHED
c4a5dc58 tcp4       0      0 192.168.2.5.58411  95.54.221.192.4924 SYN_SENT
c4a5f000 tcp4       0      0 192.168.2.5.59844  82.193.115.223.272
ESTABLISHED
c4a5f278 tcp4       0      5 192.168.2.5.31068  78.60.226.227.3494
ESTABLISHED
c4a5f4f0 tcp4       0      0 192.168.2.5.18169  77.41.19.177.45022
ESTABLISHED
c4a5f768 tcp4       0     68 192.168.2.5.14422  92.124.160.82.2752
ESTABLISHED
c4a5f9e0 tcp4       0      0 192.168.2.5.14207  94.19.183.244.1592 SYN_SENT
c4a5fc58 tcp4       0      0 192.168.2.5.63918  80.92.96.70.44250  SYN_SENT
c4894278 tcp4       0      0 192.168.2.5.58835  83.167.115.93.3549 SYN_SENT
c48944f0 tcp4       0      0 192.168.2.5.55091  89.163.99.97.24259 SYN_SENT
c4894768 tcp4       0      0 192.168.2.5.13778  79.165.161.236.514
ESTABLISHED
c48949e0 tcp4       0      5 192.168.2.5.63650  78.37.127.202.3866
ESTABLISHED
c4a5d000 tcp4       0     34 192.168.2.5.47597  77.220.58.88.24238
ESTABLISHED
c4a5d278 tcp4       0      0 192.168.2.5.46432  82.193.97.92.35975 SYN_SENT
c4a5d4f0 tcp4    5328      0 192.168.2.5.59401  92.242.90.182.2010
ESTABLISHED
c4a67000 tcp4       0      0 192.168.2.5.49951  195.82.146.121.80  TIME_WAIT
c4893278 tcp4       0      0 *.6919             *.*                LISTEN
c48934f0 tcp4       0      0 *.21               *.*                LISTEN
c4893768 tcp6       0      0 *.21               *.*                LISTEN
c4893c58 tcp4       0      0 *.22               *.*                LISTEN
c4894000 tcp6       0      0 *.22               *.*                LISTEN
c46db6e0 udp4       0      0 *.514              *.*
c46db7bc udp6       0      0 *.514              *.*
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
c46deec8 stream      0      0 c487b430        0        0        0
/var/run/devd.pipe
c46de4b4 dgram       0      0        0 c46de8bc        0 c46de560
c46de560 dgram       0      0        0 c46de8bc        0 c46de60c
c46de60c dgram       0      0        0 c46de8bc        0        0
c46de8bc dgram       0      0 c4882b84        0 c46de4b4        0
/var/run/logpriv
c46de968 dgram       0      0 c4882c90        0        0        0
/var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/50         *.6919
tcp4  0/0/32         *.ftp
tcp6  0/0/32         *.ftp
tcp4  0/0/128        *.ssh
tcp6  0/0/128        *.ssh
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

Segmentation fault

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RC1 #0: Thu Sep 17 20:45:19 UTC 2009
    root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) M processor 1.60GHz (1600.06-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6d6  Stepping = 6

Features=0xafe9f9bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,TM,PBE>
  Features2=0x180<EST,TM2>
real memory  = 1073741824 (1024 MB)
avail memory = 1027059712 (979 MB)
kbd1 at kbdmux0
acpi0: <FUJ FJNB185> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xfc08-0xfc0b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <base peripheral> at device 0.1 (no driver attached)
pci0: <base peripheral> at device 0.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x2450-0x2457 mem
0xd8000000-0xdfffffff,0xd0000000-0xd007ffff irq 11 at device 2.0 on pci0
agp0: <Intel 82855GME (855GME GMCH) SVGA controller> on vgapci0
agp0: detected 8060k stolen memory
agp0: aperture size is 128M
vgapci1: <VGA-compatible display> mem
0xe0000000-0xe7ffffff,0xd0080000-0xd00fffff at device 2.1 on pci0
uhci0: <Intel 82801DB (ICH4) USB controller USB-A> port 0x20c0-0x20df irq 11
at device 29.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <Intel 82801DB (ICH4) USB controller USB-A> on uhci0
uhci1: <Intel 82801DB (ICH4) USB controller USB-B> port 0x20e0-0x20ff irq 11
at device 29.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <Intel 82801DB (ICH4) USB controller USB-B> on uhci1
uhci2: <Intel 82801DB (ICH4) USB controller USB-C> port 0x2400-0x241f irq 11
at device 29.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus2: <Intel 82801DB (ICH4) USB controller USB-C> on uhci2
ehci0: <Intel 82801DB/L/M (ICH4) USB 2.0 controller> mem
0xd0100000-0xd01003ff irq 11 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801DB/L/M (ICH4) USB 2.0 controller> on ehci0
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
cbb0: <O2Micro OZ711M3 PCI-CardBus Bridge> irq 11 at device 10.0 on pci1
cardbus0: <CardBus bus> on cbb0
pccard0: <16-bit PCCard bus> on cbb0
cbb0: [FILTER]
cbb1: <O2Micro OZ711M3 PCI-CardBus Bridge> irq 11 at device 10.1 on pci1
cardbus1: <CardBus bus> on cbb1
pccard1: <16-bit PCCard bus> on cbb1
cbb1: [FILTER]
pci1: <base peripheral> at device 10.2 (no driver attached)
cbb2: <O2Micro OZ711M3 PCI-CardBus Bridge> at device 10.3 on pci1
cardbus2: <CardBus bus> on cbb2
pccard2: <16-bit PCCard bus> on cbb2
cbb2: [FILTER]
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x3003> mem
0xd0200000-0xd020ffff irq 11 at device 12.0 on pci1
miibus0: <MII bus> on bge0
brgphy0: <BCM5705 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge0: Ethernet address: 00:0b:5d:4c:4e:3f
bge0: [ITHREAD]
pci1: <network> at device 13.0 (no driver attached)
fwohci0: <Texas Instruments TSB43AB21/A/AI/A-EP> mem
0xd0216000-0xd02167ff,0xd0210000-0xd0213fff irq 11 at device 14.0 on pci1
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.10 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:00:0e:10:02:90:a7:1d
fwohci0: Phy 1394a available S400, 1 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0x3e4a4000
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:00:0e:90:a7:1d
fwe0: Ethernet address: 02:00:0e:90:a7:1d
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:00:0e:10:02:90:a7:1d @ 0xfffe00000000, S400,
maxrec 2048
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: fwohci_intr_core: BUS reset
fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER
mode
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH4 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x2440-0x244f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pci0: <multimedia, audio> at device 31.5 (no driver attached)
pci0: <simple comms, generic modem> at device 31.6 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_lid0: <Control Method Lid Switch> on acpi0
acpi_acad0: <AC Adapter> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
battery1: <ACPI Control Method Battery> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model Generic PS/2 mouse, device ID 0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77b irq 7 on acpi0
ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xdc000-0xdffff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1600061907 Hz quality 800
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
firewire0: bus manager 0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ad0: 76319MB <WDC WD800UE-22HCT0 09.07D09> at ata0-master UDMA100
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
acd0: DVDR <HL-DT-ST DVD-RW GWA-4082N/CG02> at ata1-master UDMA33
GEOM: ad0s2: geometry does not match label (255h,63s != 16h,63s).
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/ad0s2a
Entropy harvesting:
 interrupts
 ethernet
 point_to_point
 kickstart
.
/dev/ad0s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2a: clean, 145310 free (3262 frags, 17756 blocks, 1.4%
fragmentation)
/dev/ad0s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2e: clean, 215633 free (33 frags, 26950 blocks, 0.0% fragmentation)
/dev/ad0s2f: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2f: clean, 2754824 free (28080 frags, 340843 blocks, 0.8%
fragmentation)
/dev/ad0s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2d: clean, 374758 free (62 frags, 46837 blocks, 0.0% fragmentation)
Starting fusefs.
fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
Starting Network: lo0 bge0.
add net default: gateway 192.168.2.2
Configuring syscons:
 blanktime
.
bge0: link state changed to UP
Oct  8 19:27:29 localhost sm-mta[1038]: My unqualified host name (localhost)
unknown; sleeping for retry
Script /etc/rc.d/sendmail interrupted

Thu Oct  8 19:27:37 UTC 2009
Oct  8 19:27:42 localhost login: ROOT LOGIN (root) ON ttyv0


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address    = 0x0
fault code        = supervisor read, page not present
instruction pointer    = 0x20:0x0
stack pointer            = 0x28:0xe672bc44
frame pointer            = 0x28:0xe672bc68
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, def32 1, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 1128 (rtorrent)
trap number        = 12
panic: page fault
cpuid = 0
Uptime: 1m2s
Physical memory: 1002 MB
Dumping 56 MB: 41 25 9

------------------------------------------------------------------------
kernel config

config: File /boot/kernel/kernel doesn't contain configuration file. Either
unsupported, or not compiled with INCLUDE_CONFIG_FILE
From fbsdlist at src.cx  Thu Oct  8 17:06:28 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Thu Oct  8 17:06:35 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <20091008160718.GB2134@garage.freebsd.pl>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl> <4ACDA5EA.2010600@fsn.hu>
	<4ACDDED0.2070707@fsn.hu> <20091008160718.GB2134@garage.freebsd.pl>
Message-ID: <ed91d4a80910081006u644c94c1x12a65c2718b3a739@mail.gmail.com>

I've tested with Kip's patch -- no lockups so far.

--Artem



On Thu, Oct 8, 2009 at 9:07 AM, Pawel Jakub Dawidek <pjd@freebsd.org> wrote:
> On Thu, Oct 08, 2009 at 02:45:04PM +0200, Attila Nagy wrote:
>> Attila Nagy wrote:
>> >Hello,
>> >
>> >Pawel Jakub Dawidek wrote:
>> >>On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>> >>
>> >>>Backing out this change from the 8-STABLE kernel:
>> >>>http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902
>> >>>
>> >>>
>> >>>makes it survive about half and hour of IMAP searching. Of course
>> >>>only time will tell whether this helps in the long run, but so far
>> >>>10/10 tries succeeded to kill the machine with this method...
>> >>>
>> >>
>> >>Could you try this patch:
>> >>
>> >> ? ?http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>> >>
>> >It seems (after running for two days) that this fixes my problem. And
>> >I see that Kip has came out with a similar version (which I couldn't
>> >yet test, but hope that will also do).
>> It seems that I was a little bit quick regarding this.
>> The machine just stopped with this:
>> last pid: 32358; ?load averages: ?0.01, ?0.04, ?0.12 ? ?up 2+06:33:56
>> 14:36:25
>> 114 processes: 1 running, 112 sleeping, 1 zombie
>> CPU: ?0.0% user, ?0.0% nice, ?0.0% system, ?0.0% interrupt, ?100% idle
>> Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
>> Swap: 4096M Total, 15M Used, 4081M Free
>>
>> ?PID USERNAME ?THR PRI NICE ? SIZE ? ?RES STATE ? C ? TIME ? WCPU COMMAND
>> 24025 root ? ? ? ?1 ?44 ? ?0 ?3932K ? 992K vmwait ?0 ? 6:06 ?0.00% zpool
>> 84190 root ? ? ? ?1 ?44 ? ?0 ?4700K ?1592K CPU1 ? ?1 ? 4:17 ?0.00% top
>> 99029 root ? ? ? ?1 ?44 ? ?0 ?4132K ?1212K nanslp ?1 ? 3:53 ?0.00% gstat
>> 26317 root ? ? ? ?1 ?44 ? ?0 ?1528K ? 352K piperd ?1 ? 3:38 ?0.00%
>> readproctitl
>> 49143 ? ?125 ? ? ?4 ?45 ? ?0 12248K ?3788K sigwai ?0 ? 2:50 ?0.00%
>> milter-greyl
>> 39969 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 516K vmwait ?0 ? 2:50 ?0.00% supervise
>> 40241 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 516K vmwait ?0 ? 2:47 ?0.00% supervise
>> 44633 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 512K vmwait ?0 ? 2:43 ?0.00% supervise
>> 43434 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 516K vmwait ?0 ? 2:43 ?0.00% supervise
>> 50575 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 516K vmwait ?0 ? 2:42 ?0.00% supervise
>> 45510 root ? ? ? ?1 ?44 ? ?0 ?1536K ? 512K vmwait ?0 ? 2:42 ?0.00% supervise
>> 58146 ? ? 60 ? ? ?1 ?44 ? ?0 ? 264M ?8828K pfault ?0 ? 2:32 ?0.00% imapd
>> 47526 ? ?389 ? ? ?6 ?44 ? ?0 92688K ?2296K ucond ? 1 ? 1:29 ?0.00% slapd
>> 5417 root ? ? ? ?1 ?44 ? ?0 ?9396K ?1680K pfault ?1 ? 1:26 ?0.00% sshd
>> 13147 root ? ? ? ?1 ?44 ? ?0 ?3340K ? 860K vmwait ?1 ? 0:45 ?0.00% syslogd
>> 92597 root ? ? ? ?1 ?44 ? ?0 ?9396K ?1676K pfault ?1 ? 0:39 ?0.00% sshd
>> 26437 ? ?125 ? ? ?1 ?44 ? ?0 ?6924K ?1700K vmwait ?0 ? 0:33 ?0.00% qmgr
>>
>> The above top was refreshing, but every other stuff on different ssh
>> consoles (like a running zpool iostat and gstat) was frozen.
>> Even top stopped when I have resized the window.
>
> Please try Kip's patch that was committed, it changes priorities a bit,
> which should help.
>
> --
> Pawel Jakub Dawidek ? ? ? ? ? ? ? ? ? ? ? http://www.wheel.pl
> pjd@FreeBSD.org ? ? ? ? ? ? ? ? ? ? ? ? ? http://www.FreeBSD.org
> FreeBSD committer ? ? ? ? ? ? ? ? ? ? ? ? Am I Evil? Yes, I Am!
>
From linimon at FreeBSD.org  Thu Oct  8 20:14:02 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Thu Oct  8 20:14:34 2009
Subject: kern/139440: [ntfs] [panic] 8.0 RC1 panics on writing large
	files to ntfs-3g mounted partitions
Message-ID: <200910082014.n98KE2C9010668@freefall.freebsd.org>

Old Synopsis: 8.0 RC1 panics on writing large files to ntfs-3g mounted partitions
New Synopsis: [ntfs] [panic] 8.0 RC1 panics on writing large files to ntfs-3g mounted partitions

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Thu Oct 8 20:13:50 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139440
From gavin at FreeBSD.org  Sat Oct 10 08:54:27 2009
From: gavin at FreeBSD.org (gavin@FreeBSD.org)
Date: Sat Oct 10 08:54:39 2009
Subject: ports/139440: [panic] 8.0 RC1 panics on writing large files to
	sysutils/fusefs-ntfs mounted partitions
Message-ID: <200910100854.n9A8sQ4T043638@freefall.freebsd.org>

Old Synopsis: [ntfs] [panic] 8.0 RC1 panics on writing large files to ntfs-3g mounted partitions
New Synopsis: [panic] 8.0 RC1 panics on writing large files to sysutils/fusefs-ntfs mounted partitions

Responsible-Changed-From-To: freebsd-fs->freebsd-ports-bugs
Responsible-Changed-By: gavin
Responsible-Changed-When: Sat Oct 10 08:53:18 UTC 2009
Responsible-Changed-Why: 
This appears to be a bug with the ntfs-3g fuse port (or fuse itself)

http://www.freebsd.org/cgi/query-pr.cgi?pr=139440
From pepelac at gmail.com  Sat Oct 10 13:23:31 2009
From: pepelac at gmail.com (Alexander Shevchenko)
Date: Sat Oct 10 13:23:44 2009
Subject: missing kstat.arcstat.l2_read_bytes
Message-ID: <8c9ae7950910100623if9c5149id9af6a5cfdbb3697@mail.gmail.com>

Good time of day!

Comparing Solaris 10u8(ZFS pool version 15) and FreeBSD 8RC(ZFS pool version
13)1 kstat output, i'v found that FreeBSD missing kernel variable
kstat.zfs.misc.arcstats.l2_read_bytes. It exists in OpenSolaris source
http://fxr.watson.org/fxr/source//common/fs/zfs/arc.c?v=OPENSOLARIS but
missing FreeBSD HEAD source
http://fxr.watson.org/fxr/source//cddl/contrib/opensolaris/uts/common/fs/zfs/arc.cIs
it a bug or a feature?

WBR,
Alexander Shevchenko
From pjd at FreeBSD.org  Sat Oct 10 15:58:57 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Sat Oct 10 15:59:03 2009
Subject: missing kstat.arcstat.l2_read_bytes
In-Reply-To: <8c9ae7950910100623if9c5149id9af6a5cfdbb3697@mail.gmail.com>
References: <8c9ae7950910100623if9c5149id9af6a5cfdbb3697@mail.gmail.com>
Message-ID: <20091010155846.GA1756@garage.freebsd.pl>

On Sat, Oct 10, 2009 at 05:23:30PM +0400, Alexander Shevchenko wrote:
> Good time of day!
> 
> Comparing Solaris 10u8(ZFS pool version 15) and FreeBSD 8RC(ZFS pool version
> 13)1 kstat output, i'v found that FreeBSD missing kernel variable
> kstat.zfs.misc.arcstats.l2_read_bytes. It exists in OpenSolaris source
> http://fxr.watson.org/fxr/source//common/fs/zfs/arc.c?v=OPENSOLARIS but
> missing FreeBSD HEAD source
> http://fxr.watson.org/fxr/source//cddl/contrib/opensolaris/uts/common/fs/zfs/arc.cIs
> it a bug or a feature?

Neither, you are simply comparing two different versions.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091010/da0bc87d/attachment.pgp
From gavin.atkinson at ury.york.ac.uk  Sun Oct 11 00:03:59 2009
From: gavin.atkinson at ury.york.ac.uk (Gavin Atkinson)
Date: Sun Oct 11 00:04:31 2009
Subject: Analysis of disk file block with ZFS checksum error
In-Reply-To: <47ACF0AE.3040802@skyrush.com>
References: <47ACD7D4.5050905@skyrush.com>
	<D6B0BBFB-D6DB-4DE1-9094-8EA69710A10C@apple.com>
	<47ACDE82.1050100@skyrush.com>
	<20080208173517.rdtobnxqg4g004c4@www.wolves.k12.mo.us>
	<47ACF0AE.3040802@skyrush.com>
Message-ID: <1202747953.27277.7.camel@buffy.york.ac.uk>

On Fri, 2008-02-08 at 17:15 -0700, Joe Peterson wrote:
> Chris Dillon wrote:
> > That is a chunk of a Mozilla Mork-format database.  Perhaps the  
> > Firefox URL history or address book from Thunderbird.
> 
> Interesting (thanks to all who recognized Mork).  I do use Firefox and
> Thunderbird, so it's feasible, but how the heck would a piece of one of
> those files find its way into 1/2 of a ZFS block in one of my mp3 files?
>    I wonder if it could have been done on write when the file was copied
> to the ZFS pool (maybe some write-caching issue?), but I thought ZFS
> would have verified the block after write.  It seems unlikely that it
> would get changed later - I never rewrote that file after the original
> copy...

Are the datestamps (Thu Jan 24 23:20:58 2008) found within the corrupt
block before or after the datestamp of the file it was found within?
i.e. was the corrupt block on the disk before or after the mp3 was
written there?

You could possibly confirm this by grepping for that datestamp in the
files in your home directory, and with the aid of 
http://developer.mozilla.org/en/docs/Mork_Structure#Rows, try to
establish exactly what the datestamp means (ie was it the time you
visited a URL, etc).

Gavin
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
From josh.carroll at gmail.com  Sun Oct 11 00:03:59 2009
From: josh.carroll at gmail.com (Josh Carroll)
Date: Sun Oct 11 00:04:42 2009
Subject: ext2 inode size patch - RE: PR kern/124621
In-Reply-To: <20081125142827.GI2042@deviant.kiev.zoral.com.ua>
References: <8cb6106e0811241129o642dcf28re4ae177c8ccbaa25@mail.gmail.com>
	<20081125140601.GH2042@deviant.kiev.zoral.com.ua>
	<8cb6106e0811250617q5fffb41exe20dfb8314fc4a9d@mail.gmail.com>
	<20081125142827.GI2042@deviant.kiev.zoral.com.ua>
Message-ID: <8cb6106e0811250657q6fdf08b0x1e94f35fd0a7ed4f@mail.gmail.com>

> I do not suggest testing. I suggest understand what inode metadata is stored
> in the added 128 bytes and evaluate whether this information can be ignored
> without dangerous consequences for filesystem consistency or user data.
>

Well, to be clear I didn't just double the size of the inode table. It
is dynamically
determined based on the data structure. I'm not a file system expert (to call me
a novice would probably be stretching it), so I'm hoping someone more versed
can chime in.

All the code does is query the data structure (specifically, the s_inode_size
field of the structure) and use that value instead of blindly assuming an inode
size of 128. I don't think it's a matter of what is done with the
extra bits, since
it's just querying the size of an already created filesystem.

Thanks,
Josh
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
From 000.fbsd at quip.cz  Sun Oct 11 04:39:13 2009
From: 000.fbsd at quip.cz (Miroslav Lachman)
Date: Sun Oct 11 04:39:21 2009
Subject: ZFS vs. df -h completely different size of filesystem
Message-ID: <4AD1616C.8060504@quip.cz>

Hi,
I am using ZFS on this machine for one year without any problems. The 
main purpose is to run jails on filesystems with ZFS quotas.

The biggest jail has 360G (322GB used) running Lighttpd. I must change 
IP for this jail, so I stop the jail, asign new IP alias on interface, 
change IP in config and start the jail again. But now the jail reports 
only 24G quota and by another accident I deleted stored files by rsync.

The strange thing is this:

root@cage ~/# df -h /vol0/jail/kurt2
Filesystem              Size    Used   Avail Capacity  Mounted on
tank/vol0/jail/kurt2     24G     19G    4.9G    80%    /vol0/jail/kurt2


root@cage ~/# zfs list | grep tank/vol0/jail/kurt2
tank/vol0/jail/kurt2                             355G  4.89G  19.2G 
/vol0/jail/kurt2
tank/vol0/jail/kurt2@manual-2009-08-18_11-04    7.29G      -   299G  -
tank/vol0/jail/kurt2@manual-2009-09-04_10-24    2.24G      -   308G  -
tank/vol0/jail/kurt2@manual-2009-09-28_11-41    21.4G      -   317G  -


root@cage ~/# zfs get -r quota tank/vol0/jail/kurt2
NAME                     PROPERTY  VALUE    SOURCE
tank/vol0/jail/kurt2     quota     360G     local

As you can see, df reports 24G size and zfs list shows 355G.

What is wrong? I tired to write some date on to this filesystem and it 
stoped after 24G were filled.

Here are more details:

root@cage ~/# zfs get all tank/vol0/jail/kurt2
NAME                  PROPERTY       VALUE                  SOURCE
tank/vol0/jail/kurt2  type           filesystem             -
tank/vol0/jail/kurt2  creation       Sat Feb 28  0:52 2009  -
tank/vol0/jail/kurt2  used           355G                   -
tank/vol0/jail/kurt2  available      4.89G                  -
tank/vol0/jail/kurt2  referenced     19.2G                  -
tank/vol0/jail/kurt2  compressratio  1.02x                  -
tank/vol0/jail/kurt2  mounted        yes                    -
tank/vol0/jail/kurt2  quota          360G                   local
tank/vol0/jail/kurt2  reservation    none                   default
tank/vol0/jail/kurt2  recordsize     128K                   default
tank/vol0/jail/kurt2  mountpoint     /vol0/jail/kurt2       inherited 
from tank/vol0
tank/vol0/jail/kurt2  sharenfs       off                    default
tank/vol0/jail/kurt2  checksum       on                     default
tank/vol0/jail/kurt2  compression    gzip                   inherited 
from tank/vol0
tank/vol0/jail/kurt2  atime          off                    inherited 
from tank
tank/vol0/jail/kurt2  devices        on                     default
tank/vol0/jail/kurt2  exec           on                     inherited 
from tank/vol0/jail
tank/vol0/jail/kurt2  setuid         on                     inherited 
from tank/vol0/jail
tank/vol0/jail/kurt2  readonly       off                    default
tank/vol0/jail/kurt2  jailed         off                    default
tank/vol0/jail/kurt2  snapdir        hidden                 default
tank/vol0/jail/kurt2  aclmode        groupmask              default
tank/vol0/jail/kurt2  aclinherit     secure                 default
tank/vol0/jail/kurt2  canmount       on                     default
tank/vol0/jail/kurt2  shareiscsi     off                    default
tank/vol0/jail/kurt2  xattr          off                    temporary
tank/vol0/jail/kurt2  copies         1                      default

root@cage ~/# df -ht zfs
Filesystem                    Size    Used   Avail Capacity  Mounted on
tank                           76G     16G     59G    22%    /tank
tank/vol0                      59G    128K     59G     0%    /vol0
tank/vol0/jail                 60G    605M     59G     1%    /vol0/jail
tank/vol0/jail/china           20G    786M     19G     4% 
/vol0/jail/china
tank/vol0/jail/costa           40G    527M     39G     1% 
/vol0/jail/costa
tank/vol0/jail/kurt2           24G     19G    4.9G    80% 
/vol0/jail/kurt2
tank/vol0/jail/tester1         20G    874M     19G     4% 
/vol0/jail/tester1
tank/vol0/jail/tvujweb         25G    494M     24G     2% 
/vol0/jail/tvujweb
tank/vol0/jail/yamaha          19G    1.2G     18G     6% 
/vol0/jail/yamaha
tank/vol0/jail/kurt2_clone    376G    317G     59G    84% 
/vol0/jail/kurt2_clone

root@cage ~/# zfs list -t filesystem
NAME                         USED  AVAIL  REFER  MOUNTPOINT
tank                         378G  59.2G  16.4G  /tank
tank/vol0                    361G  59.2G    19K  /vol0
tank/vol0/jail               361G  59.2G   605M  /vol0/jail
tank/vol0/jail/china         996M  19.0G   786M  /vol0/jail/china
tank/vol0/jail/costa         938M  39.1G   527M  /vol0/jail/costa
tank/vol0/jail/kurt2         355G  4.89G  19.2G  /vol0/jail/kurt2
tank/vol0/jail/kurt2_clone      0  59.2G   317G  /vol0/jail/kurt2_clone
tank/vol0/jail/tester1      1.23G  18.8G   874M  /vol0/jail/tester1
tank/vol0/jail/tvujweb       691M  24.3G   494M  /vol0/jail/tvujweb
tank/vol0/jail/yamaha       1.72G  18.3G  1.18G  /vol0/jail/yamaha


kurt2_clone was created as clone of snapshot 
kurt2@manual-2009-09-28_11-41 after this crazyness

Did somebody seen this error behavior?

I can get all the data from master server by rsync, but 320GB of data, 
it will take many hours. Is there any way to move the data from 
kurt2_clone to kurt2? (after I find a reason, why kurt2 is so small now)

I cannot use promote:
root@cage ~/# zfs promote tank/vol0/jail/kurt2_clone
cannot promote 'tank/vol0/jail/kurt2_clone': out of space

system: FreeBSD 7.2-RELEASE #1: Tue May  5 16:17:50 CEST 2009

root@cage ~/# zdb
tank
     version=6
     name='tank'
     state=0
     txg=4
     pool_guid=15882109565948440736
     hostid=2083894993
     hostname='cage.xxxxx.yyy'
     vdev_tree
         type='root'
         id=0
         guid=15882109565948440736
         children[0]
                 type='mirror'
                 id=0
                 guid=15485397551239445092
                 metaslab_array=14
                 metaslab_shift=32
                 ashift=9
                 asize=478648991744
                 children[0]
                         type='disk'
                         id=0
                         guid=7324230677133648509
                         path='/dev/ad4s2'
                         devid='ad:GEB531RE0B213Fs1'
                         whole_disk=0
                 children[1]
                         type='disk'
                         id=1
                         guid=16876684646990874563
                         path='/dev/ad6s2'
                         devid='ad:GEB531RE0BJ72Fs1'
                         whole_disk=0


Miroslav Lachman
From delphij at FreeBSD.org  Sun Oct 11 07:04:20 2009
From: delphij at FreeBSD.org (delphij@FreeBSD.org)
Date: Sun Oct 11 07:04:26 2009
Subject: kern/122038: [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type
	0xc7d2fab0 0
Message-ID: <200910110704.n9B74JdG011809@freefall.freebsd.org>

Synopsis: [tmpfs] [panic] tmpfs: panic: tmpfs_alloc_vp: type 0xc7d2fab0 0

Responsible-Changed-From-To: freebsd-fs->delphij
Responsible-Changed-By: delphij
Responsible-Changed-When: Sun Oct 11 07:04:00 UTC 2009
Responsible-Changed-Why: 
Take since I have committed a patch from gk@

http://www.freebsd.org/cgi/query-pr.cgi?pr=122038
From alextzfs at googlemail.com  Sun Oct 11 15:02:14 2009
From: alextzfs at googlemail.com (Alex Trull)
Date: Sun Oct 11 15:02:21 2009
Subject: zraid2 loses a single disk and becomes difficult to recover
Message-ID: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com>

Hi All,

My zraid2 has broken this morning on releng_7 zfs13.

System failed this morning and came back without pool - having lost a disk.

This is how I found the system:

  pool: fatman
 state: FAULTED
status: One or more devices could not be used because the label is missing
    or invalid.  There are insufficient replicas for the pool to continue
    functioning.
action: Destroy and re-create the pool from a backup source.
   see: http://www.sun.com/msg/ZFS-8000-5E
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    fatman      FAULTED      0     0     1  corrupted data
      raidz2    DEGRADED     0     0     6
        da2     FAULTED      0     0     0  corrupted data
        ad4     ONLINE       0     0     0
        ad6     ONLINE       0     0     0
        ad20    ONLINE       0     0     0
        ad22    ONLINE       0     0     0
        ad17    ONLINE       0     0     0
        da2     ONLINE       0     0     0
        ad10    ONLINE       0     0     0
        ad16    ONLINE       0     0     0

Initialy it complained that da3 had gone to da2 (da2 had failed and was no
longer seen)

I replaced the original da2 and bumped what was originaly da3 back up to da3
using the controllers ordering.

[root@potjie /dev]# zpool status
  pool: fatman
 state: FAULTED
status: One or more devices could not be used because the label is missing
    or invalid.  There are insufficient replicas for the pool to continue
    functioning.
action: Destroy and re-create the pool from a backup source.
   see: http://www.sun.com/msg/ZFS-8000-5E
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    fatman      FAULTED      0     0     1  corrupted data
      raidz2    ONLINE       0     0     6
        da2     UNAVAIL      0     0     0  corrupted data
        ad4     ONLINE       0     0     0
        ad6     ONLINE       0     0     0
        ad20    ONLINE       0     0     0
        ad22    ONLINE       0     0     0
        ad17    ONLINE       0     0     0
        da3     ONLINE       0     0     0
        ad10    ONLINE       0     0     0
        ad16    ONLINE       0     0     0

Issue looks very similar to this (JMR's issue) :
http://freebsd.monkey.org/freebsd-fs/200902/msg00017.html

I've tried the methods there without much result.

Using JMR's patches/debugs to see what is going on, this is what I got:

JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834
JMR: vdev_uberblock_load_done ub_txg=46475459 ub_timestamp=1255231780
JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834
JMR: vdev_uberblock_load_done ub_txg=46475458 ub_timestamp=1255231750
JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834
JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834
JMR: vdev_uberblock_load_done ub_txg=46481472 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46488653 ub_timestamp=1255246834

But JMR's patch still doesn't let me import even with a decremented txg

I then had a look around the drives using zdb and some dirty script:

[root@potjie /dev]# ls /dev/ad* /dev/da2 /dev/da3 | awk '{print "echo
"$1";zdb -l "$1" |grep txg"}' | sh
/dev/ad10
    txg=46488654
    txg=46488654
    txg=46488654
    txg=46488654
/dev/ad16
    txg=46408223 <- old TXGid ?
    txg=46408223
    txg=46408223
    txg=46408223
/dev/ad17
    txg=46408223 <- old TXGid ?
    txg=46408223
    txg=46408223
    txg=46408223
/dev/ad18 (ssd)
/dev/ad19 (spare drive, removed from pool some time ago)
    txg=0
    create_txg=0
    txg=0
    create_txg=0
    txg=0
    create_txg=0
    txg=0
    create_txg=0
/dev/ad20
    txg=46488654
    txg=46488654
    txg=46488654
    txg=46488654
/dev/ad22
    txg=46488654
    txg=46488654
    txg=46488654
    txg=46488654
/dev/ad4
    txg=46488654
    txg=46488654
    txg=46488654
    txg=46488654
/dev/ad6
    txg=46488654
    txg=46488654
    txg=46488654
    txg=46488654
/dev/da2 < new drive replaced broken da2
/dev/da3
    txg=46488654
    txg=46488654
    txg=46488654
    txg=46488654

I did not see any checksums or other issues on ad16 and ad17 previously, and
I do check regularly.

Any thoughts on what to try next ?

Regards,

Alex
From alextzfs at googlemail.com  Sun Oct 11 16:27:22 2009
From: alextzfs at googlemail.com (Alex Trull)
Date: Sun Oct 11 16:27:28 2009
Subject: zraid2 loses a single disk and becomes difficult to recover
In-Reply-To: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com>
References: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com>
Message-ID: <4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com>

Well after trying alot of things (zpool import with or without cache file in
place, etc), it randomly managed to mount the pool up, atleast, with errors
- :

zfs list output:
cannot iterate filesystems: I/O error
NAME                      USED  AVAIL  REFER  MOUNTPOINT
fatman                   1.40T  1.70T  51.2K  /fatman
fatman/backup             100G  99.5G  95.5G  /fatman/backup
fatman/jail               422G  1.70T  60.5K  /fatman/jail
fatman/jail/havnor        198G  51.7G   112G  /fatman/jail/havnor
fatman/jail/mail         19.4G  30.6G  13.0G  /fatman/jail/mail
fatman/jail/syndicate    16.6G   103G  10.5G  /fatman/jail/syndicate
fatman/jail/thirdforces   159G  41.4G  78.1G  /fatman/jail/thirdforces
fatman/jail/web          24.8G  25.2G  22.3G  /fatman/jail/web
fatman/stash              913G  1.70T   913G  /fatman/stash

(end of the dmesg)
JMR: vdev_uberblock_load_done ubbest ub_txg=46475461 ub_timestamp=1255231841
JMR: vdev_uberblock_load_done ub_txg=46481476 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46481476 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46475459 ub_timestamp=1255231780
JMR: vdev_uberblock_load_done ubbest ub_txg=46475458 ub_timestamp=1255231750
JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46481473 ub_timestamp=1255234263
JMR: vdev_uberblock_load_done ubbest ub_txg=46481472 ub_timestamp=1255234263
Solaris: WARNING: can't open objset for fatman/jail/margaret
Solaris: WARNING: can't open objset for fatman/jail/margaret
Solaris: WARNING: ZFS replay transaction error 86, dataset
fatman/jail/havnor, seq 0x25442, txtype 9

Solaris: WARNING: ZFS replay transaction error 86, dataset fatman/jail/mail,
seq 0x1e200, txtype 9

Solaris: WARNING: ZFS replay transaction error 86, dataset
fatman/jail/thirdforces, seq 0x732e3, txtype 9

[root@potjie /fatman/jail/mail]# zpool status -v
  pool: fatman
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver in progress for 0h4m, 0.83% done, 8h21m to go
config:

    NAME           STATE     READ WRITE CKSUM
    fatman         DEGRADED     0     0    34
      raidz2       DEGRADED     0     0   384
        replacing  DEGRADED     0     0     0
          da2/old  REMOVED      0    24     0
          da2      ONLINE       0     0     0  1.71G resilvered
        ad4        ONLINE       0     0     0  21.3M resilvered
        ad6        ONLINE       0     0     0  21.4M resilvered
        ad20       ONLINE       0     0     0  21.3M resilvered
        ad22       ONLINE       0     0     0  21.3M resilvered
        ad17       ONLINE       0     0     0  21.3M resilvered
        da3        ONLINE       0     0     0  21.3M resilvered
        ad10       ONLINE       0     0     1  21.4M resilvered
        ad16       ONLINE       0     0     0  21.2M resilvered
    cache
      ad18         ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        fatman/jail/margaret:<0x0>
        fatman/jail/syndicate:<0x0>
        fatman/jail/mail:<0x0>
        /fatman/jail/mail/tmp
        fatman/jail/havnor:<0x0>
        fatman/jail/thirdforces:<0x0>
        fatman/backup:<0x0>

jail/margaret & backup isn't showing up in zfs list
jail/syndicate is showing up but isn't viewable

It seems the latest content on the better-looking zfs filesystems are quite
recent.

Any thoughts about what is going on ?

I have snapshots for africa on these zfs filesystems - any suggestions on
trying to get them back ?

--
Alex

2009/10/11 Alex Trull <alextzfs@googlemail.com>

> Hi All,
>
> My zraid2 has broken this morning on releng_7 zfs13.
>
> System failed this morning and came back without pool - having lost a disk.
>
> This is how I found the system:
>
>   pool: fatman
>  state: FAULTED
> status: One or more devices could not be used because the label is missing
>     or invalid.  There are insufficient replicas for the pool to continue
>     functioning.
> action: Destroy and re-create the pool from a backup source.
>    see: http://www.sun.com/msg/ZFS-8000-5E
>  scrub: none requested
> config:
>
>     NAME        STATE     READ WRITE CKSUM
>     fatman      FAULTED      0     0     1  corrupted data
>       raidz2    DEGRADED     0     0     6
>         da2     FAULTED      0     0     0  corrupted data
>         ad4     ONLINE       0     0     0
>         ad6     ONLINE       0     0     0
>         ad20    ONLINE       0     0     0
>         ad22    ONLINE       0     0     0
>         ad17    ONLINE       0     0     0
>         da2     ONLINE       0     0     0
>         ad10    ONLINE       0     0     0
>         ad16    ONLINE       0     0     0
>
> Initialy it complained that da3 had gone to da2 (da2 had failed and was no
> longer seen)
>
> I replaced the original da2 and bumped what was originaly da3 back up to
> da3 using the controllers ordering.
>
> [root@potjie /dev]# zpool status
>   pool: fatman
>  state: FAULTED
> status: One or more devices could not be used because the label is missing
>     or invalid.  There are insufficient replicas for the pool to continue
>     functioning.
> action: Destroy and re-create the pool from a backup source.
>    see: http://www.sun.com/msg/ZFS-8000-5E
>  scrub: none requested
> config:
>
>     NAME        STATE     READ WRITE CKSUM
>     fatman      FAULTED      0     0     1  corrupted data
>       raidz2    ONLINE       0     0     6
>         da2     UNAVAIL      0     0     0  corrupted data
>         ad4     ONLINE       0     0     0
>         ad6     ONLINE       0     0     0
>         ad20    ONLINE       0     0     0
>         ad22    ONLINE       0     0     0
>         ad17    ONLINE       0     0     0
>         da3     ONLINE       0     0     0
>         ad10    ONLINE       0     0     0
>         ad16    ONLINE       0     0     0
>
> Issue looks very similar to this (JMR's issue) :
> http://freebsd.monkey.org/freebsd-fs/200902/msg00017.html
>
> I've tried the methods there without much result.
>
> Using JMR's patches/debugs to see what is going on, this is what I got:
>
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46475459 ub_timestamp=1255231780
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46475458 ub_timestamp=1255231750
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
> JMR: vdev_uberblock_load_done ub_txg=46481472 ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
> ub_timestamp=1255246834
>
> But JMR's patch still doesn't let me import even with a decremented txg
>
> I then had a look around the drives using zdb and some dirty script:
>
> [root@potjie /dev]# ls /dev/ad* /dev/da2 /dev/da3 | awk '{print "echo
> "$1";zdb -l "$1" |grep txg"}' | sh
> /dev/ad10
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad16
>     txg=46408223 <- old TXGid ?
>     txg=46408223
>     txg=46408223
>     txg=46408223
> /dev/ad17
>     txg=46408223 <- old TXGid ?
>     txg=46408223
>     txg=46408223
>     txg=46408223
> /dev/ad18 (ssd)
> /dev/ad19 (spare drive, removed from pool some time ago)
>     txg=0
>     create_txg=0
>     txg=0
>     create_txg=0
>     txg=0
>     create_txg=0
>     txg=0
>     create_txg=0
> /dev/ad20
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad22
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad4
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/ad6
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
> /dev/da2 < new drive replaced broken da2
> /dev/da3
>     txg=46488654
>     txg=46488654
>     txg=46488654
>     txg=46488654
>
> I did not see any checksums or other issues on ad16 and ad17 previously,
> and I do check regularly.
>
> Any thoughts on what to try next ?
>
> Regards,
>
> Alex
>
>
From 000.fbsd at quip.cz  Sun Oct 11 16:43:50 2009
From: 000.fbsd at quip.cz (Miroslav Lachman)
Date: Sun Oct 11 16:43:58 2009
Subject: ZFS vs. df -h completely different size of filesystem [solved]
In-Reply-To: <4AD1616C.8060504@quip.cz>
References: <4AD1616C.8060504@quip.cz>
Message-ID: <4AD20B41.3070405@quip.cz>

Solved after few hours of sleep - main problem was ... me. (working for 
more than 18 hours is not a good idea)

Miroslav Lachman wrote:

[...]

> The strange thing is this:
> 
> root@cage ~/# df -h /vol0/jail/kurt2
> Filesystem              Size    Used   Avail Capacity  Mounted on
> tank/vol0/jail/kurt2     24G     19G    4.9G    80%    /vol0/jail/kurt2
> 
> 
> root@cage ~/# zfs list | grep tank/vol0/jail/kurt2
> tank/vol0/jail/kurt2                             355G  4.89G  19.2G 
> /vol0/jail/kurt2
> tank/vol0/jail/kurt2@manual-2009-08-18_11-04    7.29G      -   299G  -
> tank/vol0/jail/kurt2@manual-2009-09-04_10-24    2.24G      -   308G  -
> tank/vol0/jail/kurt2@manual-2009-09-28_11-41    21.4G      -   317G  -
> 
> 
> root@cage ~/# zfs get -r quota tank/vol0/jail/kurt2
> NAME                     PROPERTY  VALUE    SOURCE
> tank/vol0/jail/kurt2     quota     360G     local

Directory on source server was moved and rsync started from cron 
accidentaly deleted all data.
But space remains occupied by data of snapshots! Thats why df showed 
just 24G size and not 360G. Df knows nothing about snapshots.

The fix was relatively easy.
1] make a backup of a new files (logs, configs, etc.)
2] destroy clone: zfs destroy tank/vol0/jail/kurt2_clone
3] rollback last snapshot
zfs rollback -f tank/vol0/jail/kurt2@manual-2009-09-28_11-41
4] restore backup from step 1)
5] fix rsync settings
6] run rsync as usual ;)

Miroslav Lachman
From josh at multipart-mixed.com  Sun Oct 11 18:18:44 2009
From: josh at multipart-mixed.com (Josh Carter)
Date: Sun Oct 11 18:18:51 2009
Subject: ZFS vs. df -h completely different size of filesystem [solved]
In-Reply-To: <4AD20B41.3070405@quip.cz>
References: <4AD1616C.8060504@quip.cz> <4AD20B41.3070405@quip.cz>
Message-ID: <DADFFD2B-469D-47B4-A57D-A89F6CB59B14@multipart-mixed.com>

Miroslav,

> But space remains occupied by data of snapshots! Thats why df showed  
> just 24G size and not 360G. Df knows nothing about snapshots.

Some additional factors to keep in mind when looking at ZFS and used/ 
available disk space:

- Snapshots (as you discovered).

- Compression: when compression is turned on (as you have), you can't  
know exactly how much more data will fit into the filesystem because  
it depends on how well the data compresses.

- Sparse files: "ls -h" will show you how large a file says it is; "du  
-h" and "zfs list" should show how much space is actually used. These  
will disagree on sparse files.

- Space reserved for copy-on-write: "zpool list" and "zfs list" will  
differ on available space because ZFS reserves some amount of slop  
space; truly running out of blocks is disastrous in a COW system.

In general, learn to trust "zfs list" because the traditional tools  
don't know the full story.

Best regards,
Josh

From bra at fsn.hu  Mon Oct 12 07:56:12 2009
From: bra at fsn.hu (Attila Nagy)
Date: Mon Oct 12 07:56:20 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <20091008160718.GB2134@garage.freebsd.pl>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu>
	<20091008160718.GB2134@garage.freebsd.pl>
Message-ID: <4AD2E118.2050202@fsn.hu>

Pawel Jakub Dawidek wrote:
> On Thu, Oct 08, 2009 at 02:45:04PM +0200, Attila Nagy wrote:
>   
>> Attila Nagy wrote:
>>     
>>> Hello,
>>>
>>> Pawel Jakub Dawidek wrote:
>>>       
>>>> On Fri, Oct 02, 2009 at 09:59:03AM +0200, Attila Nagy wrote:
>>>>
>>>>         
>>>>> Backing out this change from the 8-STABLE kernel:
>>>>> http://svn.freebsd.org/viewvc/base/head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c?r1=191901&r2=191902 
>>>>>
>>>>>
>>>>> makes it survive about half and hour of IMAP searching. Of course 
>>>>> only time will tell whether this helps in the long run, but so far 
>>>>> 10/10 tries succeeded to kill the machine with this method...
>>>>>    
>>>>>           
>>>> Could you try this patch:
>>>>
>>>>    http://people.freebsd.org/~pjd/patches/arc.c.4.patch
>>>>  
>>>>         
>>> It seems (after running for two days) that this fixes my problem. And 
>>> I see that Kip has came out with a similar version (which I couldn't 
>>> yet test, but hope that will also do).
>>>       
>> It seems that I was a little bit quick regarding this.
>> The machine just stopped with this:
>> last pid: 32358;  load averages:  0.01,  0.04,  0.12    up 2+06:33:56  
>> 14:36:25
>> 114 processes: 1 running, 112 sleeping, 1 zombie
>> CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
>> Mem: 536M Active, 63M Inact, 393M Wired, 8K Cache, 111M Buf
>> Swap: 4096M Total, 15M Used, 4081M Free
>>
>>  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
>> 24025 root        1  44    0  3932K   992K vmwait  0   6:06  0.00% zpool
>> 84190 root        1  44    0  4700K  1592K CPU1    1   4:17  0.00% top
>> 99029 root        1  44    0  4132K  1212K nanslp  1   3:53  0.00% gstat
>> 26317 root        1  44    0  1528K   352K piperd  1   3:38  0.00% 
>> readproctitl
>> 49143    125      4  45    0 12248K  3788K sigwai  0   2:50  0.00% 
>> milter-greyl
>> 39969 root        1  44    0  1536K   516K vmwait  0   2:50  0.00% supervise
>> 40241 root        1  44    0  1536K   516K vmwait  0   2:47  0.00% supervise
>> 44633 root        1  44    0  1536K   512K vmwait  0   2:43  0.00% supervise
>> 43434 root        1  44    0  1536K   516K vmwait  0   2:43  0.00% supervise
>> 50575 root        1  44    0  1536K   516K vmwait  0   2:42  0.00% supervise
>> 45510 root        1  44    0  1536K   512K vmwait  0   2:42  0.00% supervise
>> 58146     60      1  44    0   264M  8828K pfault  0   2:32  0.00% imapd
>> 47526    389      6  44    0 92688K  2296K ucond   1   1:29  0.00% slapd
>> 5417 root        1  44    0  9396K  1680K pfault  1   1:26  0.00% sshd
>> 13147 root        1  44    0  3340K   860K vmwait  1   0:45  0.00% syslogd
>> 92597 root        1  44    0  9396K  1676K pfault  1   0:39  0.00% sshd
>> 26437    125      1  44    0  6924K  1700K vmwait  0   0:33  0.00% qmgr
>>
>> The above top was refreshing, but every other stuff on different ssh 
>> consoles (like a running zpool iostat and gstat) was frozen.
>> Even top stopped when I have resized the window.
>>     
>
> Please try Kip's patch that was committed, it changes priorities a bit,
> which should help.
>   
My i386 machine is still alive after two days of uptime (with your 
patch, it lived for about two days, so I can't say -at least now- that 
it's OK).

The amd64 machine started to loose ARC memory again. See these:
http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem-week.png
http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png

Your patch was active between 7 and 9. You can see that the ARC size was 
somewhat constant.
On october 9, I installed Kip's modification, and ARC size started to 
decrease.
BTW, previously (before october 7) I set the arc min size to 10-15GB 
(can't remember the exact value), but now it runs with the defaults 
(only the max size is set):
vfs.zfs.arc_min: 3623878656
vfs.zfs.arc_max: 28991029248

As you can see, there are plenty of memory. This machine uses UFS as 
well (and writes it heavily), maybe that's what affects ZFS size, by 
caching a lot of stuff?

From bra at fsn.hu  Mon Oct 12 08:32:29 2009
From: bra at fsn.hu (Attila Nagy)
Date: Mon Oct 12 08:32:35 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu>
	<20091008160718.GB2134@garage.freebsd.pl> <4AD2E118.2050202@fsn.hu>
	<47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org>
Message-ID: <4AD2E998.8090409@fsn.hu>

K. Macy wrote:
>>
>> The amd64 machine started to loose ARC memory again. See these:
>> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem-week.png
>> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png
>>
>> Your patch was active between 7 and 9. You can see that the ARC size 
>> was somewhat constant.
>> On october 9, I installed Kip's modification, and ARC size started to 
>> decrease.
>> BTW, previously (before october 7) I set the arc min size to 10-15GB 
>> (can't remember the exact value), but now it runs with the defaults 
>> (only the max size is set):
>> vfs.zfs.arc_min: 3623878656
>> vfs.zfs.arc_max: 28991029248
>>
>> As you can see, there are plenty of memory. This machine uses UFS as 
>> well (and writes it heavily), maybe that's what affects ZFS size, by 
>> caching a lot of stuff?
>>
>
> Currently, the inactive page queue will grow until ARC is shrunk to 
> arc_min. 
>
>
> I think I'll probably spend some time making the ARC play better with 
> the page cache this week. Unfortunately, under heavy memory pressure 
> when competing with UFS the ARC will degrade to LRU, but I think that 
> is still an improvement over the current static sizing with low and 
> high water marks.
Will setting ARC's minimum size help here? (I will try)

For me, it's OK, and I think it's generally not a problem, if somebody 
uses UFS as well.
Is it possible to merge the memory management of the two, or they are 
completely different beasts?

Thanks,

From kmacy at freebsd.org  Mon Oct 12 08:55:18 2009
From: kmacy at freebsd.org (K. Macy)
Date: Mon Oct 12 08:55:52 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <4AD2E118.2050202@fsn.hu>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu>
	<20091008160718.GB2134@garage.freebsd.pl> <4AD2E118.2050202@fsn.hu>
Message-ID: <47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org>

>
> The amd64 machine started to loose ARC memory again. See these:
> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem- 
> week.png
> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png
>
> Your patch was active between 7 and 9. You can see that the ARC size  
> was somewhat constant.
> On october 9, I installed Kip's modification, and ARC size started  
> to decrease.
> BTW, previously (before october 7) I set the arc min size to 10-15GB  
> (can't remember the exact value), but now it runs with the defaults  
> (only the max size is set):
> vfs.zfs.arc_min: 3623878656
> vfs.zfs.arc_max: 28991029248
>
> As you can see, there are plenty of memory. This machine uses UFS as  
> well (and writes it heavily), maybe that's what affects ZFS size, by  
> caching a lot of stuff?
>

Currently, the inactive page queue will grow until ARC is shrunk to  
arc_min.


I think I'll probably spend some time making the ARC play better with  
the page cache this week. Unfortunately, under heavy memory pressure  
when competing with UFS the ARC will degrade to LRU, but I think that  
is still an improvement over the current static sizing with low and  
high water marks.

-Kip
From kmacy at freebsd.org  Mon Oct 12 09:12:20 2009
From: kmacy at freebsd.org (K. Macy)
Date: Mon Oct 12 09:12:25 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <4AD2E998.8090409@fsn.hu>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl>
	<4ACDA5EA.2010600@fsn.hu> <4ACDDED0.2070707@fsn.hu>
	<20091008160718.GB2134@garage.freebsd.pl> <4AD2E118.2050202@fsn.hu>
	<47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org>
	<4AD2E998.8090409@fsn.hu>
Message-ID: <A9EED983-8382-4C8B-BFB9-5A4CA1D62A37@freebsd.org>

>>
>> Currently, the inactive page queue will grow until ARC is shrunk to  
>> arc_min.
>>
>> I think I'll probably spend some time making the ARC play better  
>> with the page cache this week. Unfortunately, under heavy memory  
>> pressure when competing with UFS the ARC will degrade to LRU, but I  
>> think that is still an improvement over the current static sizing  
>> with low and high water marks.
> Will setting ARC's minimum size help here? (I will try)
>
> For me, it's OK, and I think it's generally not a problem, if  
> somebody uses UFS as well.
> Is it possible to merge the memory management of the two, or they  
> are completely different beasts?

To some degree it is possible to merge them by partly backing the arc  
from the page cache. This would allow for a fair amount of auto- 
tuning. However, it isn't possibly to completely merge them - the ARC  
is a virtual device block cache, UFS caches pages in the vm object  
based on their offset in the file. Thus it would never be possible to  
use blocks in the ARC for mmap - for applications that dirty file  
backed mmaped memory it will always be necessary to have two copies of  
the page, one in the vm object for the file and one in ZFS that maps  
to a block offset. It all makes a bit more sense if you understand  
that ZFS is a transactional object store with a posix file system  
interface.
From bugmaster at FreeBSD.org  Mon Oct 12 11:06:52 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Oct 12 11:07:57 2009
Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org
Message-ID: <200910121106.n9CB6pkQ036372@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o kern/139407  fs         [smbfs] [panic] smb mount causes system crash if remot
o kern/139363  fs         [nfs] diskless root nfs mount from non FreeBSD server 
o kern/138790  fs         [zfs] ZFS ceases caching when mem demand is high
o kern/138524  fs         [msdosfs] disks and usb flashes/cards with Russian lab
o kern/138421  fs         [ufs] [patch] remove UFS label limitations
o kern/138367  fs         [tmpfs] [panic] 'panic: Assertion pages > 0 failed' wh
o kern/138202  fs         mount_msdosfs(1) see only 2Gb
o kern/138109  fs         [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2f
f kern/137037  fs         [zfs] [hang] zfs rollback on root causes FreeBSD to fr
o kern/136968  fs         [ufs] [lor] ufs/bufwait/ufs (open)
o kern/136945  fs         [ufs] [lor] filedesc structure/ufs (poll)
o kern/136944  fs         [ffs] [lor] bufwait/snaplk (fsync)
o kern/136873  fs         [ntfs] Missing directories/files on NTFS volume
o kern/136865  fs         [nfs] [patch] NFS exports atomic and on-the-fly atomic
o kern/136470  fs         [nfs] Cannot mount / in read-only, over NFS
o kern/135594  fs         [zfs] Single dataset unresponsive with Samba
o kern/135546  fs         [zfs] zfs.ko module doesn't ignore zpool.cache filenam
o kern/135469  fs         [ufs] [panic] kernel crash on md operation in ufs_dirb
o kern/135050  fs         [zfs] ZFS clears/hides disk errors on reboot
o kern/134491  fs         [zfs] Hot spares are rather cold...
o kern/133980  fs         [panic] [ffs] panic: ffs_valloc: dup alloc
o kern/133676  fs         [smbfs] [panic] umount -f'ing a vnode-based memory dis
o kern/133614  fs         [smbfs] [panic] panic: ffs_truncate: read-only filesys
o kern/133174  fs         [msdosfs] [patch] msdosfs must support utf-encoded int
f kern/133150  fs         [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w
o kern/132960  fs         [ufs] [panic] panic:ffs_blkfree: freeing free frag
o kern/132597  fs         [tmpfs] [panic] tmpfs-related panic while interrupting
o kern/132397  fs         reboot causes filesystem corruption (failure to sync b
o kern/132331  fs         [ufs] [lor] LOR ufs and syncer
o kern/132237  fs         [msdosfs] msdosfs has problems to read MSDOS Floppy
o kern/132145  fs         [panic] File System Hard Crashes
o kern/131995  fs         [nfs] Failure to mount NFSv4 server
o kern/131441  fs         [unionfs] [nullfs] unionfs and/or nullfs not combineab
o kern/131360  fs         [nfs] poor scaling behavior of the NFS server under lo
o kern/131342  fs         [nfs] mounting/unmounting of disks causes NFS to fail
o bin/131341   fs         makefs: error "Bad file descriptor"  on the mount poin
o kern/131086  fs         [ext2fs] [patch] mkfs.ext2 creates rotten partition
o kern/130979  fs         [smbfs] [panic] boot/kernel/smbfs.ko
o kern/130920  fs         [msdosfs] cp(1) takes 100% CPU time while copying file
o kern/130229  fs         [iconv] usermount fails on fs that need iconv
o kern/130210  fs         [nullfs] Error by check nullfs
o kern/129760  fs         [nfs] after 'umount -f' of a stale NFS share FreeBSD l
o kern/129488  fs         [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: 
o kern/129231  fs         [ufs] [patch] New UFS mount (norandom) option - mostly
o kern/129152  fs         [panic] non-userfriendly panic when trying to mount(8)
o kern/129059  fs         [zfs] [patch] ZFS bootloader whitelistable via WITHOUT
f kern/128829  fs         smbd(8) causes periodic panic on 7-RELEASE
f kern/128173  fs         [ext2fs] ls gives "Input/output error" on mounted ext3
o kern/127659  fs         [tmpfs] tmpfs memory leak
o kern/127420  fs         [gjournal] [panic] Journal overflow on gmirrored gjour
o kern/127029  fs         [panic] mount(8): trying to mount a write protected zi
o kern/126287  fs         [ufs] [panic] Kernel panics while mounting an UFS file
s kern/125738  fs         [zfs] [request] SHA256 acceleration in ZFS
f kern/125536  fs         [ext2fs] ext 2 mounts cleanly but fails on commands li
f kern/124621  fs         [ext3] [patch] Cannot mount ext2fs partition
f bin/124424   fs         [zfs] zfs(8): zfs list -r shows strange snapshots' siz
o kern/123939  fs         [msdosfs] corrupts new files
o kern/122380  fs         [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash
o bin/122172   fs         [fs]: amd(8) automount daemon dies on 6.3-STABLE i386,
o kern/122047  fs         [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / 
o bin/121898   fs         [nullfs] pwd(1)/getcwd(2) fails with Permission denied
o bin/121779   fs         [ufs] snapinfo(8) (and related tools?) only work for t
o bin/121366   fs         [zfs] [patch] Automatic disk scrubbing from periodic(8
o bin/121072   fs         [smbfs] mount_smbfs(8) cannot normally convert the cha
f kern/120991  fs         [panic] [fs] [snapshot] System crashes when manipulati
o kern/120483  fs         [ntfs] [patch] NTFS filesystem locking changes
o kern/120482  fs         [ntfs] [patch] Sync style changes between NetBSD and F
f kern/119735  fs         [zfs] geli + ZFS + samba starting on boot panics 7.0-B
o kern/118912  fs         [2tb] disk sizing/geometry problem with large array
o kern/118713  fs         [minidump] [patch] Display media size required for a k
o bin/118249   fs         mv(1): moving a directory changes its mtime
o kern/118107  fs         [ntfs] [panic] Kernel panic when accessing a file at N
o bin/117315   fs         [smbfs] mount_smbfs(8) and related options can't mount
o kern/117314  fs         [ntfs] Long-filename only NTFS fs'es cause kernel pani
o kern/117158  fs         [zfs] zpool scrub causes panic if geli vdevs detach on
o bin/116980   fs         [msdosfs] [patch] mount_msdosfs(8) resets some flags f
o kern/116913  fs         [ffs] [panic] ffs_blkfree: freeing free block
p kern/116608  fs         [msdosfs] [patch] msdosfs fails to check mount options
o kern/116583  fs         [ffs] [hang] System freezes for short time when using 
o kern/116170  fs         [panic] Kernel panic when mounting /tmp
o kern/115645  fs         [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex
o bin/115361   fs         [zfs] mount(8) gets into a state where it won't set/un
o kern/114955  fs         [cd9660] [patch] [request] support for mask,dirmask,ui
o kern/114847  fs         [ntfs] [patch] [request] dirmask support for NTFS ala 
o kern/114676  fs         [ufs] snapshot creation panics: snapacct_ufs2: bad blo
o bin/114468   fs         [patch] [request] add -d option to umount(8) to detach
o kern/113852  fs         [smbfs] smbfs does not properly implement DFS referral
o bin/113838   fs         [patch] [request] mount(8): add support for relative p
o bin/113049   fs         [patch] [request] make quot(8) use getopt(3) and show 
o kern/112658  fs         [smbfs] [patch] smbfs and caching problems (resolves b
f usb/112640   fs         [ext2fs] [hang] Kernel freezes when writing a file to 
o kern/111843  fs         [msdosfs] Long Names of files are incorrectly created 
o kern/111782  fs         [ufs] dump(8) fails horribly for large filesystems
s bin/111146   fs         [2tb] fsck(8) fails on 6T filesystem
o kern/109024  fs         [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not 
o kern/109010  fs         [msdosfs] can't mv directory within fat32 file system
o bin/107829   fs         [2TB] fdisk(8): invalid boundary checking in fdisk / w
o kern/106030  fs         [ufs] [panic] panic in ufs from geom when a dead disk 
o kern/105093  fs         [ext2fs] [patch] ext2fs on read-only media cannot be m
o kern/104406  fs         [ufs] Processes get stuck in "ufs" state under persist
o kern/104133  fs         [ext2fs] EXT2FS module corrupts EXT2/3 filesystems
o kern/103035  fs         [ntfs] Directories in NTFS mounted disc images appear 
o kern/101324  fs         [smbfs] smbfs sometimes not case sensitive when it's s
o kern/99290   fs         [ntfs] mount_ntfs ignorant of cluster sizes
o kern/97377   fs         [ntfs] [patch] syntax cleanup for ntfs_ihash.c
o kern/95222   fs         [iso9660] File sections on ISO9660 level 3 CDs ignored
o kern/94849   fs         [ufs] rename on UFS filesystem is not atomic
o kern/94769   fs         [ufs] Multiple file deletions on multi-snapshotted fil
o kern/94733   fs         [smbfs] smbfs may cause double unlock
o kern/93942   fs         [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D
o kern/92272   fs         [ffs] [hang] Filling a filesystem while creating a sna
f kern/91568   fs         [ufs] [panic] writing to UFS/softupdates DVD media in 
o kern/91134   fs         [smbfs] [patch] Preserve access and modification time 
a kern/90815   fs         [smbfs] [patch] SMBFS with character conversions somet
o kern/89991   fs         [ufs] softupdates with mount -ur causes fs UNREFS
o kern/88657   fs         [smbfs] windows client hang when browsing a samba shar
o kern/88266   fs         [smbfs] smbfs does not implement UIO_NOCOPY and sendfi
o kern/87859   fs         [smbfs] System reboot while umount smbfs.
o kern/86587   fs         [msdosfs] rm -r /PATH fails with lots of small files
o kern/85326   fs         [smbfs] [panic] saving a file via samba to an overquot
o kern/84589   fs         [2TB] 5.4-STABLE unresponsive during background fsck 2
o kern/80088   fs         [smbfs] Incorrect file time setting on NTFS mounted vi
o kern/77826   fs         [ext2fs] ext2fs usb filesystem will not mount RW
o kern/73484   fs         [ntfs] Kernel panic when doing `ls` from the client si
o bin/73019    fs         [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino
o kern/71774   fs         [ntfs] NTFS cannot "see" files on a WinXP filesystem
o kern/68978   fs         [panic] [ufs] crashes with failing hard disk, loose po
o kern/65920   fs         [nwfs] Mounted Netware filesystem behaves strange
o kern/65901   fs         [smbfs] [patch] smbfs fails fsx write/truncate-down/tr
o kern/61503   fs         [smbfs] mount_smbfs does not work as non-root
o kern/55617   fs         [smbfs] Accessing an nsmb-mounted drive via a smb expo
o kern/51685   fs         [hang] Unbounded inode allocation causes kernel to loc
o kern/51583   fs         [nullfs] [patch] allow to work with devices and socket
o kern/36566   fs         [smbfs] System reboot with dead smb mount and umount
o kern/18874   fs         [2TB] 32bit NFS servers export wrong negative values t

135 problems total.

From avg at icyb.net.ua  Mon Oct 12 12:33:18 2009
From: avg at icyb.net.ua (Andriy Gapon)
Date: Mon Oct 12 12:33:24 2009
Subject: zfs boot vs loader
Message-ID: <4AD3220B.5030502@icyb.net.ua>


I am using ZFS for boot-and-root filesystem. gptzfsboot is used for bootstrapping,
if that matters.
Maybe the following is some kind of a pilot error, but I think that I see a
problem with selecting a different kernel using loader prompt.
That is, I escape to a loader prompt from boot menu. There I do 'unload'. Then I
'load' a different kernel, opensolaris.ko and zfs.ko. Then I 'boot'. Then root
mounting inevitably fails because ZFS root filesystem can not be found.

This is an unexpected result for me.
I wonder if something important for ZFS get unloaded when I unload preloaded
kernel and modules. Perhaps I have to manually re-load zpool.cache?

This happens with recent CURRENT, amd64.
What's strange is that I think that I can switch kernels from loader prompt
without any problems when using RELENG_7 and UFS-boot-plus-ZFS-root approach.

-- 
Andriy Gapon
From kraduk at googlemail.com  Mon Oct 12 15:16:16 2009
From: kraduk at googlemail.com (krad)
Date: Mon Oct 12 15:16:22 2009
Subject: zfs boot vs loader
In-Reply-To: <4AD3220B.5030502@icyb.net.ua>
References: <4AD3220B.5030502@icyb.net.ua>
Message-ID: <d36406630910120755k464eddbaled03810a723c9e30@mail.gmail.com>

2009/10/12 Andriy Gapon <avg@icyb.net.ua>

>
> I am using ZFS for boot-and-root filesystem. gptzfsboot is used for
> bootstrapping,
> if that matters.
> Maybe the following is some kind of a pilot error, but I think that I see a
> problem with selecting a different kernel using loader prompt.
> That is, I escape to a loader prompt from boot menu. There I do 'unload'.
> Then I
> 'load' a different kernel, opensolaris.ko and zfs.ko. Then I 'boot'. Then
> root
> mounting inevitably fails because ZFS root filesystem can not be found.
>
> This is an unexpected result for me.
> I wonder if something important for ZFS get unloaded when I unload
> preloaded
> kernel and modules. Perhaps I have to manually re-load zpool.cache?
>
> This happens with recent CURRENT, amd64.
> What's strange is that I think that I can switch kernels from loader prompt
> without any problems when using RELENG_7 and UFS-boot-plus-ZFS-root
> approach.
>
> --
> Andriy Gapon
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
>


before you boot the kernels check the values of the following are still set

eg

vfs.root.mountfrom="zfs:system/root"
vfs.root.mountfrom.options=rw
From simon at comsys.ntu-kpi.kiev.ua  Mon Oct 12 16:20:02 2009
From: simon at comsys.ntu-kpi.kiev.ua (Andrey Simonenko)
Date: Mon Oct 12 16:20:09 2009
Subject: kern/136865: NFS exports atomic and on-the-fly atomic updates
Message-ID: <200910121620.n9CGK2Q8020292@freefall.freebsd.org>

The following reply was made to PR kern/136865; it has been noted by GNATS.

From: Andrey Simonenko <simon@comsys.ntu-kpi.kiev.ua>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/136865: NFS exports atomic and on-the-fly atomic updates
Date: Mon, 12 Oct 2009 19:19:15 +0300

 Updated version nfse-20091012 can be used on 9.0-CURRENT and 8.0-RC1.
 with experimental and regular NFS server.
 
 Can be downloaded here http://comsys.ntu-kpi.kiev.ua/~simon/nfse/
 MD5 (nfse-20091012.tar.bz2) = 3562d449406ac0a728928de5d0583884
 
 List of changes:
 
     * Command "flush" can be combined with "add" commands.
 
     * Added new command "clear" -- clear configuration for the given
       pathname.
 
     * Added new command "show" -- show current configuration.
 
     * Now nfse recognizes shadowed mount points.  Manual page nfs.exports(5)
       was updated and contains information how nfse(8) works with mount
       points.
 
     * Now all settings (export specifications for file systems, export
       specifications for NFSv4 root directory and WebNFS settings) are
       updated atomically.  There is one exception for WebNFS settings
       for new exported file system, but this exception can be suppressed
       by the administrator.
 
     * Added support for NFSv4 root directory configuration
       (atomic and on-the-fly atomic updates are supported).
 
     * Added support for experimental NFS server.
 
     * Several mistakes were corrected, some parts were optimized and/or
       simplified.
 
     * Support for obsolete options was removed.
 
     * mountd(8) was renamed to nfse(8) (actually nfse uses only modified
       RPC and XDR related code from mountd), exports(5) was renamed to
       nfs.exports(5).
 
     * This distribution contains changes for FreeBSD 9.0-CURRENT and 8.0-RC1.
 
 List of open-questions:
 
 1. WebNFS settings cannot be updated by "-c command", it is necessary to
    discuss semantics for experimental NFS server.
 
 2. WebNFS settings are not protected by any lock in NFS server.
 
 3. NFSv4 root directory settings are not protected by any lock in NFS server.
 
 4. It is possible to make better integration of nfs_export.c with NFS server
    (see nfs_export.c:nfse_fs_check() and nfs_export.c:nfse_rd_check()),
    it is necessary to discuss this.
From swell.k at gmail.com  Mon Oct 12 17:36:50 2009
From: swell.k at gmail.com (Anonymous)
Date: Mon Oct 12 17:36:57 2009
Subject: zfs boot vs loader
In-Reply-To: <4AD3220B.5030502@icyb.net.ua> (Andriy Gapon's message of "Mon,
	12 Oct 2009 15:33:15 +0300")
References: <4AD3220B.5030502@icyb.net.ua>
Message-ID: <86ws30ikfw.fsf@gmail.com>

Andriy Gapon <avg@icyb.net.ua> writes:

> I am using ZFS for boot-and-root filesystem. gptzfsboot is used for bootstrapping,
> if that matters.
> Maybe the following is some kind of a pilot error, but I think that I see a
> problem with selecting a different kernel using loader prompt.
> That is, I escape to a loader prompt from boot menu. There I do 'unload'. Then I
> 'load' a different kernel, opensolaris.ko and zfs.ko. Then I 'boot'. Then root
> mounting inevitably fails because ZFS root filesystem can not be found.
>

You can just type

    OK boot kernel.old

here to switch to kernel.old (or any other one). No need to manually
load modules if you specified them in loader.conf.

> This is an unexpected result for me.
> I wonder if something important for ZFS get unloaded when I unload preloaded
> kernel and modules. Perhaps I have to manually re-load zpool.cache?

Does following help?

    OK load -t blah /boot/zfs/zpool.cache

>
> This happens with recent CURRENT, amd64.
> What's strange is that I think that I can switch kernels from loader prompt
> without any problems when using RELENG_7 and UFS-boot-plus-ZFS-root approach.
From alextzfs at googlemail.com  Mon Oct 12 19:49:40 2009
From: alextzfs at googlemail.com (Alex Trull)
Date: Mon Oct 12 19:49:48 2009
Subject: zraid2 loses a single disk and becomes difficult to recover
In-Reply-To: <4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com>
References: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com>
	<4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com>
Message-ID: <4d98b5320910121249q36c68b8vf63ec27cf4bb94c9@mail.gmail.com>

I managed to cleanly recover all critical data by cloning the most recent
snapshots of all my filesystems (which worked even for those filesystems
that had disappeared from 'zfs list') - and moving back to ufs2

The 'live' filesystems since the snapshots had pretty much gone corrupt.

Intereresting note is that even if I promoted those clones - if the system
was rebooted the contents of the snapshots became gobbledygooked (invalid
byte sequence errors on numerous files).

As it stands I managed to recover 100% of the data, so I'm out the woods.

How does a dual-parity array lose its mind when only one disk is lost ?
Might it have been related to the old TXGid I found on ad16 and ad17 ?

--
Alex

2009/10/11 Alex Trull <alextzfs@googlemail.com>

> Well after trying alot of things (zpool import with or without cache file
> in place, etc), it randomly managed to mount the pool up, atleast, with
> errors - :
>
> zfs list output:
> cannot iterate filesystems: I/O error
> NAME                      USED  AVAIL  REFER  MOUNTPOINT
> fatman                   1.40T  1.70T  51.2K  /fatman
> fatman/backup             100G  99.5G  95.5G  /fatman/backup
> fatman/jail               422G  1.70T  60.5K  /fatman/jail
> fatman/jail/havnor        198G  51.7G   112G  /fatman/jail/havnor
> fatman/jail/mail         19.4G  30.6G  13.0G  /fatman/jail/mail
> fatman/jail/syndicate    16.6G   103G  10.5G  /fatman/jail/syndicate
> fatman/jail/thirdforces   159G  41.4G  78.1G  /fatman/jail/thirdforces
> fatman/jail/web          24.8G  25.2G  22.3G  /fatman/jail/web
> fatman/stash              913G  1.70T   913G  /fatman/stash
>
> (end of the dmesg)
> JMR: vdev_uberblock_load_done ubbest ub_txg=46475461
> ub_timestamp=1255231841
> JMR: vdev_uberblock_load_done ub_txg=46481476 ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46481476
> ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46475459
> ub_timestamp=1255231780
> JMR: vdev_uberblock_load_done ubbest ub_txg=46475458
> ub_timestamp=1255231750
> JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46481473
> ub_timestamp=1255234263
> JMR: vdev_uberblock_load_done ubbest ub_txg=46481472
> ub_timestamp=1255234263
> Solaris: WARNING: can't open objset for fatman/jail/margaret
> Solaris: WARNING: can't open objset for fatman/jail/margaret
> Solaris: WARNING: ZFS replay transaction error 86, dataset
> fatman/jail/havnor, seq 0x25442, txtype 9
>
> Solaris: WARNING: ZFS replay transaction error 86, dataset
> fatman/jail/mail, seq 0x1e200, txtype 9
>
> Solaris: WARNING: ZFS replay transaction error 86, dataset
> fatman/jail/thirdforces, seq 0x732e3, txtype 9
>
> [root@potjie /fatman/jail/mail]# zpool status -v
>   pool: fatman
>  state: DEGRADED
> status: One or more devices has experienced an error resulting in data
>     corruption.  Applications may be affected.
> action: Restore the file in question if possible.  Otherwise restore the
>     entire pool from backup.
>    see: http://www.sun.com/msg/ZFS-8000-8A
>  scrub: resilver in progress for 0h4m, 0.83% done, 8h21m to go
> config:
>
>     NAME           STATE     READ WRITE CKSUM
>     fatman         DEGRADED     0     0    34
>       raidz2       DEGRADED     0     0   384
>         replacing  DEGRADED     0     0     0
>           da2/old  REMOVED      0    24     0
>           da2      ONLINE       0     0     0  1.71G resilvered
>         ad4        ONLINE       0     0     0  21.3M resilvered
>         ad6        ONLINE       0     0     0  21.4M resilvered
>         ad20       ONLINE       0     0     0  21.3M resilvered
>         ad22       ONLINE       0     0     0  21.3M resilvered
>         ad17       ONLINE       0     0     0  21.3M resilvered
>         da3        ONLINE       0     0     0  21.3M resilvered
>         ad10       ONLINE       0     0     1  21.4M resilvered
>         ad16       ONLINE       0     0     0  21.2M resilvered
>     cache
>       ad18         ONLINE       0     0     0
>
> errors: Permanent errors have been detected in the following files:
>
>         fatman/jail/margaret:<0x0>
>         fatman/jail/syndicate:<0x0>
>         fatman/jail/mail:<0x0>
>         /fatman/jail/mail/tmp
>         fatman/jail/havnor:<0x0>
>         fatman/jail/thirdforces:<0x0>
>         fatman/backup:<0x0>
>
> jail/margaret & backup isn't showing up in zfs list
> jail/syndicate is showing up but isn't viewable
>
> It seems the latest content on the better-looking zfs filesystems are quite
> recent.
>
> Any thoughts about what is going on ?
>
> I have snapshots for africa on these zfs filesystems - any suggestions on
> trying to get them back ?
>
> --
> Alex
>
> 2009/10/11 Alex Trull <alextzfs@googlemail.com>
>
> Hi All,
>>
>> My zraid2 has broken this morning on releng_7 zfs13.
>>
>> System failed this morning and came back without pool - having lost a
>> disk.
>>
>> This is how I found the system:
>>
>>   pool: fatman
>>  state: FAULTED
>> status: One or more devices could not be used because the label is missing
>>
>>     or invalid.  There are insufficient replicas for the pool to continue
>>     functioning.
>> action: Destroy and re-create the pool from a backup source.
>>    see: http://www.sun.com/msg/ZFS-8000-5E
>>  scrub: none requested
>> config:
>>
>>     NAME        STATE     READ WRITE CKSUM
>>     fatman      FAULTED      0     0     1  corrupted data
>>       raidz2    DEGRADED     0     0     6
>>         da2     FAULTED      0     0     0  corrupted data
>>         ad4     ONLINE       0     0     0
>>         ad6     ONLINE       0     0     0
>>         ad20    ONLINE       0     0     0
>>         ad22    ONLINE       0     0     0
>>         ad17    ONLINE       0     0     0
>>         da2     ONLINE       0     0     0
>>         ad10    ONLINE       0     0     0
>>         ad16    ONLINE       0     0     0
>>
>> Initialy it complained that da3 had gone to da2 (da2 had failed and was no
>> longer seen)
>>
>> I replaced the original da2 and bumped what was originaly da3 back up to
>> da3 using the controllers ordering.
>>
>> [root@potjie /dev]# zpool status
>>   pool: fatman
>>  state: FAULTED
>> status: One or more devices could not be used because the label is missing
>>
>>     or invalid.  There are insufficient replicas for the pool to continue
>>     functioning.
>> action: Destroy and re-create the pool from a backup source.
>>    see: http://www.sun.com/msg/ZFS-8000-5E
>>  scrub: none requested
>> config:
>>
>>     NAME        STATE     READ WRITE CKSUM
>>     fatman      FAULTED      0     0     1  corrupted data
>>       raidz2    ONLINE       0     0     6
>>         da2     UNAVAIL      0     0     0  corrupted data
>>         ad4     ONLINE       0     0     0
>>         ad6     ONLINE       0     0     0
>>         ad20    ONLINE       0     0     0
>>         ad22    ONLINE       0     0     0
>>         ad17    ONLINE       0     0     0
>>         da3     ONLINE       0     0     0
>>         ad10    ONLINE       0     0     0
>>         ad16    ONLINE       0     0     0
>>
>> Issue looks very similar to this (JMR's issue) :
>> http://freebsd.monkey.org/freebsd-fs/200902/msg00017.html
>>
>> I've tried the methods there without much result.
>>
>> Using JMR's patches/debugs to see what is going on, this is what I got:
>>
>> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
>> ub_timestamp=1255246834
>> JMR: vdev_uberblock_load_done ub_txg=46475459 ub_timestamp=1255231780
>> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
>> ub_timestamp=1255246834
>> JMR: vdev_uberblock_load_done ub_txg=46475458 ub_timestamp=1255231750
>> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
>> ub_timestamp=1255246834
>> JMR: vdev_uberblock_load_done ub_txg=46481473 ub_timestamp=1255234263
>> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
>> ub_timestamp=1255246834
>> JMR: vdev_uberblock_load_done ub_txg=46481472 ub_timestamp=1255234263
>> JMR: vdev_uberblock_load_done ubbest ub_txg=46488653
>> ub_timestamp=1255246834
>>
>> But JMR's patch still doesn't let me import even with a decremented txg
>>
>> I then had a look around the drives using zdb and some dirty script:
>>
>> [root@potjie /dev]# ls /dev/ad* /dev/da2 /dev/da3 | awk '{print "echo
>> "$1";zdb -l "$1" |grep txg"}' | sh
>> /dev/ad10
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>> /dev/ad16
>>     txg=46408223 <- old TXGid ?
>>     txg=46408223
>>     txg=46408223
>>     txg=46408223
>> /dev/ad17
>>     txg=46408223 <- old TXGid ?
>>     txg=46408223
>>     txg=46408223
>>     txg=46408223
>> /dev/ad18 (ssd)
>> /dev/ad19 (spare drive, removed from pool some time ago)
>>     txg=0
>>     create_txg=0
>>     txg=0
>>     create_txg=0
>>     txg=0
>>     create_txg=0
>>     txg=0
>>     create_txg=0
>> /dev/ad20
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>> /dev/ad22
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>> /dev/ad4
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>> /dev/ad6
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>> /dev/da2 < new drive replaced broken da2
>> /dev/da3
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>>     txg=46488654
>>
>> I did not see any checksums or other issues on ad16 and ad17 previously,
>> and I do check regularly.
>>
>> Any thoughts on what to try next ?
>>
>> Regards,
>>
>> Alex
>>
>>
>
From eugen at kuzbass.ru  Tue Oct 13 04:17:07 2009
From: eugen at kuzbass.ru (Eugene Grosbein)
Date: Tue Oct 13 04:17:23 2009
Subject: FreeBSD Status Reports April - September, 2009
In-Reply-To: <20091011175428.GA3626@freefall.freebsd.org>
References: <20091011175428.GA3626@freefall.freebsd.org>
Message-ID: <20091013035504.GA72620@svzserv.kemerovo.su>

On Sun, Oct 11, 2009 at 05:54:29PM +0000, Daniel Gerzo wrote:


> FreeBSD/ZFS
> 
>    Contact: Pawel Dawidek <pjd@FreeBSD.org>
> 
>    We believe that the ZFS file system is now production-ready in FreeBSD
>    8.0. Most (if not all) reported bugs were fixed and ZFS is no longer
>    tagged as experimental. There is also ongoing work in Perforce to bring
>    the latest ZFS version (v19) to FreeBSD.

That's great news. However, my experience says me not place dot-zero
relese under business-critical tasks and load.

What about status of ZFS in 7.2? Does 7.2 contain the same ZFS code?

Eugene Grosbein
From avg at icyb.net.ua  Tue Oct 13 10:46:40 2009
From: avg at icyb.net.ua (Andriy Gapon)
Date: Tue Oct 13 10:46:53 2009
Subject: zfs boot vs loader
In-Reply-To: <86ws30ikfw.fsf@gmail.com>
References: <4AD3220B.5030502@icyb.net.ua> <86ws30ikfw.fsf@gmail.com>
Message-ID: <4AD45A8D.4070605@icyb.net.ua>


Thanks to all who replied!
I think that I figured it out with your help.

'unload' does indeed unload zpool.cache and the latter is needed for proper zfs
root mounting.
'boot /other/kernel' executes some loader scripts and zpool.cache gets loaded in
this case (because of the settings in defaults/loader.conf).
When doing manual 'load xxx' and then just 'boot' no loader scripts are executed
and zpool.cache does not get loaded. In this case one has to load it manually
using the suggested 'load -t xxx' approach.


-- 
Andriy Gapon
From linimon at FreeBSD.org  Tue Oct 13 13:40:26 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Tue Oct 13 13:40:37 2009
Subject: kern/139564: [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of
	shutdown
Message-ID: <200910131340.n9DDePKE063200@freefall.freebsd.org>

Old Synopsis: zfs - 8.0-RC1 - Fatal trap 12 at end of shutdown
New Synopsis: [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdown

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Tue Oct 13 13:40:02 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139564
From fjwcash at gmail.com  Tue Oct 13 15:57:07 2009
From: fjwcash at gmail.com (Freddie Cash)
Date: Tue Oct 13 15:57:14 2009
Subject: FreeBSD Status Reports April - September, 2009
In-Reply-To: <20091013035504.GA72620@svzserv.kemerovo.su>
References: <20091011175428.GA3626@freefall.freebsd.org>
	<20091013035504.GA72620@svzserv.kemerovo.su>
Message-ID: <b269bc570910130832g7c902eedxd34027e1fca1c54a@mail.gmail.com>

On Mon, Oct 12, 2009 at 8:55 PM, Eugene Grosbein <eugen@kuzbass.ru> wrote:

> On Sun, Oct 11, 2009 at 05:54:29PM +0000, Daniel Gerzo wrote:
>  > FreeBSD/ZFS
> >
> >    Contact: Pawel Dawidek <pjd@FreeBSD.org>
> >
> >    We believe that the ZFS file system is now production-ready in FreeBSD
> >    8.0. Most (if not all) reported bugs were fixed and ZFS is no longer
> >    tagged as experimental. There is also ongoing work in Perforce to
> bring
> >    the latest ZFS version (v19) to FreeBSD.
>
> That's great news. However, my experience says me not place dot-zero
> relese under business-critical tasks and load.
>
> What about status of ZFS in 7.2? Does 7.2 contain the same ZFS code?
>
> 7.2-RELEASE includes ZFSv6.

7-STABLE (after the release of 7.2) includes ZFSv13.

8.0-RELEASE will include ZFSv13.

IOW, 8.0 and 7.3 will have roughly the same ZFS code, although I believe the
plan is to leave it marked as "experimental" in 7.x and only remove that
warning in 8.x.



-- 
Freddie Cash
fjwcash@gmail.com
From pjd at FreeBSD.org  Wed Oct 14 06:21:16 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Wed Oct 14 06:21:24 2009
Subject: zraid2 loses a single disk and becomes difficult to recover
In-Reply-To: <4d98b5320910121249q36c68b8vf63ec27cf4bb94c9@mail.gmail.com>
References: <4d98b5320910110741w794c154cs22b527485c1938da@mail.gmail.com>
	<4d98b5320910110927o62f8f588r9acdeb40a19587ea@mail.gmail.com>
	<4d98b5320910121249q36c68b8vf63ec27cf4bb94c9@mail.gmail.com>
Message-ID: <20091014062107.GB1696@garage.freebsd.pl>

On Mon, Oct 12, 2009 at 08:49:37PM +0100, Alex Trull wrote:
> I managed to cleanly recover all critical data by cloning the most recent
> snapshots of all my filesystems (which worked even for those filesystems
> that had disappeared from 'zfs list') - and moving back to ufs2
> 
> The 'live' filesystems since the snapshots had pretty much gone corrupt.
> 
> Intereresting note is that even if I promoted those clones - if the system
> was rebooted the contents of the snapshots became gobbledygooked (invalid
> byte sequence errors on numerous files).
> 
> As it stands I managed to recover 100% of the data, so I'm out the woods.

I'm glad to hear that.

> How does a dual-parity array lose its mind when only one disk is lost ?
> Might it have been related to the old TXGid I found on ad16 and ad17 ?

Yes, definiately. For some reason ZFS didn't update txg on those two
disks, so at this point you were running without parity. The problem is
that ZFS didn't start resilver automatically and also didn't report this
situation properly. I think I saw this in the past. Running 'zpool scrub'
on this pool will trigger resilver. There must be a bug. I tried to
reproduce it by modifying the code not to update txg on one of the
components. There are three places where this can happen on sytem
crash/power failure and I tried all of them - no luck, ZFS was able to
recover properly.

It would be good idea to run 'zpool scrub' on regular basis, even if
only to see if it won't trigger resilver (it can be stopped after few
minutes with 'zpool scrub -s'). Of course it is adviced to run full
scrub from time to time.

Do you have this pool around still?

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091014/81be3a82/attachment.pgp
From pjd at FreeBSD.org  Wed Oct 14 06:28:35 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Wed Oct 14 06:28:41 2009
Subject: zfs boot vs loader
In-Reply-To: <4AD45A8D.4070605@icyb.net.ua>
References: <4AD3220B.5030502@icyb.net.ua> <86ws30ikfw.fsf@gmail.com>
	<4AD45A8D.4070605@icyb.net.ua>
Message-ID: <20091014062827.GC1696@garage.freebsd.pl>

On Tue, Oct 13, 2009 at 01:46:37PM +0300, Andriy Gapon wrote:
> 
> Thanks to all who replied!
> I think that I figured it out with your help.
> 
> 'unload' does indeed unload zpool.cache and the latter is needed for proper zfs
> root mounting.
> 'boot /other/kernel' executes some loader scripts and zpool.cache gets loaded in
> this case (because of the settings in defaults/loader.conf).
> When doing manual 'load xxx' and then just 'boot' no loader scripts are executed
> and zpool.cache does not get loaded. In this case one has to load it manually
> using the suggested 'load -t xxx' approach.

Could you repeat what you were doing but with vfs.zfs.debug set to 1
from the loader? You should see some messages about missing file, maybe
we should do them visible always and not after increasing debug level?

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091014/a180eeb9/attachment.pgp
From solon at pyro.de  Wed Oct 14 16:47:39 2009
From: solon at pyro.de (Solon Lutz)
Date: Wed Oct 14 16:47:45 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <673550066.20091013133544@pyro.de>
References: <20091004185738.GI1660@garage.freebsd.pl>
	<165582258.20091012155536@pyro.de>
	<20091012193636.GC1762@garage.freebsd.pl>
	<756365088.20091013005202@pyro.de>
	<20091013055754.GA3197@garage.freebsd.pl>
	<90685589.20091013092418@pyro.de>
	<20091013072712.GA1597@garage.freebsd.pl>
	<12910471099.20091013095322@pyro.de>
	<20091013075511.GC1597@garage.freebsd.pl>
	<1433853337.20091013100348@pyro.de>
	<20091013082116.GE1597@garage.freebsd.pl>
	<673550066.20091013133544@pyro.de>
Message-ID: <473227349.20091014184731@pyro.de>

>>> >> > As I understand the values 13462283 and 13462284 were showing errors?

>>> >> > Yes, just try to import the pool and set vfs.zfs.maxtxg back to zero
>>> >> > afterwards.

>>> >> 13462283 and 13462284 were showing errors.
>>> >> vfs.zfs.maxtxg was at -1 before I changed it.

>>> > So does it work now?

>>> radium# zpool import -f temp
>>> cannot iterate filesystems: I/O error
>>> radium# ll /temp/
>>> ls: backup: Input/output error
>>> total 9
>>> drwxr-xr-x  2 root   wheel   2 May 16  2007 audio
>>> drwxr-xr-x  2 root   wheel   2 May 16  2007 misc
>>> drwxr-xr-x  2 root   wheel   2 May 16  2007 video
>>> drwxr-xr-x  8 dhcpd  dhcpd  11 Jul  8 10:36 www
>>> radium# ll /temp/audio/
>>> total 0
>>> radium# ll /temp/misc/
>>> total 0
>>> radium# zfs unmount temp
>>> cannot iterate filesystems: I/O error

>> Can you show 'zpool status'?

>> It might be worth trying to go even more in the past with maxtxg.

> sysctl vfs.zfs.maxtxg=13462260

>   pool: temp
>  state: ONLINE
> status: One or more devices has experienced an error resulting in data
>         corruption.  Applications may be affected.
> action: Restore the file in question if possible.  Otherwise restore the
>         entire pool from backup.
>    see: http://www.sun.com/msg/ZFS-8000-8A
>  scrub: none requested
> config:

>         NAME        STATE     READ WRITE CKSUM
>         temp        ONLINE       0     0     9
>           da0       ONLINE       0     0    36

> errors: Permanent errors have been detected in the following files:

>         temp:<0x0>
>         temp:<0x49681>
>         temp:<0x499a4>
>         temp:<0x495fd>
>         temp/space1:<0x0>
>         temp/space2:<0x0>
>         temp/space3:<0x0>
>         temp/space4:<0x0>
>         temp/space5:<0x0>

I just tried it with more TXGs, even with a jump of -300, but it always gives
an "cannot iterate filesystems: I/O error" error if I try to import the pool.

Also because of mounting the pool, the TXg has gone up to 13445935 from
initially 13462284.

solon


From linimon at FreeBSD.org  Wed Oct 14 20:10:56 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Wed Oct 14 20:11:08 2009
Subject: kern/139597: [patch] [tmpfs] tmpfs initializes va_gen but
	doesn't update it on vnode change
Message-ID: <200910142010.n9EKAute076100@freefall.freebsd.org>

Synopsis: [patch] [tmpfs] tmpfs initializes va_gen but doesn't update it on vnode change

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Wed Oct 14 20:10:47 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139597
From pjd at FreeBSD.org  Wed Oct 14 21:05:41 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Wed Oct 14 21:05:48 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <473227349.20091014184731@pyro.de>
References: <756365088.20091013005202@pyro.de>
	<20091013055754.GA3197@garage.freebsd.pl>
	<90685589.20091013092418@pyro.de>
	<20091013072712.GA1597@garage.freebsd.pl>
	<12910471099.20091013095322@pyro.de>
	<20091013075511.GC1597@garage.freebsd.pl>
	<1433853337.20091013100348@pyro.de>
	<20091013082116.GE1597@garage.freebsd.pl>
	<673550066.20091013133544@pyro.de>
	<473227349.20091014184731@pyro.de>
Message-ID: <20091014210528.GC1727@garage.freebsd.pl>

On Wed, Oct 14, 2009 at 06:47:31PM +0200, Solon Lutz wrote:
> >   pool: temp
> >  state: ONLINE
> > status: One or more devices has experienced an error resulting in data
> >         corruption.  Applications may be affected.
> > action: Restore the file in question if possible.  Otherwise restore the
> >         entire pool from backup.
> >    see: http://www.sun.com/msg/ZFS-8000-8A
> >  scrub: none requested
> > config:
> 
> >         NAME        STATE     READ WRITE CKSUM
> >         temp        ONLINE       0     0     9
> >           da0       ONLINE       0     0    36
> 
> > errors: Permanent errors have been detected in the following files:
> 
> >         temp:<0x0>
> >         temp:<0x49681>
> >         temp:<0x499a4>
> >         temp:<0x495fd>
> >         temp/space1:<0x0>
> >         temp/space2:<0x0>
> >         temp/space3:<0x0>
> >         temp/space4:<0x0>
> >         temp/space5:<0x0>
> 
> I just tried it with more TXGs, even with a jump of -300, but it always gives
> an "cannot iterate filesystems: I/O error" error if I try to import the pool.
> 
> Also because of mounting the pool, the TXg has gone up to 13445935 from
> initially 13462284.

We can try to turn off checksum verification entirely, but it will most
likely just panic your system.

If you want to do this, edit
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h file and change
ZIO_CHECKSUM_EQUAL() macro to something like this:

#define	ZIO_CHECKSUM_EQUAL(zc1, zc2)	(1)

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091014/643df3f6/attachment.pgp
From grarpamp at gmail.com  Wed Oct 14 22:02:34 2009
From: grarpamp at gmail.com (grarpamp)
Date: Wed Oct 14 22:02:47 2009
Subject: ZFS repeatable reboot 8.0-RC1
Message-ID: <d2e731a10910141502q513b0e6fq7d4d8de31b05d1e7@mail.gmail.com>

Hi.

I'm running i386 on i386, single P4 cpu, 1GiB RAM.
SiI 3114 -> SATA [single disk] -> GELI [AES-128] -> ZFS [sha256]
Straight RELENG_8 as of cvsup Oct 12 14:49:00 aka 8.0-RC1 plus.
ZFS pool is at v13, ZFS fs is at v3.
Hardware seems stable.
The only modification to config defaults is:
 loader.conf.local: vfs.zfs.arc_max=100663296

After boot -v, geli, zpool import, xf86, browser, etc my mem looks like:
 Mem: 33M Active, 22M Inact, 105M Wired, 676K Cache, 37M Buf, 827M Free
When putting load on ZFS it usually grows to about:
 Mem: 95M Active, 22M Inact, 302M Wired, 468K Cache, 37M Buf, 569M Free
Ls -l in one of the dirs takes 10min plus and I get:
  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   11 root     171 ki31     0K     8K RUN     21:24 47.27% idle
 1092 user      76    0 77328K 76116K zio->i   3:25 37.89% ls
  802 root      -8    -     0K     8K geli:w   1:42  8.98% g_eli[0] ad6
    9 root      -8    -     0K   128K arc_re   0:23  4.88% {arc_reclaim_thre}
I did not watch these during rm.

I have 1 parent dir holding 4 subdirs. The file count in each subdir
is respectively: 256363, 254086, 256017, 178054
Two thirds of files are about 14KiB in size, not many are more than
a few MiB nor less than 1KiB though a third are 1 byte.

I issue rm -r <parent_dir> and after maybe 30 seconds the machine
reboots. No syslog, panic or console messages. Dmesg from the prior
boot is still present in ram to prove kernel didn't emit any message.
memtest86 passes. There are maybe 10 seconds of complete GUI hangup
before the reboot occurs.

I also see it when make release'ing. Usually during what I _think_
is distributeworld or rolling up the tarballs under /R.

This is a big repeatable problem. How can I debug or fix it? Can
someone else create some mega sized dirs as above and replicate?

Thanks.
From grarpamp at gmail.com  Wed Oct 14 22:55:55 2009
From: grarpamp at gmail.com (grarpamp)
Date: Wed Oct 14 22:56:01 2009
Subject: ZFS repeatable reboot 8.0-RC1
Message-ID: <d2e731a10910141555o604e0abewf8c5b5bdc948f4a0@mail.gmail.com>

Happened again :) So some more notes...

Watched it this time, it jumped from about 29xMiB straight to 366MiB,
then hung and rebooted itself.

The first zpool import after reboot takes about a minute more than
the usual ten seconds to import. And uses up to maybe 125MiB instead
of maybe 40MiB. Could be ZFS fscking itself? Second and further
manual reboots are all normal speed and mem use.

The fs is fine, I can do all sorts of normal ops on it. Even rm -r
<that_dir>, ^C after a few seconds, and repeat ad nauseum until the
tree is removed. Only continuous rm -r reboots.

I have 1 more GiB RAM I can put in.

Sorry to break threads, I'm not on list.
From solon at pyro.de  Wed Oct 14 23:11:44 2009
From: solon at pyro.de (Solon Lutz)
Date: Wed Oct 14 23:11:51 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <20091014210528.GC1727@garage.freebsd.pl>
References: <756365088.20091013005202@pyro.de>
	<20091013055754.GA3197@garage.freebsd.pl>
	<90685589.20091013092418@pyro.de>
	<20091013072712.GA1597@garage.freebsd.pl>
	<12910471099.20091013095322@pyro.de>
	<20091013075511.GC1597@garage.freebsd.pl>
	<1433853337.20091013100348@pyro.de>
	<20091013082116.GE1597@garage.freebsd.pl>
	<673550066.20091013133544@pyro.de>
	<473227349.20091014184731@pyro.de>
	<20091014210528.GC1727@garage.freebsd.pl>
Message-ID: <572088116.20091015011137@pyro.de>

>> I just tried it with more TXGs, even with a jump of -300, but it always gives
>> an "cannot iterate filesystems: I/O error" error if I try to import the pool.

>> Also because of mounting the pool, the TXg has gone up to 13445935 from
>> initially 13462284.

> We can try to turn off checksum verification entirely, but it will most
> likely just panic your system.

> If you want to do this, edit
> sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h file and change
> ZIO_CHECKSUM_EQUAL() macro to something like this:

> #define ZIO_CHECKSUM_EQUAL(zc1, zc2)    (1)

Yes, it paniced as soon as I tried zpool import. Can you tell where ZFS is taking
the information from, that there are I/O errors? Is this based on checksums or is
there some kind of I/O-error-flag?


solon



From jhell at DataIX.net  Thu Oct 15 01:35:59 2009
From: jhell at DataIX.net (jhell)
Date: Thu Oct 15 01:36:05 2009
Subject: ZFS repeatable reboot 8.0-RC1
In-Reply-To: <d2e731a10910141555o604e0abewf8c5b5bdc948f4a0@mail.gmail.com>
References: <d2e731a10910141555o604e0abewf8c5b5bdc948f4a0@mail.gmail.com>
Message-ID: <alpine.BSF.2.00.0910142107230.72893@qvzrafvba.5c.ybpny>


On Wed, 14 Oct 2009 18:55, grarpamp@ wrote:
> Happened again :) So some more notes...
>
> Watched it this time, it jumped from about 29xMiB straight to 366MiB,
> then hung and rebooted itself.
>
> The first zpool import after reboot takes about a minute more than
> the usual ten seconds to import. And uses up to maybe 125MiB instead
> of maybe 40MiB. Could be ZFS fscking itself? Second and further
> manual reboots are all normal speed and mem use.
>
> The fs is fine, I can do all sorts of normal ops on it. Even rm -r
> <that_dir>, ^C after a few seconds, and repeat ad nauseum until the
> tree is removed. Only continuous rm -r reboots.
>
> I have 1 more GiB RAM I can put in.
>
> Sorry to break threads, I'm not on list.
>

Your machine is starving! what you can do to improve upon performance of 
the pool is add a separate disk to the machine in which you can configure 
your ZIL(ZFS Intent Log) and a Cache. Those two things can improve greatly 
upon performance and reduce eating up so much ram that your system starts 
to starve itself. Add that other 1G of RAM if you have it just sitting 
around then put it to good use it will certainly help.

The disk that your doing your remove operation on ? is that being done on 
a ZFS GELI ?

PS: You can use thumb drives as caches and intent logs but beware I have 
had them stop and disappear in my system to only appear again on reboot.



-- 

  ;; dataix.net!jhell         2048R/89D8547E 2009-09-30
  ;; BSD since FreeBSD 4.2    Linux since Slackware 2.1
  ;; 85EF E26B 07BB 3777 76BE  B12A 9057 8789 89D8 547E

From grarpamp at gmail.com  Thu Oct 15 06:47:02 2009
From: grarpamp at gmail.com (grarpamp)
Date: Thu Oct 15 06:47:09 2009
Subject: ZFS repeatable reboot 8.0-RC1
In-Reply-To: <alpine.BSF.2.00.0910142107230.72893@qvzrafvba.5c.ybpny>
References: <d2e731a10910141555o604e0abewf8c5b5bdc948f4a0@mail.gmail.com>
	<alpine.BSF.2.00.0910142107230.72893@qvzrafvba.5c.ybpny>
Message-ID: <d2e731a10910142347n52f60bey8e9058d2b76dccf7@mail.gmail.com>

> Your machine is starving!

How can this be, there is over 500MiB free ram at all times? I'm
running literally no userland apps other than X and xterms when it
reboots.

I think I may be hitting some limit with this 366MiB and reboot bit.
How can I tell what my kernel limits are on this platform?
Don't I have to limit ARC to ( kern + kern headroom + ARC = kern
addressablility limit ) or something like that?
Anything I should use for that besides sysctl -a and vmstat -z?
I thought I looked at my wired history before using zfs and set
arc_max to 96MiB so wired wouldn't even get close 512.

> what you can do to improve upon performance of the pool

Performance is not the problem. Yes, it's dog slow, but it's usable,
sortof :) The issue is that it's rebooting spontaneously. No OS
should do that. Though unlike a userland process that the kernel
kills when out of ram, I don't know how the kernel would recover
when its own processes bloat up.

I expect slow performance with this setup. Especially if I'm blowing
out some cache somewhere. Take UFS2 with dirhash for example. If
the size of the directory inode is much bigger than vfs.ufs.dirhash_maxmem,
it just slows down to spindle speed ... not reboot, no big deal.

> is add a separate disk to the machine in which you can configure
> your ZIL(ZFS Intent Log) and a Cache. Those two things can ...
> reduce eating up so much ram that your system starts to starve
> itself.

Reduce ram?, how so? I already have a ZIL in the main pool by
default, presumably using just as much ram as a separate one would,
so no need for a separate log. Similarly for cache, which is in
core in the default case. They just help speed if on 'faster than
spindles' devices.

I suppose I could as easily set vfs.zfs.zil_disable=0 as a test if
it wouldn't risk loss of the entire pool.

> Add that other 1G of RAM

Isn't that a game of whack a mole? What happens when I rm a dir
with 1M files in it? Add more ram? 2M files? ...

> The disk that your doing your remove operation on ? is that being
> done on a ZFS GELI ?

As mentioned, yes.

> PS: You can use thumb drives as caches and intent logs

I would presume their bit error rate is higher than platters. There
was a time when SSD's meant RAM drives, not flash drives.


# vmstat -m | egrep -i 'requests|zfs|zil|zio|arc|solaris|geom|eli'
         Type InUse MemUse HighUse Requests  Size(s)
         GEOM   208    27K       -     1708  16,32,64,128,512,1024,2048,4096
      solaris 145673 135033K       -  3413390
16,32,64,128,256,512,1024,2048,4096
     eli data     8     5K       -    23628  32,256,512,1024,2048,4096

# vmstat -z | egrep -i 'requests|zfs|zil|zio|arc|solaris|geom|eli'
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
zio_cache:                596,        0,        0,     4596,   489861,        0
arc_buf_hdr_t:            136,        0,     9367,       29,    15769,        0
arc_buf_t:                 40,        0,     2128,      264,    19641,        0
zil_lwb_cache:            176,        0,        3,       85,      488,        0
zfs_znode_cache:          232,        0,    19298,      473,    62000,        0

# sysctl -a vfs.zfs kstat.zfs
vfs.zfs.arc_meta_limit: 25165824
vfs.zfs.arc_meta_used: 39459076
vfs.zfs.mdcomp_disable: 0
vfs.zfs.arc_min: 16777216
vfs.zfs.arc_max: 100663296
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.block_cap: 256
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 1
vfs.zfs.recover: 0
vfs.zfs.txg.synctime: 5
vfs.zfs.txg.timeout: 30
vfs.zfs.scrub_limit: 10
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.ramp_rate: 2
vfs.zfs.vdev.time_shift: 6
vfs.zfs.vdev.min_pending: 4
vfs.zfs.vdev.max_pending: 35
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.version.zpl: 3
vfs.zfs.version.vdev_boot: 1
vfs.zfs.version.spa: 13
vfs.zfs.version.dmu_backup_stream: 1
vfs.zfs.version.dmu_backup_header: 2
vfs.zfs.version.acl: 1
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
kstat.zfs.misc.arcstats.hits: 102514
kstat.zfs.misc.arcstats.misses: 12662
kstat.zfs.misc.arcstats.demand_data_hits: 8150
kstat.zfs.misc.arcstats.demand_data_misses: 741
kstat.zfs.misc.arcstats.demand_metadata_hits: 94364
kstat.zfs.misc.arcstats.demand_metadata_misses: 11921
kstat.zfs.misc.arcstats.prefetch_data_hits: 0
kstat.zfs.misc.arcstats.prefetch_data_misses: 0
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0
kstat.zfs.misc.arcstats.mru_hits: 49617
kstat.zfs.misc.arcstats.mru_ghost_hits: 2511
kstat.zfs.misc.arcstats.mfu_hits: 52897
kstat.zfs.misc.arcstats.mfu_ghost_hits: 1193
kstat.zfs.misc.arcstats.deleted: 1429
kstat.zfs.misc.arcstats.recycle_miss: 5314
kstat.zfs.misc.arcstats.mutex_miss: 0
kstat.zfs.misc.arcstats.evict_skip: 3645
kstat.zfs.misc.arcstats.hash_elements: 9362
kstat.zfs.misc.arcstats.hash_elements_max: 9363
kstat.zfs.misc.arcstats.hash_collisions: 8135
kstat.zfs.misc.arcstats.hash_chains: 2042
kstat.zfs.misc.arcstats.hash_chain_max: 5
kstat.zfs.misc.arcstats.p: 57449472
kstat.zfs.misc.arcstats.c: 100663296
kstat.zfs.misc.arcstats.c_min: 16777216
kstat.zfs.misc.arcstats.c_max: 100663296
kstat.zfs.misc.arcstats.size: 99728132
kstat.zfs.misc.arcstats.hdr_size: 1273504
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.vdev_cache_stats.delegations: 961
kstat.zfs.misc.vdev_cache_stats.hits: 8593
kstat.zfs.misc.vdev_cache_stats.misses: 3377
From valin at buchlovice.org  Thu Oct 15 12:28:20 2009
From: valin at buchlovice.org (=?UTF-8?B?UmFkZWsgVmFsw6HFoWVr?=)
Date: Thu Oct 15 12:28:26 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
	unavailable"
Message-ID: <4AD710D6.70404@buchlovice.org>

Hi,

I want to ask if there is something new in adding support to 
gptzfsboot/zfsboot for reading gang-blocks?

 From Sun's docs:

Gang blocks

When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.

Everything works fine for me, until I rewrite kernel/world after system 
upgrade to latest one (releng_8). After this am I no longer able to boot 
from zfs raidz1 pool with following messages:

 >/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ boot:
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
/>/ boot:
//
/I presume it's the same issue as talked in june-2009 current mailing 
list 
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html

Any success in that matter?

Thnx for answer.

vaLin
From ivoras at freebsd.org  Thu Oct 15 13:19:54 2009
From: ivoras at freebsd.org (Ivan Voras)
Date: Thu Oct 15 13:20:00 2009
Subject: ZFS threads?
Message-ID: <hb77gc$es0$1@ger.gmane.org>

A VM running 8.0-RC1 with ZFS has load average of 18 while doing rsync 
to it.

I notice there are two sets of 8 ZFS kernel threads present on the 
system, which look like might be the cause of the high load. Is there a 
way to reduce the number of these worker threads?


     0 root            -16    0     0K   280K -       11:47  3.96% 
{spa_zio_6}
     0 root            -16    0     0K   280K -       12:34  0.00% 
{spa_zio_5}
     0 root            -16    0     0K   280K -       12:11  0.00% 
{spa_zio_3}
     0 root            -16    0     0K   280K -       11:43  0.00% 
{spa_zio_1}
     0 root            -16    0     0K   280K -       11:24  0.00% 
{spa_zio_0}
     0 root            -16    0     0K   280K -       11:20  0.00% 
{spa_zio_7}
     0 root            -16    0     0K   280K -       11:19  0.00% 
{spa_zio_2}
     0 root            -16    0     0K   280K -       10:54  0.00% 
{spa_zio_4}
     4 root             -8    -     0K     8K -        3:10  0.00% [g_down]
     0 root            -16    0     0K   280K -        2:49  0.00% {spa_zio}
    12 root            -40    -     0K   128K WAIT     1:31  0.00% 
{swi2: cambio}
    12 root            -64    -     0K   128K WAIT     0:42  0.00% 
{irq17: mpt0}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_7}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_3}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_6}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_1}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_5}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_0}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_2}
     0 root            -16    0     0K   280K -        0:38  0.00% 
{spa_zio_4}

From rnoland at FreeBSD.org  Thu Oct 15 13:53:35 2009
From: rnoland at FreeBSD.org (Robert Noland)
Date: Thu Oct 15 13:53:41 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
 unavailable"
In-Reply-To: <4AD710D6.70404@buchlovice.org>
References: <4AD710D6.70404@buchlovice.org>
Message-ID: <1255612465.2356.808.camel@balrog.2hip.net>

On Thu, 2009-10-15 at 14:08 +0200, Radek Val??ek wrote:
> Hi,
> 
> I want to ask if there is something new in adding support to 
> gptzfsboot/zfsboot for reading gang-blocks?

I've been thinking of trying to fix this, but haven't really come up
with a repeatable way to test it.  I might be able to come up with at
least a hack to allow booting in the short term, but if you can try this
patch so that we can verify that the issue is indeed gang blocks.  This
doesn't fix anything yet, but it should report when it finds a gang
block.  I know that it is tricky to test when you can't boot, but if you
can apply this patch and reinstall gptzfsboot, it should tell us for
sure that gang blocks are the issue.  I assume that you have a partition
layout something like mine:

balrog% gpart show
=>        34  1953525101  ada0  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162     8388608     2  freebsd-swap  (4.0G)
     8388770  1945136365     3  freebsd-zfs  (928G)

If so, all you should need to do is get this built and then:

#gpart bootcode -p /boot/gptzfsboot -i 1 ada0

substituting appropriate partition index and device info obviously.

robert.

>  From Sun's docs:
> 
> Gang blocks
> 
> When there is not enough contiguous space to write a complete block, the ZIO
> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> assembled transparently to appear as complete blocks.
> 
> Everything works fine for me, until I rewrite kernel/world after system 
> upgrade to latest one (releng_8). After this am I no longer able to boot 
> from zfs raidz1 pool with following messages:
> 
>  >/ ZFS: i/o error - all block copies unavailable
> />/ ZFS: can't read MOS
> />/ ZFS: unexpected object set type lld
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: z:/boot/kernel/kernel
> />/ boot:
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: tank:/boot/kernel/kernel
> />/ boot:
> //
> /I presume it's the same issue as talked in june-2009 current mailing 
> list 
> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
> 
> Any success in that matter?
> 
> Thnx for answer.
> 
> vaLin
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
-- 
Robert Noland <rnoland@FreeBSD.org>
FreeBSD
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zfs-report-gb.patch
Type: text/x-patch
Size: 512 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091015/e9517e7c/zfs-report-gb.bin
From pjd at FreeBSD.org  Thu Oct 15 17:42:07 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Thu Oct 15 17:42:14 2009
Subject: Help needed! ZFS I/O error recovery?
In-Reply-To: <572088116.20091015011137@pyro.de>
References: <90685589.20091013092418@pyro.de>
	<20091013072712.GA1597@garage.freebsd.pl>
	<12910471099.20091013095322@pyro.de>
	<20091013075511.GC1597@garage.freebsd.pl>
	<1433853337.20091013100348@pyro.de>
	<20091013082116.GE1597@garage.freebsd.pl>
	<673550066.20091013133544@pyro.de>
	<473227349.20091014184731@pyro.de>
	<20091014210528.GC1727@garage.freebsd.pl>
	<572088116.20091015011137@pyro.de>
Message-ID: <20091015174200.GB1880@garage.freebsd.pl>

On Thu, Oct 15, 2009 at 01:11:37AM +0200, Solon Lutz wrote:
> >> I just tried it with more TXGs, even with a jump of -300, but it always gives
> >> an "cannot iterate filesystems: I/O error" error if I try to import the pool.
> 
> >> Also because of mounting the pool, the TXg has gone up to 13445935 from
> >> initially 13462284.
> 
> > We can try to turn off checksum verification entirely, but it will most
> > likely just panic your system.
> 
> > If you want to do this, edit
> > sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h file and change
> > ZIO_CHECKSUM_EQUAL() macro to something like this:
> 
> > #define ZIO_CHECKSUM_EQUAL(zc1, zc2)    (1)
> 
> Yes, it paniced as soon as I tried zpool import. Can you tell where ZFS is taking
> the information from, that there are I/O errors? Is this based on checksums or is
> there some kind of I/O-error-flag?

Checksum mismatch is reported as EIO error. Not sure what else we can
try. I guess you are not able to snapshot problematic datasets?

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091015/b4597d59/attachment.pgp
From rnoland at FreeBSD.org  Thu Oct 15 19:04:00 2009
From: rnoland at FreeBSD.org (Robert Noland)
Date: Thu Oct 15 19:04:07 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
 unavailable"
In-Reply-To: <4AD710D6.70404@buchlovice.org>
References: <4AD710D6.70404@buchlovice.org>
Message-ID: <1255633430.2175.12.camel@balrog.2hip.net>

On Thu, 2009-10-15 at 14:08 +0200, Radek Val??ek wrote:
> Hi,
> 
> I want to ask if there is something new in adding support to 
> gptzfsboot/zfsboot for reading gang-blocks?

Ok, I can't figure out any way to test this... beyond the fact that it
builds and doesn't break my currently working setup.  Can you give this
a try?  It should still report if it finds gang blocks, but hopefully
now will read them as well.

robert.

>  From Sun's docs:
> 
> Gang blocks
> 
> When there is not enough contiguous space to write a complete block, the ZIO
> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> assembled transparently to appear as complete blocks.
> 
> Everything works fine for me, until I rewrite kernel/world after system 
> upgrade to latest one (releng_8). After this am I no longer able to boot 
> from zfs raidz1 pool with following messages:
> 
>  >/ ZFS: i/o error - all block copies unavailable
> />/ ZFS: can't read MOS
> />/ ZFS: unexpected object set type lld
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: z:/boot/kernel/kernel
> />/ boot:
> />/ ZFS: unexpected object set type lld
> />/
> />/ FreeBSD/i386 boot
> />/ Default: tank:/boot/kernel/kernel
> />/ boot:
> //
> /I presume it's the same issue as talked in june-2009 current mailing 
> list 
> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
> 
> Any success in that matter?
> 
> Thnx for answer.
> 
> vaLin
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
-- 
Robert Noland <rnoland@FreeBSD.org>
FreeBSD
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zfs-gang-block.patch
Type: text/x-patch
Size: 3137 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091015/178ac281/zfs-gang-block.bin
From alexander.haderer at loescap.de  Thu Oct 15 19:10:04 2009
From: alexander.haderer at loescap.de (Alexander Haderer)
Date: Thu Oct 15 19:10:14 2009
Subject: kern/136470: [nfs] Cannot mount / in read-only, over NFS
Message-ID: <200910151910.n9FJA4Ot094516@freefall.freebsd.org>

The following reply was made to PR kern/136470; it has been noted by GNATS.

From: Alexander Haderer <alexander.haderer@loescap.de>
To: bug-followup@FreeBSD.org, admin@lissyara.su
Cc:  
Subject: Re: kern/136470: [nfs] Cannot mount / in read-only, over NFS
Date: Thu, 15 Oct 2009 20:53:12 +0200

 hello, I submitted a new PR because this problem comes from a bug in the 
 'mount' command, see PR bin/139651
 
 with best regards,
 
 	Alexander
From tzim at tzim.net  Thu Oct 15 19:17:46 2009
From: tzim at tzim.net (Arnaud Houdelette)
Date: Thu Oct 15 19:17:58 2009
Subject: ZFS send from FreeBSD 7.1 (v6) to 8.0 (v13).
Message-ID: <4AD77556.7050607@tzim.net>

Hi.

I use ZFS send/receive to do daily (or nightly) remote replication from 
a dedicated server (at ISP's ) to my Home server.

Dedicated server use 7.1-RELEASE.
Home server is 7.2-RELEASE.
Each night, a snapshot is taken on the dedicated server and an 
incremental send used to replicate the content to my Home server. No 
problem so far.

With incoming 8.0, I'd like to upgrade my home server (and maybe upgrade 
pool to v13) before even thinking to do anything to the dedicated 
server. Will the send/receive scheme continue to work ?

Regards,

Arnaud
From stef-list at memberwebs.com  Thu Oct 15 19:18:02 2009
From: stef-list at memberwebs.com (Stef Walter)
Date: Thu Oct 15 19:18:09 2009
Subject: Deadlock after canceled zfs recv
Message-ID: <4AD77230.3030803@memberwebs.com>

I'm running the latest RELENG_8, and been doing some pre-production
stress testing.

I can deadlock (reproduceable) after a canceled ssh + zfs recv. Here's
how I reproduce the problem:

Do this in a new tank without any data in it. Reboot the system, and
make this the first zfs operations done. Files available here:

http://memberwebs.com/stef/misc/recv-snapshots-zfs-hang.tbz

Receive new file system, and then incremental snapshot:

# cat step-one | zfs recv tank/received
# cat step-two | zfs recv tank/received

At this point should look like:

# zfs list -t snapshot,filesystem | grep received
tank                   2.35G  16.4G    22K  /tank
tank/received           491M  16.4G   489M  /tank/received
tank/received@justnow  1.32M      -   160M  -
tank/received@later        0      -   489M  -

The third one goes through ssh. Count about three to five seconds (one
one thousand, two one thousand, three one thousand) and press Ctrl-C

# cat step-three | ssh localhost zfs recv tank/received

Execute the above 'zfs list' command, and more often than not, parts of
the zfs system are hung, and remain deadlocked until reboot.

If it doesn't happen the first time, try the step three + ctrl-c again.

When run through ssh, and ctrl-c cancelled it seems like 'zfs recv'
doesn't have time to do the cleanup that it normally does if run directly.

FreeBSD zfs8.ws.local 8.0-RC1 FreeBSD 8.0-RC1 #0: Wed Oct 14 16:04:50
UTC 2009     root@zfs8.ws.local:/usr/obj/usr/src/sys/GENERIC  i386

I'm available for any further information and want to help nail down
this bug.

Cheers,

Stef

From linimon at FreeBSD.org  Thu Oct 15 19:31:45 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Thu Oct 15 19:31:51 2009
Subject: bin/139651: [nfs] mount(8): read-only remount of NFS volume
	does not work
Message-ID: <200910151931.n9FJVjU9020994@freefall.freebsd.org>

Old Synopsis: mount: read-only remount of NFS volume does not work
New Synopsis: [nfs] mount(8): read-only remount of NFS volume does not work

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Thu Oct 15 19:30:48 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139651
From valin at buchlovice.org  Thu Oct 15 19:37:35 2009
From: valin at buchlovice.org (=?ISO-8859-2?Q?Radek_Val=E1=B9ek?=)
Date: Thu Oct 15 19:37:50 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
	unavailable"
In-Reply-To: <1255633430.2175.12.camel@balrog.2hip.net>
References: <4AD710D6.70404@buchlovice.org>
	<1255633430.2175.12.camel@balrog.2hip.net>
Message-ID: <4AD779FC.1070204@buchlovice.org>

Robert Noland napsal(a):
> On Thu, 2009-10-15 at 14:08 +0200, Radek Val??ek wrote:
>   
>> Hi,
>>
>> I want to ask if there is something new in adding support to 
>> gptzfsboot/zfsboot for reading gang-blocks?
>>     
>
> Ok, I can't figure out any way to test this... beyond the fact that it
> builds and doesn't break my currently working setup.  Can you give this
> a try?  It should still report if it finds gang blocks, but hopefully
> now will read them as well.
>
> robert.
>
>   
Big thanks for the patches Robert, I will definitely test them as soon 
as possible (tomorrow) and report the results immediately to list. I can 
repeat this issue probably at any time (up to cca 30 times tested with 
the same result), so don't bother about the broken booting, I'm prepared 
for it...

vaLin
>>  From Sun's docs:
>>
>> Gang blocks
>>
>> When there is not enough contiguous space to write a complete block, the ZIO
>> pipeline will break the I/O up into smaller 'gang blocks' which can later be
>> assembled transparently to appear as complete blocks.
>>
>> Everything works fine for me, until I rewrite kernel/world after system 
>> upgrade to latest one (releng_8). After this am I no longer able to boot 
>> from zfs raidz1 pool with following messages:
>>
>>  >/ ZFS: i/o error - all block copies unavailable
>> />/ ZFS: can't read MOS
>> />/ ZFS: unexpected object set type lld
>> />/ ZFS: unexpected object set type lld
>> />/
>> />/ FreeBSD/i386 boot
>> />/ Default: z:/boot/kernel/kernel
>> />/ boot:
>> />/ ZFS: unexpected object set type lld
>> />/
>> />/ FreeBSD/i386 boot
>> />/ Default: tank:/boot/kernel/kernel
>> />/ boot:
>> //
>> /I presume it's the same issue as talked in june-2009 current mailing 
>> list 
>> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
>>
>> Any success in that matter?
>>
>> Thnx for answer.
>>
>> vaLin
>> _______________________________________________
>> freebsd-current@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
>>     

From grarpamp at gmail.com  Thu Oct 15 22:15:20 2009
From: grarpamp at gmail.com (grarpamp)
Date: Thu Oct 15 22:15:32 2009
Subject: ZFS repeatable reboot 8.0-RC1
In-Reply-To: <43425103-6DAE-43FD-82E7-9B71BAB862B0@gothic.net.au>
References: <d2e731a10910141555o604e0abewf8c5b5bdc948f4a0@mail.gmail.com>
	<alpine.BSF.2.00.0910142107230.72893@qvzrafvba.5c.ybpny>
	<d2e731a10910142347n52f60bey8e9058d2b76dccf7@mail.gmail.com>
	<43425103-6DAE-43FD-82E7-9B71BAB862B0@gothic.net.au>
Message-ID: <d2e731a10910151515q601d9b90o66be76228a7d908a@mail.gmail.com>

Note: I tried setting vfs.zfs.arc_max=32M and zfs mem usage still
grew past its limits and the machine rebooted.


Forwarding a note I received...

"""""
>> Your machine is starving!
> How can this be, there is over 500MiB free ram at all times? I'm

sysctl vm.kmem_size_max

I've got the following in my kernel configuration file
 options KVA_PAGES=512

and in /boot/loader.conf
 vm.kmem_size_max=1536M
 vm.kmem_size=1536M

On two machines with 2G of RAM....both 8.0-RC1/i386

the ZFS tuning guide gives a better idea of how to play with things like that
 http://wiki.freebsd.org/ZFSTuningGuide
"""""


Some questions...

Am I reading correctly that vm.kmem_size is how much ram the kernel
initially allocates for itself on boot? And that vm.kmem_size_min
and vm.kmem_size_max are the range that vm.kmem_size is allowed to
float naturally within at runtime?

Is KVA_PAGES the hard max space the kern is allowed/capable of
addressing/using at runtime? Such that I could set kmem_size_max
to the KVA_PAGES limit and then vm.kmem_size will grow into it as
needed?

With the caveat of course that with the below defaults and hardware,
if I just bumped vm.kmem_size_max to 1GiB [as per KVA_PAGES limit]
I'd have to add maybe another 1GiB ram so that this new vm.kmem_size_max
kernel reservation wouldn't conflict with userland memory needs
when vm.kmem_size grows to it?

And KVA_PAGES is typically say 1/3 of installed ram?

If vm.kmem_size starts out being under vm.kmem_size_max, can user
apps use the unused space (vm.kmem_size_max - vm.kmem_size) until
vm.kmem_size grows to vm.kmem_size_max and the kernel kills them
off? Or can user apps only use (ram = user apps + [KVA_PAGES hard
limit and/or vm.kmem_size_max])?

What is the idea behind setting vm.kmem_size = vm.kmem_size_max?
Should not just vm.kmem_size_max be set and allow vm.kmem_size
[unset] to grow up to vm.kmem_size_max instead of allocating it all
at boot with vm.kmem_size?

Maybe someone can wikify these answers?

I think I need to find more to read and then test one by one to
see what changes.


With untuned defaults and 1GiB ram I have:

#define KVA_PAGES       256  # gives 1GiB kern address space
vm.kmem_size_max: 335544320  # auto calculated by the kernel at boot?
Less than KVA_PAGES?
vm.kmem_size_min: 0
vm.kmem_size: 335544320      # amount in use at runtime?

I'm still figuring out how to find and add all the kernel memory.
Here's zfs:

vfs.zfs.arc_meta_limit: 52428800
vfs.zfs.arc_meta_used:  56241732  # greater than meta_limit?
vfs.zfs.arc_min:  26214400
vfs.zfs.arc_max: 209715200
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max:     16384
kstat.zfs.misc.arcstats.p:  20589785  # was 104857600 on boot
kstat.zfs.misc.arcstats.c: 128292242  # was 209715200 on boot
kstat.zfs.misc.arcstats.c_min:  26214400
kstat.zfs.misc.arcstats.c_max: 209715200


loader(8)
 vm.kmem_size  Sets the size of kernel memory (bytes).  This overrides the
               value determined when the kernel was compiled.  Modifies
               VM_KMEM_SIZE.
 vm.kmem_size_min
 vm.kmem_size_max
               Sets the minimum and maximum (respectively) amount of ker-
               nel memory that will be automatically allocated by the ker-
               nel.  These override the values determined when the kernel
               was compiled.  Modifies VM_KMEM_SIZE_MIN and
               VM_KMEM_SIZE_MAX.


sys/i386/include/pmap.h
/*
 * Size of Kernel address space.  This is the number of page table pages
 * (4MB each) to use for the kernel.  256 pages == 1 Gigabyte.
 * This **MUST** be a multiple of 4 (eg: 252, 256, 260, etc).
 * For PAE, the page table page unit size is 2MB.  This means that 512 pages
 * is 1 Gigabyte.  Double everything.  It must be a multiple of 8 for PAE.
 */
#ifndef KVA_PAGES
#ifdef PAE
#define KVA_PAGES       512
#else
#define KVA_PAGES       256
#endif
#endif


sys/i386/conf/NOTES
# Change the size of the kernel virtual address space.  Due to
# constraints in loader(8) on i386, this must be a multiple of 4.
# 256 = 1 GB of kernel address space.  Increasing this also causes
# a reduction of the address space in user processes.  512 splits
# the 4GB cpu address space in half (2GB user, 2GB kernel).  For PAE
# kernels, the value will need to be double non-PAE.  A value of 1024
# for PAE kernels is necessary to split the address space in half.
# This will likely need to be increased to handle memory sizes >4GB.
# PAE kernels default to a value of 512.
#
options         KVA_PAGES=260
From grarpamp at gmail.com  Thu Oct 15 22:24:04 2009
From: grarpamp at gmail.com (grarpamp)
Date: Thu Oct 15 22:24:10 2009
Subject: ZFS threads?
Message-ID: <d2e731a10910151524r5fea9a9enbc8c97d03f8498c7@mail.gmail.com>

I don't know about reducing them directly,
maybe vfs.zfs.zfetch.max_streams. On mine
it's 8 which might be that 0 through 7 you see.

This system has 9 zfs mountpoints and
104 spa_zio* threads. So you might try
cutting mountpoints if possible.
From ml at infosec.pl  Thu Oct 15 22:30:08 2009
From: ml at infosec.pl (Michal)
Date: Thu Oct 15 22:30:14 2009
Subject: root on ZFS and zpool.cache importance
Message-ID: <4AD7AA19.8050802@infosec.pl>

Hello.

I'm a bit confused about zpool.cache file. I've got a configuration with 
/boot sitting on a usb dongle (UFS) and everything else on internal ZFS 
hard drive. I'm booting my system off the usb drive so zpool.cache file 
is there (it was generated when I created zpool). Basic zfs root + ufs 
boot setup. Everything works like a charm no problems so far and I would 
like to keep it that way hence my question.

I've noticed that zpool.cache on ZFS drive is being created and updated 
from time to time and it is different from zpool.cache file on usb 
drive. Even when I remove zpool.cache file on hard disk then it gets 
recreated automatically and system still boots and works fine since it 
starts with zpool.cache on usb drive which is intact.

Now how important it is to keep them in sync and what I'm risking by not 
doing that? Should I copy zpool.cache from hard drive to usb boot disk 
every day or should I leave it how it is and don't even bother?

Michal
-- 
"Never forget what a man says to you when he is angry." -Henry Ward Beecher
From 000.fbsd at quip.cz  Thu Oct 15 23:32:42 2009
From: 000.fbsd at quip.cz (Miroslav Lachman)
Date: Thu Oct 15 23:32:48 2009
Subject: ZFS threads?
In-Reply-To: <d2e731a10910151524r5fea9a9enbc8c97d03f8498c7@mail.gmail.com>
References: <d2e731a10910151524r5fea9a9enbc8c97d03f8498c7@mail.gmail.com>
Message-ID: <4AD7B115.2080104@quip.cz>

grarpamp wrote:

> I don't know about reducing them directly,
> maybe vfs.zfs.zfetch.max_streams. On mine
> it's 8 which might be that 0 through 7 you see.
> 
> This system has 9 zfs mountpoints and
> 104 spa_zio* threads. So you might try
> cutting mountpoints if possible.

I don't know how it is related, but I have system with 17 ZFS 
mountpoints and just 25 spa_zio threads (named as spa_zio_issue_2, 
spa_zio_intr_2 etc).
This is on 7.2-RELEASE-p2 amd64

Miroslav Lachman
From avg at icyb.net.ua  Fri Oct 16 08:28:21 2009
From: avg at icyb.net.ua (Andriy Gapon)
Date: Fri Oct 16 08:28:33 2009
Subject: zfs boot vs loader
In-Reply-To: <20091014062827.GC1696@garage.freebsd.pl>
References: <4AD3220B.5030502@icyb.net.ua>
	<86ws30ikfw.fsf@gmail.com>	<4AD45A8D.4070605@icyb.net.ua>
	<20091014062827.GC1696@garage.freebsd.pl>
Message-ID: <4AD82E91.3030703@icyb.net.ua>

on 14/10/2009 09:28 Pawel Jakub Dawidek said the following:
> On Tue, Oct 13, 2009 at 01:46:37PM +0300, Andriy Gapon wrote:
>> Thanks to all who replied!
>> I think that I figured it out with your help.
>>
>> 'unload' does indeed unload zpool.cache and the latter is needed for proper zfs
>> root mounting.
>> 'boot /other/kernel' executes some loader scripts and zpool.cache gets loaded in
>> this case (because of the settings in defaults/loader.conf).
>> When doing manual 'load xxx' and then just 'boot' no loader scripts are executed
>> and zpool.cache does not get loaded. In this case one has to load it manually
>> using the suggested 'load -t xxx' approach.
> 
> Could you repeat what you were doing but with vfs.zfs.debug set to 1
> from the loader? You should see some messages about missing file, maybe
> we should do them visible always and not after increasing debug level?

Yes, I tried this and got the following message:
spa_config_load:92[1]: Cannot open /boot/zfs/zpool.cache
Perhaps it's a good idea indeed to be more verbose about this.


-- 
Andriy Gapon
From ivoras at freebsd.org  Fri Oct 16 09:19:25 2009
From: ivoras at freebsd.org (Ivan Voras)
Date: Fri Oct 16 09:19:31 2009
Subject: ZFS threads?
In-Reply-To: <4AD7B115.2080104@quip.cz>
References: <d2e731a10910151524r5fea9a9enbc8c97d03f8498c7@mail.gmail.com>
	<4AD7B115.2080104@quip.cz>
Message-ID: <hb9dpu$lmt$1@ger.gmane.org>

Miroslav Lachman wrote:
> grarpamp wrote:
> 
>> I don't know about reducing them directly,
>> maybe vfs.zfs.zfetch.max_streams. On mine
>> it's 8 which might be that 0 through 7 you see.
>>
>> This system has 9 zfs mountpoints and
>> 104 spa_zio* threads. So you might try
>> cutting mountpoints if possible.
> 
> I don't know how it is related, but I have system with 17 ZFS 
> mountpoints and just 25 spa_zio threads (named as spa_zio_issue_2, 
> spa_zio_intr_2 etc).
> This is on 7.2-RELEASE-p2 amd64

Hmm, this is interesting. On a quad-core 7.2-R box amd64 with 11 mount 
points I have 6 sets of 4 zio threads (spa_zio_intr_[0-5] and 
spa_zio_issue_[0-5] in each).

The original machine I wrote about is 8.0-RC1 i386 with a single CPU.

Maybe CPU count detection is turned off in 8?

From simon at comsys.ntu-kpi.kiev.ua  Fri Oct 16 17:00:09 2009
From: simon at comsys.ntu-kpi.kiev.ua (Andrey Simonenko)
Date: Fri Oct 16 17:00:16 2009
Subject: kern/136865: NFS exports atomic and on-the-fly atomic updates
Message-ID: <200910161700.n9GH09dR034486@freefall.freebsd.org>

The following reply was made to PR kern/136865; it has been noted by GNATS.

From: Andrey Simonenko <simon@comsys.ntu-kpi.kiev.ua>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/136865: NFS exports atomic and on-the-fly atomic updates
Date: Fri, 16 Oct 2009 19:51:22 +0300

 New version nfse-20091016 does not have limitation for number of groups:
 nfse(8) gets number of groups via sysconf(_SC_NGROUPS_MAX) and
 nfsserver/nfs_exports.c allows up to NGROUPS groups.
 
 URL: http://comsys.ntu-kpi.kiev.ua/~simon/nfse/
 MD5 (nfse-20091016.tar.bz2) = 49328852374275763876a03f666e016a
From torbjoern at gmail.com  Fri Oct 16 22:44:07 2009
From: torbjoern at gmail.com (Torbjorn Kristoffersen)
Date: Fri Oct 16 22:44:13 2009
Subject: Advice needed with ZFS on root filesystem (installing remotely via 
	mfsBSD)
Message-ID: <103e3fa20910161518t57e2c3afkddf160cb1f4ecfdb@mail.gmail.com>

I am trying to install FreeBSD 8.0-RC1 on amd64, and I plan to use ZFS
(mirrored) as the root filesystem. I have been struggling
with this for many hours -- it just won't boot up and I can't
understand what I'm doing wrong.

Before I describe my steps, keep in mind that I am doing this using
the Remote Install method
of mfsBSD.  My mfsBSD system boots up just fine and I log in via SSH.
 However, seeing that it's a remote
installation I'm unable to see any error message if the system does
not boot up correctly. A sad situation, indeed.

What is wrong with my approach ? Please see below.

--

First I create a GPL label on my two disks
    % gpart create -s gpt ad4
    % gpart create -s gpt ad6

Then I add a boot-partition that will hold the ZFS bootcode
    % gpart add -b 34 -s 128 -t freebsd-boot ad4
    % gpart add -b 34 -s 128 -t freebsd-boot ad6

I then add the freebsd-zfs partition on each disk
    % gpart add -b 162 -s 1465148973 -t freebsd-zfs ad4
    % gpart add -b 162 -s 1465148973 -t freebsd-zfs ad6


The results:
    % gpart show ad4
    =>      34  1465149101  ad4  GPT  (699G)
              34         128    1  freebsd-boot  (64K)
             162  1465148973    2  freebsd-zfs  (699G)

    % gpart show ad6
    =>        34  1465149101  ad6  GPT  (699G)
              34         128    1  freebsd-boot  (64K)
             162  1465148973    2  freebsd-zfs  (699G)

I add GPT bootcode to the MBR and add the ZFS bootcode:
    % gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot -i 1 ad4
    % gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot -i 1 ad6

I make sure that the first partition is set to active
    % fdisk -a /dev/ad4
    ...
    % fdisk -a /dev/ad6

I create zpool on ad4p2 and ad6p2 (mirrored)
    % zpool create zroot mirror ad4p2 ad6p2

Create filesystem
    % zfs create -p zroot

Make sure that I enable the bootfs
    % zpool set bootfs=zroot zroot

At this point, /zroot is mounted on my filesystem, so I go ahead and
do a sysinstall
and install FreeBSD.  I make sure that /zroot is set as "root folder"
in sysinstall's Options menu.

I add zfs_load="YES" to /zroot/boot/loader.conf
and   vfs.root.mountfrom="zfs:zroot" to /zroot/boot/loader.conf

And I also copy /boot/zfs/zpool.cache to /zroot/boot/zfs/zpool.cache

Then I do
    % chroot /zroot
    % sysinstall
...  I go into configuration, set the proper network settings, make
sure that sshd will get started. I change
root's password and create a user.  My /etc/rc.conf (on zroot) looks like this:

    ifconfig_re0="inet xxx.xxx.121.11  netmask 255.255.255.192"
    defaultrouter="xxx.xxx.121.1"
    sshd_enable="YES"
    hostname="grim"

At this point I have also tried compiling a kernel, making sure that
GEOM_PART_BSD, GEOM_PART_MBR and other
required options are included.  I also installed a ZFS aware /boot/loader (**)

(Note: I will create a swap partition later once I get the installation booted.)

At this point
    % exit     (from the chroot)
    % zfs unmount -a
    % reboot

At this point the computer never comes back and I have to use my
colo's rescue system to put mfsBSD back on.

One thing I noticed at that point, is that the freebsd-boot on ad4 is
gone and the freebsd-zfs partition has been replaced with a smaller
freebsd-ufs. I did not touch "Partitions" or "Labels" in the
sysinstall installation. What could possibly cause this?    (In case
it matters; ad6 is unchanged.)


Kind Regards,
Torbjorn Kristoffersen

(**) http://wiki.freebsd.org/ZFSOnRootWithZFSboot#installLoader
From kickbsd at ya.ru  Sat Oct 17 04:16:09 2009
From: kickbsd at ya.ru (kickbsd kickbsd)
Date: Sat Oct 17 04:16:16 2009
Subject: zfs and vfs.numvnodes issue
Message-ID: <66051255751808@webmail71.yandex.ru>

Hi!

I have a strange behavior on zfs filesystem.
vfs.numvnodes tends to grow and when reach kern.maxvnodes no new files can be created or modified.
System AMD64 8.0-RC1 FreeBSD 8.0-RC1 CVS from Oct 13 2009.
I have increased kern.maxvnodes but vfs.numvnodes grows slowly.
Any suggestions ? 
From linimon at FreeBSD.org  Sun Oct 18 02:45:43 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Sun Oct 18 02:45:49 2009
Subject: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
Message-ID: <200910180245.n9I2jgGT020357@freefall.freebsd.org>

Old Synopsis: vfs.numvnodes leak on bussy zfs
New Synopsis: [zfs] vfs.numvnodes leak on busy zfs

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Sun Oct 18 02:45:27 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139715
From ndenev at gmail.com  Sun Oct 18 08:39:41 2009
From: ndenev at gmail.com (Nikolay Denev)
Date: Sun Oct 18 08:39:47 2009
Subject: ZFS send from FreeBSD 7.1 (v6) to 8.0 (v13).
In-Reply-To: <4AD77556.7050607@tzim.net>
References: <4AD77556.7050607@tzim.net>
Message-ID: <A8B612AE-895B-4368-9C4C-745CDAD28EFF@gmail.com>

On 15 Oct, 2009, at 22:17 , Arnaud Houdelette wrote:

> Hi.
>
> I use ZFS send/receive to do daily (or nightly) remote replication  
> from a dedicated server (at ISP's ) to my Home server.
>
> Dedicated server use 7.1-RELEASE.
> Home server is 7.2-RELEASE.
> Each night, a snapshot is taken on the dedicated server and an  
> incremental send used to replicate the content to my Home server. No  
> problem so far.
>
> With incoming 8.0, I'd like to upgrade my home server (and maybe  
> upgrade pool to v13) before even thinking to do anything to the  
> dedicated server. Will the send/receive scheme continue to work ?
>
> Regards,
>
> Arnaud

Check in the zfs manual page. It says this :

...
            The format of the stream is evolving. No backwards  
compatibility is
            guaranteed.  You  may not be able to receive your streams  
on future
            versions of ZFS.
...


Regards,
Niki
From kickbsd at ya.ru  Sun Oct 18 15:45:48 2009
From: kickbsd at ya.ru (kickbsd kickbsd)
Date: Sun Oct 18 15:54:46 2009
Subject: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
In-Reply-To: <200910180245.n9I2jgGT020357@freefall.freebsd.org>
References: <200910180245.n9I2jgGT020357@freefall.freebsd.org>
Message-ID: <10921255880746@webmail6.yandex.ru>


There is the link to leaking vnodes grpaph 
http://xs.to/xs.php?h=xs1144&d=09420&f=vnodes-leak215.jpg
 

18.10.09, 02:45, linimon@FreeBSD.org:

> Old Synopsis: vfs.numvnodes leak on bussy zfs
> New Synopsis: [zfs] vfs.numvnodes leak on busy zfs
> Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
> Responsible-Changed-By: linimon
> Responsible-Changed-When: Sun Oct 18 02:45:27 UTC 2009
> Responsible-Changed-Why: 
> Over to maintainer(s).
> http://www.freebsd.org/cgi/query-pr.cgi?pr=139715
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"

-- 
??????? ????? ????????? ?????: http://mail.yandex.ru/promo/new/attachments
From linimon at FreeBSD.org  Sun Oct 18 16:36:30 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Sun Oct 18 16:36:42 2009
Subject: kern/139718: [reboot] all mounted fs don't get synced during
	reboot/shutdown with >= 1 mounted inaccessible device
Message-ID: <200910181636.n9IGaUaF073450@freefall.freebsd.org>

Old Synopsis: all mounted fs don't get synced during reboot/shutdown with >= 1 mounted inaccessible device
New Synopsis: [reboot] all mounted fs don't get synced during reboot/shutdown with >= 1 mounted inaccessible device

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Sun Oct 18 16:34:44 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139718
From linimon at lonesome.com  Sun Oct 18 16:40:04 2009
From: linimon at lonesome.com (Mark Linimon)
Date: Sun Oct 18 16:40:10 2009
Subject: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
Message-ID: <200910181640.n9IGe4iH073544@freefall.freebsd.org>

The following reply was made to PR kern/139715; it has been noted by GNATS.

From: Mark Linimon <linimon@lonesome.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
Date: Sun, 18 Oct 2009 11:33:33 -0500

 ----- Forwarded message from kickbsd kickbsd <kickbsd@ya.ru> -----
 
 From: kickbsd kickbsd <kickbsd@ya.ru>
 To: linimon@freebsd.org
 Cc: freebsd-bugs@freebsd.org, freebsd-fs@freebsd.org
 Subject: Re: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
 
 There is the link to leaking vnodes grpaph 
 http://xs.to/xs.php?h=xs1144&d=09420&f=vnodes-leak215.jpg
  
 ----- End forwarded message -----
From trasz at FreeBSD.org  Sun Oct 18 18:42:21 2009
From: trasz at FreeBSD.org (trasz@FreeBSD.org)
Date: Sun Oct 18 18:42:28 2009
Subject: kern/139718: [reboot] all mounted fs don't get synced during
	reboot/shutdown with >= 1 mounted inaccessible device
Message-ID: <200910181842.n9IIgLl4084240@freefall.freebsd.org>

Synopsis: [reboot] all mounted fs don't get synced during reboot/shutdown with >= 1 mounted inaccessible device

Responsible-Changed-From-To: freebsd-fs->trasz
Responsible-Changed-By: trasz
Responsible-Changed-When: Sun Oct 18 18:42:21 UTC 2009
Responsible-Changed-Why: 
I'll take it.

http://www.freebsd.org/cgi/query-pr.cgi?pr=139718
From gavin at FreeBSD.org  Sun Oct 18 21:09:34 2009
From: gavin at FreeBSD.org (gavin@FreeBSD.org)
Date: Sun Oct 18 21:09:42 2009
Subject: i386/139725: zdb(1) dumps core on i386 when examining zpool
	contents: Assertion failed: (rwlp->rw_count == 0)
Message-ID: <200910182109.n9IL9XV6004255@freefall.freebsd.org>

Old Synopsis: zdb dumps core on FreeBSD/i386 when examining zpool contents
New Synopsis: zdb(1) dumps core on i386 when examining zpool contents: Assertion failed: (rwlp->rw_count == 0)

Responsible-Changed-From-To: freebsd-i386->freebsd-fs
Responsible-Changed-By: gavin
Responsible-Changed-When: Sun Oct 18 21:06:32 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s)

http://www.freebsd.org/cgi/query-pr.cgi?pr=139725
From bugmaster at FreeBSD.org  Mon Oct 19 11:06:52 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Oct 19 11:08:04 2009
Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org
Message-ID: <200910191106.n9JB6pK2063426@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o kern/139725  fs         [zfs] zdb(1) dumps core on i386 when examining zpool c
o kern/139715  fs         [zfs] vfs.numvnodes leak on busy zfs
o bin/139651   fs         [nfs] mount(8): read-only remount of NFS volume does n
o kern/139597  fs         [patch] [tmpfs] tmpfs initializes va_gen but doesn't u
o kern/139564  fs         [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo
o kern/139407  fs         [smbfs] [panic] smb mount causes system crash if remot
o kern/139363  fs         [nfs] diskless root nfs mount from non FreeBSD server 
o kern/138790  fs         [zfs] ZFS ceases caching when mem demand is high
o kern/138524  fs         [msdosfs] disks and usb flashes/cards with Russian lab
o kern/138421  fs         [ufs] [patch] remove UFS label limitations
o kern/138367  fs         [tmpfs] [panic] 'panic: Assertion pages > 0 failed' wh
o kern/138202  fs         mount_msdosfs(1) see only 2Gb
o kern/138109  fs         [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2f
f kern/137037  fs         [zfs] [hang] zfs rollback on root causes FreeBSD to fr
o kern/136968  fs         [ufs] [lor] ufs/bufwait/ufs (open)
o kern/136945  fs         [ufs] [lor] filedesc structure/ufs (poll)
o kern/136944  fs         [ffs] [lor] bufwait/snaplk (fsync)
o kern/136873  fs         [ntfs] Missing directories/files on NTFS volume
o kern/136865  fs         [nfs] [patch] NFS exports atomic and on-the-fly atomic
o kern/136470  fs         [nfs] Cannot mount / in read-only, over NFS
o kern/135594  fs         [zfs] Single dataset unresponsive with Samba
o kern/135546  fs         [zfs] zfs.ko module doesn't ignore zpool.cache filenam
o kern/135469  fs         [ufs] [panic] kernel crash on md operation in ufs_dirb
o kern/135050  fs         [zfs] ZFS clears/hides disk errors on reboot
o kern/134491  fs         [zfs] Hot spares are rather cold...
o kern/133980  fs         [panic] [ffs] panic: ffs_valloc: dup alloc
o kern/133676  fs         [smbfs] [panic] umount -f'ing a vnode-based memory dis
o kern/133614  fs         [smbfs] [panic] panic: ffs_truncate: read-only filesys
o kern/133174  fs         [msdosfs] [patch] msdosfs must support utf-encoded int
f kern/133150  fs         [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w
o kern/132960  fs         [ufs] [panic] panic:ffs_blkfree: freeing free frag
o kern/132597  fs         [tmpfs] [panic] tmpfs-related panic while interrupting
o kern/132397  fs         reboot causes filesystem corruption (failure to sync b
o kern/132331  fs         [ufs] [lor] LOR ufs and syncer
o kern/132237  fs         [msdosfs] msdosfs has problems to read MSDOS Floppy
o kern/132145  fs         [panic] File System Hard Crashes
o kern/131995  fs         [nfs] Failure to mount NFSv4 server
o kern/131441  fs         [unionfs] [nullfs] unionfs and/or nullfs not combineab
o kern/131360  fs         [nfs] poor scaling behavior of the NFS server under lo
o kern/131342  fs         [nfs] mounting/unmounting of disks causes NFS to fail
o bin/131341   fs         makefs: error "Bad file descriptor"  on the mount poin
o kern/131086  fs         [ext2fs] [patch] mkfs.ext2 creates rotten partition
o kern/130979  fs         [smbfs] [panic] boot/kernel/smbfs.ko
o kern/130920  fs         [msdosfs] cp(1) takes 100% CPU time while copying file
o kern/130229  fs         [iconv] usermount fails on fs that need iconv
o kern/130210  fs         [nullfs] Error by check nullfs
o kern/129760  fs         [nfs] after 'umount -f' of a stale NFS share FreeBSD l
o kern/129488  fs         [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: 
o kern/129231  fs         [ufs] [patch] New UFS mount (norandom) option - mostly
o kern/129152  fs         [panic] non-userfriendly panic when trying to mount(8)
o kern/129059  fs         [zfs] [patch] ZFS bootloader whitelistable via WITHOUT
f kern/128829  fs         smbd(8) causes periodic panic on 7-RELEASE
f kern/128173  fs         [ext2fs] ls gives "Input/output error" on mounted ext3
o kern/127659  fs         [tmpfs] tmpfs memory leak
o kern/127420  fs         [gjournal] [panic] Journal overflow on gmirrored gjour
o kern/127029  fs         [panic] mount(8): trying to mount a write protected zi
o kern/126287  fs         [ufs] [panic] Kernel panics while mounting an UFS file
s kern/125738  fs         [zfs] [request] SHA256 acceleration in ZFS
f kern/125536  fs         [ext2fs] ext 2 mounts cleanly but fails on commands li
f kern/124621  fs         [ext3] [patch] Cannot mount ext2fs partition
f bin/124424   fs         [zfs] zfs(8): zfs list -r shows strange snapshots' siz
o kern/123939  fs         [msdosfs] corrupts new files
o kern/122380  fs         [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash
o bin/122172   fs         [fs]: amd(8) automount daemon dies on 6.3-STABLE i386,
o kern/122047  fs         [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / 
o bin/121898   fs         [nullfs] pwd(1)/getcwd(2) fails with Permission denied
o bin/121779   fs         [ufs] snapinfo(8) (and related tools?) only work for t
o bin/121366   fs         [zfs] [patch] Automatic disk scrubbing from periodic(8
o bin/121072   fs         [smbfs] mount_smbfs(8) cannot normally convert the cha
f kern/120991  fs         [panic] [fs] [snapshot] System crashes when manipulati
o kern/120483  fs         [ntfs] [patch] NTFS filesystem locking changes
o kern/120482  fs         [ntfs] [patch] Sync style changes between NetBSD and F
f kern/119735  fs         [zfs] geli + ZFS + samba starting on boot panics 7.0-B
o kern/118912  fs         [2tb] disk sizing/geometry problem with large array
o kern/118713  fs         [minidump] [patch] Display media size required for a k
o bin/118249   fs         mv(1): moving a directory changes its mtime
o kern/118107  fs         [ntfs] [panic] Kernel panic when accessing a file at N
o bin/117315   fs         [smbfs] mount_smbfs(8) and related options can't mount
o kern/117314  fs         [ntfs] Long-filename only NTFS fs'es cause kernel pani
o kern/117158  fs         [zfs] zpool scrub causes panic if geli vdevs detach on
o bin/116980   fs         [msdosfs] [patch] mount_msdosfs(8) resets some flags f
o kern/116913  fs         [ffs] [panic] ffs_blkfree: freeing free block
p kern/116608  fs         [msdosfs] [patch] msdosfs fails to check mount options
o kern/116583  fs         [ffs] [hang] System freezes for short time when using 
o kern/116170  fs         [panic] Kernel panic when mounting /tmp
o kern/115645  fs         [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex
o bin/115361   fs         [zfs] mount(8) gets into a state where it won't set/un
o kern/114955  fs         [cd9660] [patch] [request] support for mask,dirmask,ui
o kern/114847  fs         [ntfs] [patch] [request] dirmask support for NTFS ala 
o kern/114676  fs         [ufs] snapshot creation panics: snapacct_ufs2: bad blo
o bin/114468   fs         [patch] [request] add -d option to umount(8) to detach
o kern/113852  fs         [smbfs] smbfs does not properly implement DFS referral
o bin/113838   fs         [patch] [request] mount(8): add support for relative p
o bin/113049   fs         [patch] [request] make quot(8) use getopt(3) and show 
o kern/112658  fs         [smbfs] [patch] smbfs and caching problems (resolves b
f usb/112640   fs         [ext2fs] [hang] Kernel freezes when writing a file to 
o kern/111843  fs         [msdosfs] Long Names of files are incorrectly created 
o kern/111782  fs         [ufs] dump(8) fails horribly for large filesystems
s bin/111146   fs         [2tb] fsck(8) fails on 6T filesystem
o kern/109024  fs         [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not 
o kern/109010  fs         [msdosfs] can't mv directory within fat32 file system
o bin/107829   fs         [2TB] fdisk(8): invalid boundary checking in fdisk / w
o kern/106030  fs         [ufs] [panic] panic in ufs from geom when a dead disk 
o kern/105093  fs         [ext2fs] [patch] ext2fs on read-only media cannot be m
o kern/104406  fs         [ufs] Processes get stuck in "ufs" state under persist
o kern/104133  fs         [ext2fs] EXT2FS module corrupts EXT2/3 filesystems
o kern/103035  fs         [ntfs] Directories in NTFS mounted disc images appear 
o kern/101324  fs         [smbfs] smbfs sometimes not case sensitive when it's s
o kern/99290   fs         [ntfs] mount_ntfs ignorant of cluster sizes
o kern/97377   fs         [ntfs] [patch] syntax cleanup for ntfs_ihash.c
o kern/95222   fs         [iso9660] File sections on ISO9660 level 3 CDs ignored
o kern/94849   fs         [ufs] rename on UFS filesystem is not atomic
o kern/94769   fs         [ufs] Multiple file deletions on multi-snapshotted fil
o kern/94733   fs         [smbfs] smbfs may cause double unlock
o kern/93942   fs         [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D
o kern/92272   fs         [ffs] [hang] Filling a filesystem while creating a sna
f kern/91568   fs         [ufs] [panic] writing to UFS/softupdates DVD media in 
o kern/91134   fs         [smbfs] [patch] Preserve access and modification time 
a kern/90815   fs         [smbfs] [patch] SMBFS with character conversions somet
o kern/89991   fs         [ufs] softupdates with mount -ur causes fs UNREFS
o kern/88657   fs         [smbfs] windows client hang when browsing a samba shar
o kern/88266   fs         [smbfs] smbfs does not implement UIO_NOCOPY and sendfi
o kern/87859   fs         [smbfs] System reboot while umount smbfs.
o kern/86587   fs         [msdosfs] rm -r /PATH fails with lots of small files
o kern/85326   fs         [smbfs] [panic] saving a file via samba to an overquot
o kern/84589   fs         [2TB] 5.4-STABLE unresponsive during background fsck 2
o kern/80088   fs         [smbfs] Incorrect file time setting on NTFS mounted vi
o kern/77826   fs         [ext2fs] ext2fs usb filesystem will not mount RW
o kern/73484   fs         [ntfs] Kernel panic when doing `ls` from the client si
o bin/73019    fs         [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino
o kern/71774   fs         [ntfs] NTFS cannot "see" files on a WinXP filesystem
o kern/68978   fs         [panic] [ufs] crashes with failing hard disk, loose po
o kern/65920   fs         [nwfs] Mounted Netware filesystem behaves strange
o kern/65901   fs         [smbfs] [patch] smbfs fails fsx write/truncate-down/tr
o kern/61503   fs         [smbfs] mount_smbfs does not work as non-root
o kern/55617   fs         [smbfs] Accessing an nsmb-mounted drive via a smb expo
o kern/51685   fs         [hang] Unbounded inode allocation causes kernel to loc
o kern/51583   fs         [nullfs] [patch] allow to work with devices and socket
o kern/36566   fs         [smbfs] System reboot with dead smb mount and umount
o kern/18874   fs         [2TB] 32bit NFS servers export wrong negative values t

140 problems total.

From linimon at lonesome.com  Tue Oct 20 15:20:04 2009
From: linimon at lonesome.com (Mark Linimon)
Date: Tue Oct 20 15:20:10 2009
Subject: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
Message-ID: <200910201520.n9KFK3ic091920@freefall.freebsd.org>

The following reply was made to PR kern/139715; it has been noted by GNATS.

From: Mark Linimon <linimon@lonesome.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
Date: Tue, 20 Oct 2009 10:13:05 -0500

 ----- Forwarded message from kickbsd kickbsd <kickbsd@ya.ru> -----
 
 From: kickbsd kickbsd <kickbsd@ya.ru>
 To: linimon@freebsd.org
 Subject: Re: kern/139715: [zfs] vfs.numvnodes leak on busy zfs
 
 There is newer graph to show tendency
 http://www.freeimagehosting.net/image.php?e5f0195542.jpg
 
 
 ----- End forwarded message -----
From rnoland at FreeBSD.org  Wed Oct 21 17:38:41 2009
From: rnoland at FreeBSD.org (Robert Noland)
Date: Wed Oct 21 17:38:53 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
 unavailable"
In-Reply-To: <4AD779FC.1070204@buchlovice.org>
References: <4AD710D6.70404@buchlovice.org>
	<1255633430.2175.12.camel@balrog.2hip.net>
	<4AD779FC.1070204@buchlovice.org>
Message-ID: <1256146709.2310.9.camel@balrog.2hip.net>

On Thu, 2009-10-15 at 21:37 +0200, Radek Val??ek wrote:
> Robert Noland napsal(a):
> > On Thu, 2009-10-15 at 14:08 +0200, Radek Val??ek wrote:
> >   
> >> Hi,
> >>
> >> I want to ask if there is something new in adding support to 
> >> gptzfsboot/zfsboot for reading gang-blocks?
> >>    

I think that the gang block patch will work, though still haven't gotten
it tested.  However, I'm fairly confident that the issue is not gang
block related.  Right now, I have setup a disk like this:

=>        34  1953525101  ada1  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162     8388608     2  freebsd-swap  (4.0G)
     8388770   648019968     3  freebsd-zfs  (309G)
   656408738   648019968     4  freebsd-zfs  (309G)
  1304428706   648019968     5  freebsd-zfs  (309G)
  1952448674     1076461        - free -  (526M)

Note that this is not a raidz pool right now.  It is just 3 toplevel
partitions setup as a single pool.  I finally have this configuration
working reliably.  At least in this case, the issue is due to all of the
partitions not being probed during early boot and so not being added to
the list of vdevs for the pool.  When zio_read finds a dva that points
to a device it doesn't know about, it gives up and whines.

Can you detail for me how you have everything configured, so that I can
try to replicate it.  gpart show, zpool status and zpool get all <pool>
would be good.  I'm not sure that I have enough spare disks lying around
to do this properly, but maybe I can use virtual disks or something.

robert.

> > Ok, I can't figure out any way to test this... beyond the fact that it
> > builds and doesn't break my currently working setup.  Can you give this
> > a try?  It should still report if it finds gang blocks, but hopefully
> > now will read them as well.
> >
> > robert.
> >
> >   
> Big thanks for the patches Robert, I will definitely test them as soon 
> as possible (tomorrow) and report the results immediately to list. I can 
> repeat this issue probably at any time (up to cca 30 times tested with 
> the same result), so don't bother about the broken booting, I'm prepared 
> for it...
> 
> vaLin
> >>  From Sun's docs:
> >>
> >> Gang blocks
> >>
> >> When there is not enough contiguous space to write a complete block, the ZIO
> >> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> >> assembled transparently to appear as complete blocks.
> >>
> >> Everything works fine for me, until I rewrite kernel/world after system 
> >> upgrade to latest one (releng_8). After this am I no longer able to boot 
> >> from zfs raidz1 pool with following messages:
> >>
> >>  >/ ZFS: i/o error - all block copies unavailable
> >> />/ ZFS: can't read MOS
> >> />/ ZFS: unexpected object set type lld
> >> />/ ZFS: unexpected object set type lld
> >> />/
> >> />/ FreeBSD/i386 boot
> >> />/ Default: z:/boot/kernel/kernel
> >> />/ boot:
> >> />/ ZFS: unexpected object set type lld
> >> />/
> >> />/ FreeBSD/i386 boot
> >> />/ Default: tank:/boot/kernel/kernel
> >> />/ boot:
> >> //
> >> /I presume it's the same issue as talked in june-2009 current mailing 
> >> list 
> >> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
> >>
> >> Any success in that matter?
> >>
> >> Thnx for answer.
> >>
> >> vaLin
> >> _______________________________________________
> >> freebsd-current@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
> >>     
> 
-- 
Robert Noland <rnoland@FreeBSD.org>
FreeBSD

From linimon at FreeBSD.org  Wed Oct 21 20:15:31 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Wed Oct 21 20:15:38 2009
Subject: kern/139806: [zfs] [panic] Write attempt to file in ZFS
	snapshot dir causes panic
Message-ID: <200910212015.n9LKFVZx024690@freefall.freebsd.org>

Old Synopsis: Write attempt to file in ZFS snapshot dir causes panic
New Synopsis: [zfs] [panic] Write attempt to file in ZFS snapshot dir causes panic

Responsible-Changed-From-To: freebsd-amd64->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Wed Oct 21 20:15:16 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=139806
From fbsdlist at src.cx  Wed Oct 21 23:22:10 2009
From: fbsdlist at src.cx (Artem Belevich)
Date: Wed Oct 21 23:22:15 2009
Subject: ARC size constantly shrinks, then ZFS slows down extremely
In-Reply-To: <A9EED983-8382-4C8B-BFB9-5A4CA1D62A37@freebsd.org>
References: <4AC1E540.9070001@fsn.hu> <4AC5B2C7.2000200@fsn.hu>
	<20091002184526.GA1660@garage.freebsd.pl> <4ACDA5EA.2010600@fsn.hu>
	<4ACDDED0.2070707@fsn.hu> <20091008160718.GB2134@garage.freebsd.pl>
	<4AD2E118.2050202@fsn.hu>
	<47C0A3F4-6431-49E5-B780-FA162946C288@freebsd.org>
	<4AD2E998.8090409@fsn.hu>
	<A9EED983-8382-4C8B-BFB9-5A4CA1D62A37@freebsd.org>
Message-ID: <ed91d4a80910211622lc42e3dasbaad953ca6877106@mail.gmail.com>

Hi,

Will r197816 from head be MFC'd to stable-8 as commit comment
suggested? I've been running my box with the r197816 applied to
stable/8 and so far it's been working fine for me.

#svn log -v -c 197816
------------------------------------------------------------------------
r197816 | kmacy | 2009-10-06 14:40:50 -0700 (Tue, 06 Oct 2009) | 5 lines
Changed paths:
   M /head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

Prevent paging pressure from draining arc too much
- always drain arc if above arc_c_max - never drain arc if arc is
below arc_c_max

MFC after:	3 days

------------------------------------------------------------------------

--Artem
From jh at FreeBSD.org  Thu Oct 22 20:10:04 2009
From: jh at FreeBSD.org (Jaakko Heinonen)
Date: Thu Oct 22 20:10:14 2009
Subject: kern/139806: [zfs] [panic] Write attempt to file in ZFS
	snapshot dir causes panic
Message-ID: <200910222010.n9MKA3YG090955@freefall.freebsd.org>

The following reply was made to PR kern/139806; it has been noted by GNATS.

From: Jaakko Heinonen <jh@FreeBSD.org>
To: Carl Chave <online@chave.us>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/139806: [zfs] [panic] Write attempt to file in ZFS
	snapshot dir causes panic
Date: Thu, 22 Oct 2009 22:50:56 +0300

 Hi,
 
 On 2009-10-21, Carl Chave wrote:
 > Fixit# echo hello >> test.txt
 > panic: dirtying snapshot!
 
 The problem seems to be that in certain conditions zfs_freebsd_access()
 uses only vaccess(9) for access check. However vaccess(9) doesn't handle
 the read-only file system case.
 
 Could you try this patch?
 
 --- patch begins here ---
 Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
 ===================================================================
 --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c	(revision 198368)
 +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c	(working copy)
 @@ -3989,7 +3989,12 @@
  		struct thread *a_td;
  	} */ *ap;
  {
 +	int error;
  
 +	error = zfs_access(ap->a_vp, ap->a_accmode, 0, ap->a_cred, NULL);
 +	if (error != 0)
 +		return (error);
 +
  	/*
  	 * ZFS itself only knowns about VREAD, VWRITE and VEXEC, the rest
  	 * we have to handle by calling vaccess().
 @@ -3999,11 +4004,11 @@
  		znode_t *zp = VTOZ(vp);
  		znode_phys_t *zphys = zp->z_phys;
  
 -		return (vaccess(vp->v_type, zphys->zp_mode, zphys->zp_uid,
 -		    zphys->zp_gid, ap->a_accmode, ap->a_cred, NULL));
 +		error = vaccess(vp->v_type, zphys->zp_mode, zphys->zp_uid,
 +		    zphys->zp_gid, ap->a_accmode, ap->a_cred, NULL);
  	}
  
 -	return (zfs_access(ap->a_vp, ap->a_accmode, 0, ap->a_cred, NULL));
 +	return (error);
  }
  
  static int
 --- patch ends here ---
 
 -- 
 Jaakko
From randy at psg.com  Sat Oct 24 00:39:57 2009
From: randy at psg.com (Randy Bush)
Date: Sat Oct 24 00:40:09 2009
Subject: to err or not to err
Message-ID: <m2zl7hwrom.wl%randy@psg.com>

so, were there errors or not?  do i worry/act or not?

randy

---

dfw1# zpool status -v
  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: scrub in progress for 0h5m, 8.59% done, 1h3m to go
config:

        NAME               STATE     READ WRITE CKSUM
        tank               ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m00-d01  ONLINE       0     0     0
            label/m00-d00  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m01-d00  ONLINE       0     0     0
            label/m01-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m02-d00  ONLINE       0     0     0
            label/m02-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m03-d00  ONLINE       0     0     0
            label/m03-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m04-d00  ONLINE       0     0     0
            label/m04-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m05-d00  ONLINE       0     0     0
            label/m05-d01  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        tank/dfw1:<0x8dffc>
        tank/dfw1:<0xcfd>
        tank/dfw1:<0x8dffd>

< some hours later >

dfw1# zpool status -v
  pool: tank
 state: ONLINE
 scrub: scrub completed after 0h53m with 0 errors on Fri Oct 23 15:20:43 2009
config:

        NAME               STATE     READ WRITE CKSUM
        tank               ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m00-d01  ONLINE       0     0     0
            label/m00-d00  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m01-d00  ONLINE       0     0     0
            label/m01-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m02-d00  ONLINE       0     0     0
            label/m02-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m03-d00  ONLINE       0     0     0
            label/m03-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m04-d00  ONLINE       0     0     0
            label/m04-d01  ONLINE       0     0     0
          mirror           ONLINE       0     0     0
            label/m05-d00  ONLINE       0     0     0
            label/m05-d01  ONLINE       0     0     0

errors: No known data errors

-30-
From valin at buchlovice.org  Sat Oct 24 17:44:28 2009
From: valin at buchlovice.org (=?ISO-8859-2?Q?Radek_Val=E1=B9ek?=)
Date: Sat Oct 24 17:44:35 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
	unavailable"
In-Reply-To: <1256146709.2310.9.camel@balrog.2hip.net>
References: <4AD710D6.70404@buchlovice.org>	
	<1255633430.2175.12.camel@balrog.2hip.net>	
	<4AD779FC.1070204@buchlovice.org>
	<1256146709.2310.9.camel@balrog.2hip.net>
Message-ID: <4AE33CF9.3050308@buchlovice.org>

Robert Noland napsal(a):
> On Thu, 2009-10-15 at 21:37 +0200, Radek Val??ek wrote:
>   
>> Robert Noland napsal(a):
>>     
>>> On Thu, 2009-10-15 at 14:08 +0200, Radek Val??ek wrote:
>>>   
>>>       
>>>> Hi,
>>>>
>>>> I want to ask if there is something new in adding support to 
>>>> gptzfsboot/zfsboot for reading gang-blocks?
>>>>    
>>>>         
>
> I think that the gang block patch will work, though still haven't gotten
> it tested.  However, I'm fairly confident that the issue is not gang
> block related.  Right now, I have setup a disk like this:
>
> =>        34  1953525101  ada1  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162     8388608     2  freebsd-swap  (4.0G)
>      8388770   648019968     3  freebsd-zfs  (309G)
>    656408738   648019968     4  freebsd-zfs  (309G)
>   1304428706   648019968     5  freebsd-zfs  (309G)
>   1952448674     1076461        - free -  (526M)
>
> Note that this is not a raidz pool right now.  It is just 3 toplevel
> partitions setup as a single pool.  I finally have this configuration
> working reliably.  At least in this case, the issue is due to all of the
> partitions not being probed during early boot and so not being added to
> the list of vdevs for the pool.  When zio_read finds a dva that points
> to a device it doesn't know about, it gives up and whines.
>
> Can you detail for me how you have everything configured, so that I can
> try to replicate it.  gpart show, zpool status and zpool get all <pool>
> would be good.  I'm not sure that I have enough spare disks lying around
> to do this properly, but maybe I can use virtual disks or something.
>
> robert.
>
>   

Sorry for not responding so long. Here are details you want from me:

# gpart show
=>        34  1953525101  ad6  GPT  (932G)
          34         128    1  freebsd-boot  (64K)
         162  1953524973    2  freebsd-zfs  (932G)

=>        34  1953525101  ad8  GPT  (932G)
          34         128    1  freebsd-boot  (64K)
         162  1953524973    2  freebsd-zfs  (932G)

=>        34  1953525101  ad10  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162  1953524973     2  freebsd-zfs  (932G)

=>        34  1953525101  ad12  GPT  (932G)
          34         128     1  freebsd-boot  (64K)
         162  1953524973     2  freebsd-zfs  (932G)

# zpool status
  pool: z
 state: ONLINE
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    z           ONLINE       0     0     0
      raidz1    ONLINE       0     0     0
        ad6p2   ONLINE       0     0     0
        ad8p2   ONLINE       0     0     0
        ad10p2  ONLINE       0     0     0
        ad12p2  ONLINE       0     0     0

errors: No known data errors

# zpool get all z
NAME  PROPERTY       VALUE       SOURCE
z     size           3.62T       -
z     used           4.62G       -
z     available      3.62T       -
z     capacity       0%          -
z     altroot        -           default
z     health         ONLINE      -
z     guid           17857007133862981114  -
z     version        13          default
z     bootfs         z/system    local
z     delegation     on          default
z     autoreplace    off         default
z     cachefile      -           default
z     failmode       wait        default
z     listsnapshots  off         default

I've tested your patches but it seems that you're right and it's not 
gang related issue. I was able to discover these things on a fully 
functional zfs pool (system compiled with your patches):

1, If I overwrite the file /boot/loader.conf (with copy of itself, or 
when upgrading kernel/world), next reboot comes with these messages:

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

Then I'm still able to boot the system, but I must set the boot 
variables included in loader.conf by hand

2, Next I overwrite the file /boot/loader (with copy of itself, or when 
upgrading kernel/world) and reboot comes with these messages:

BTX loader 1.00  BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
Unable to load a kernel!

After that I'm no longer able to boot the system from zfs pool.

Hope you have some ideas...

vaLin

>>> Ok, I can't figure out any way to test this... beyond the fact that it
>>> builds and doesn't break my currently working setup.  Can you give this
>>> a try?  It should still report if it finds gang blocks, but hopefully
>>> now will read them as well.
>>>
>>> robert.
>>>
>>>   
>>>       
>> Big thanks for the patches Robert, I will definitely test them as soon 
>> as possible (tomorrow) and report the results immediately to list. I can 
>> repeat this issue probably at any time (up to cca 30 times tested with 
>> the same result), so don't bother about the broken booting, I'm prepared 
>> for it...
>>
>> vaLin
>>     
>>>>  From Sun's docs:
>>>>
>>>> Gang blocks
>>>>
>>>> When there is not enough contiguous space to write a complete block, the ZIO
>>>> pipeline will break the I/O up into smaller 'gang blocks' which can later be
>>>> assembled transparently to appear as complete blocks.
>>>>
>>>> Everything works fine for me, until I rewrite kernel/world after system 
>>>> upgrade to latest one (releng_8). After this am I no longer able to boot 
>>>> from zfs raidz1 pool with following messages:
>>>>
>>>>  >/ ZFS: i/o error - all block copies unavailable
>>>> />/ ZFS: can't read MOS
>>>> />/ ZFS: unexpected object set type lld
>>>> />/ ZFS: unexpected object set type lld
>>>> />/
>>>> />/ FreeBSD/i386 boot
>>>> />/ Default: z:/boot/kernel/kernel
>>>> />/ boot:
>>>> />/ ZFS: unexpected object set type lld
>>>> />/
>>>> />/ FreeBSD/i386 boot
>>>> />/ Default: tank:/boot/kernel/kernel
>>>> />/ boot:
>>>> //
>>>> /I presume it's the same issue as talked in june-2009 current mailing 
>>>> list 
>>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
>>>>
>>>> Any success in that matter?
>>>>
>>>> Thnx for answer.
>>>>
>>>> vaLin
>>>> _______________________________________________
>>>> freebsd-current@freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
>>>>     
>>>>         

From stef-list at memberwebs.com  Sat Oct 24 17:46:59 2009
From: stef-list at memberwebs.com (Stef Walter)
Date: Sat Oct 24 17:47:07 2009
Subject: Deadlock after canceled zfs recv
In-Reply-To: <4AD77230.3030803@memberwebs.com>
References: <4AD77230.3030803@memberwebs.com>
Message-ID: <4AE33D89.6050807@memberwebs.com>

Stef Walter wrote:
> I'm running the latest RELENG_8, and been doing some pre-production
> stress testing.
> 
> I can deadlock (reproduceable) after a canceled ssh + zfs recv. Here's
> how I reproduce the problem:

Pawel, I hope it's okay if I CC you on this problem. I'd like to
contribute and help get this problem sorted out before the 8.0 release.

I've done a deadlock post mortem on this problem. It's below. There are
several zfs processes waiting on locks. Is there anything I can do to
further research the problem.

Thanks in advance.

Cheers,

Stef




----------------------------------------------------------------------
 DEADLOCK POSTMORTEM

db> show pcpu
cpuid        = 0
dynamic pcpu    = 0x68e280
curthread    = 0xc596fb40: pid 11 "idle: cpu0"
curpcb       = 0xc54cdd90
fpcurthread  = none
idlethread   = 0xc596fb40: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:
db> show allpcpu
Current CPU: 0

cpuid        = 0
dynamic pcpu    = 0x68e280
curthread    = 0xc596fb40: pid 11 "idle: cpu0"
curpcb       = 0xc54cdd90
fpcurthread  = none
idlethread   = 0xc596fb40: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

db> show locks
db> show alllocks
db> show lockedvnods
Locked vnodes
db> alltrace

Tracing command zfs pid 1559 tid 100126 td 0xc673ab40
sched_switch(c673ab40,0,104,191,ecf50dd2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c673ab40,0,c0c7a69a,24b,c673ab40,...) at sleepq_switch+0x15f
sleepq_wait(c97710a8,0,e8188ab4,1,0,...) at sleepq_wait+0x63
_cv_wait(c97710a8,c9771078,208,206,c97710a8,...) at _cv_wait+0x240
dsl_dataset_hold_ref(c6b0f9c0,e8188b58,c6268de1,241,e8188b28,...) at
dsl_dataset_hold_ref+0xa0
dsl_dataset_hold(d1be8000,c6b0f9c0,e8188b58,c086d87d,1,...) at
dsl_dataset_hold+0x10c
dmu_objset_open(d1be8000,5,9,e8188b84,e8188b94,...) at dmu_objset_open+0xc3
zfs_ioc_objset_stats(d1be8000,24,d1be800e,0,d1be8938,...) at
zfs_ioc_objset_stats+0x38
zfs_ioc_dataset_list_next(d1be8000,0,0,0,0,...) at
zfs_ioc_dataset_list_next+0x11f
zfsdev_ioctl(c6136100,cc285a13,d1be8000,3,c673ab40,...) at zfsdev_ioctl+0x7d
devfs_ioctl_f(c5ed3a10,cc285a13,d1be8000,c60ff680,c673ab40,...) at
devfs_ioctl_f+0xf8
kern_ioctl(c673ab40,3,cc285a13,d1be8000,173ab40,...) at kern_ioctl+0x1fd
ioctl(c673ab40,e8188cf8,c,e8188d38,c0d5c928,...) at ioctl+0x134
syscall(e8188d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp =
0xbfbfc9cc, ebp = 0xbfbfd638 ---
Tracing command zfs pid 1558 tid 100124 td 0xc6122480
sched_switch(c6122480,0,104,191,fc4c78d2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122480,0,c0c7a69a,26e,2,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcac64,0,c62689d8,2,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcac64,0,0,c62689d8,1,...) at _sleep+0x339
pause(c62689d8,1,c9771200,e818250c,c61a1ec2,...) at pause+0x47
dnode_special_close(c9446000,c626870f,c61a0c60,c9771200,c9771200,...) at
dnode_special_close+0x2c
dmu_objset_evict(c9771000,c9771200,1cb,0,c6265be3,...) at
dmu_objset_evict+0x82
dsl_dataset_destroy(c9771000,c6268520,16,e8182984,e8182944,...) at
dsl_dataset_destroy+0x24c
dmu_recv_abort_cleanup(e8182984,c626d723,1cb,c6265810,e818292c,...) at
dmu_recv_abort_cleanup+0x2f
dmu_recv_stream(e8182984,c5ed4dc8,e818299c,0,0,...) at dmu_recv_stream+0x9de
zfs_ioc_recv(cfa8f000,0,0,0,0,...) at zfs_ioc_recv+0x2d4
zfsdev_ioctl(c6136100,cc285a1c,cfa8f000,3,c6122480,...) at zfsdev_ioctl+0x7d
devfs_ioctl_f(c5ed4700,cc285a1c,cfa8f000,c8488d00,c6122480,...) at
devfs_ioctl_f+0xf8
kern_ioctl(c6122480,3,cc285a1c,cfa8f000,1000000,...) at kern_ioctl+0x1fd
ioctl(c6122480,e8182cf8,c,c0c8e3e7,c0d5c928,...) at ioctl+0x134
syscall(e8182d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp =
0xbfbfbcfc, ebp = 0xbfbfbd28 ---
Tracing command zfskern pid 1521 tid 100117 td 0xc61226c0
sched_switch(c61226c0,0,104,191,a1331866,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c61226c0,0,c0c7a69a,26e,c61226c0,...) at sleepq_switch+0x15f
sleepq_timedwait(c65a65dc,0,e8164c28,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c65a65dc,c65a6584,7530,c65a65dc,c65a6580,...) at
_cv_timedwait+0x250
txg_thread_wait(7530,0,3b9aca00,0,3e8,...) at txg_thread_wait+0x44
txg_sync_thread(c65a6400,e8164d38,c0c717a4,343,c611d550,...) at
txg_sync_thread+0x122
fork_exit(c61d37e0,c65a6400,e8164d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8164d70, ebp = 0 ---

Tracing command zfskern pid 1521 tid 100116 td 0xc6122900
sched_switch(c6122900,0,104,191,a133372e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122900,0,c0c7a69a,24b,c6122900,...) at sleepq_switch+0x15f
sleepq_wait(c65a65ec,0,e8160c50,1,0,...) at sleepq_wait+0x63
_cv_wait(c65a65ec,c65a6584,c65a65f4,c65a65ec,c65a6580,...) at _cv_wait+0x240
txg_thread_wait(0,0,c626d723,18d,10,...) at txg_thread_wait+0x87
txg_quiesce_thread(c65a6400,e8160d38,c0c717a4,343,c611d550,...) at
txg_quiesce_thread+0xb5
fork_exit(c61d3070,c65a6400,e8160d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8160d70, ebp = 0 ---

Tracing command zfskern pid 1521 tid 100114 td 0xc6122d80
sched_switch(c6122d80,0,104,191,a12ac4fe,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c6122d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541908,4c,c62713d8,0,0,...) at sleepq_wait+0x63
_sleep(c6541908,c654191c,24c,c62713d8,0,...) at _sleep+0x36b
vdev_geom_worker(c6541900,e8154d38,c0c717a4,343,c611d550,...) at
vdev_geom_worker+0x124
fork_exit(c621a170,c6541900,e8154d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8154d70, ebp = 0 ---

Tracing command zfskern pid 1521 tid 100087 td 0xc6120b40
sched_switch(c6120b40,0,104,191,7f209aca,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120b40,0,c0c7a69a,26e,c6120b40,...) at sleepq_switch+0x15f
sleepq_timedwait(c627e534,0,e8109c24,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c627e534,c627e520,3e8,1131,0,...) at _cv_timedwait+0x250
l2arc_feed_thread(0,e8109d38,c0c717a4,343,c611d550,...) at
l2arc_feed_thread+0x131
fork_exit(c618f090,0,e8109d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8109d70, ebp = 0 ---

Tracing command zfskern pid 1521 tid 100082 td 0xc61216c0
sched_switch(c61216c0,0,104,191,7f20c4da,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c61216c0,0,c0c7a69a,26e,c61216c0,...) at sleepq_switch+0x15f
sleepq_timedwait(c627614c,0,c57f6c80,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c627614c,c6276138,3e8,0,3e8,...) at _cv_timedwait+0x250
arc_reclaim_thread(0,c57f6d38,c0c717a4,343,c611d550,...) at
arc_reclaim_thread+0x39f
fork_exit(c6191f00,0,c57f6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc57f6d70, ebp = 0 ---

Tracing command bash pid 1519 tid 100080 td 0xc6121b40
sched_switch(c6121b40,0,104,191,ebff50ce,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c6121b40,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c611daa0,5c,c0c7cf0b,100,0,...) at sleepq_wait_sig+0x17
_sleep(c611daa0,c611db28,15c,c0c7cf0b,0,...) at _sleep+0x354
kern_wait(c6121b40,ffffffff,c57f0c74,6,0,...) at kern_wait+0xb76
wait4(c6121b40,c57f0cf8,10,c0c7cdfa,c0d5c404,...) at wait4+0x3b
syscall(c57f0d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x80e1f4f, esp =
0xbfbfe9dc, ebp = 0xbfbfe9f8 ---

Tracing command sshd pid 1516 tid 100075 td 0xc5ede480
sched_switch(c5ede480,0,104,191,ebe40d02,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede480,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c087111a,c5e7cd50,0,c0c74a49,c5ede480,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5e7cd64,0,c57cda7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5e7cd64,c5e7cd50,c0c7c95a,5d1,c5ed30e0,...) at
_cv_wait_sig+0x240
seltdwait(c5ed30e0,58,c5f40200,c5ede480,58,...) at seltdwait+0xa2
kern_select(c5ede480,9,286030c4,286030cc,0,0,20,65,280fdd50) at
kern_select+0x4f4
select(c5ede480,c57cdcf8,14,c0c5e8da,c0d5cd6c,...) at select+0x66
syscall(c57cdd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp =
0xbfbfde5c, ebp = 0xbfbfdea8 ---

Tracing command login pid 1514 tid 100070 td 0xc5ee0000
sched_switch(c5ee0000,0,104,191,2065f36,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0000,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c087111a,c59ede04,0,c0c74a49,c5ee0000,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ede70,0,c57bcb0c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ede70,c59ede04,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ede00,c59ede70,1,c57bcb87,c57bcb94,...) at tty_wait+0x71
ttydisc_read(c59ede00,c57bcc58,0,9e,0,...) at ttydisc_read+0xef
ttydev_read(c5afd400,c57bcc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed39a0,c57bcc58,c596b100,0,c5ee0000,...) at devfs_read_f+0x7e
dofileread(c57bcc58,ffffffff,ffffffff,0,c5ed39a0,...) at dofileread+0x96
kern_readv(c5ee0000,0,c57bcc58,c57bcc78,1,...) at kern_readv+0x58
read(c5ee0000,c57bccf8,c,c0c7cefa,c0d5c394,...) at read+0x4f
syscall(c57bcd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x281af253, esp =
0xbfbfe6bc, ebp = 0xbfbfe958 ---

Tracing command getty pid 1513 tid 100083 td 0xc6121480
sched_switch(c6121480,0,104,191,51bbee4e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121480,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c6121480,c6121480,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5c7d470,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5c7d470,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c5c7d400,c5c7d470,c57f9c58,1,0,...) at tty_wait+0x71
ttydisc_read(c5c7d400,c57f9c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85900,c57f9c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed33f0,c57f9c58,c596b100,0,c6121480,...) at devfs_read_f+0x7e
dofileread(c57f9c58,ffffffff,ffffffff,0,c5ed33f0,...) at dofileread+0x96
kern_readv(c6121480,0,c57f9c58,c57f9c78,1,...) at kern_readv+0x58
read(c6121480,c57f9cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(c57f9d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1512 tid 100077 td 0xc5ede000
sched_switch(c5ede000,0,104,191,519be396,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede000,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c5ede000,c5ede000,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59eda70,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59eda70,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59eda00,c59eda70,c57e7c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59eda00,c57e7c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85700,c57e7c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed38c0,c57e7c58,c596b100,0,c5ede000,...) at devfs_read_f+0x7e
dofileread(c57e7c58,ffffffff,ffffffff,0,c5ed38c0,...) at dofileread+0x96
kern_readv(c5ede000,0,c57e7c58,c57e7c78,1,...) at kern_readv+0x58
read(c5ede000,c57e7cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(c57e7d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1511 tid 100069 td 0xc5ee0240
sched_switch(c5ee0240,0,104,191,517bb2e2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0240,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c5ee0240,c5ee0240,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed670,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed670,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed600,c59ed670,c57b8c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed600,c57b8c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85e00,c57b8c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3268,c57b8c58,c596b100,0,c5ee0240,...) at devfs_read_f+0x7e
dofileread(c57b8c58,ffffffff,ffffffff,0,c5ed3268,...) at dofileread+0x96
kern_readv(c5ee0240,0,c57b8c58,c57b8c78,1,...) at kern_readv+0x58
read(c5ee0240,c57b8cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(c57b8d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1510 tid 100081 td 0xc6121900
sched_switch(c6121900,0,104,191,515ba0da,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121900,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c6121900,c6121900,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed470,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed470,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed400,c59ed470,c57f3c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed400,c57f3c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c86100,c57f3c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3ce8,c57f3c58,c596b100,0,c6121900,...) at devfs_read_f+0x7e
dofileread(c57f3c58,ffffffff,ffffffff,0,c5ed3ce8,...) at dofileread+0x96
kern_readv(c6121900,0,c57f3c58,c57f3c78,1,...) at kern_readv+0x58
read(c6121900,c57f3cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(c57f3d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1509 tid 100063 td 0xc5e56240
sched_switch(c5e56240,0,104,191,513b103e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e56240,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c5e56240,c5e56240,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59eca70,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59eca70,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59eca00,c59eca70,c5799c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59eca00,c5799c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85b00,c5799c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3818,c5799c58,c596b100,0,c5e56240,...) at devfs_read_f+0x7e
dofileread(c5799c58,ffffffff,ffffffff,0,c5ed3818,...) at dofileread+0x96
kern_readv(c5e56240,0,c5799c58,c5799c78,1,...) at kern_readv+0x58
read(c5e56240,c5799cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(c5799d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1508 tid 100086 td 0xc6120d80
sched_switch(c6120d80,0,104,191,511a69a2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120d80,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c6120d80,c6120d80,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed270,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed270,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed200,c59ed270,e80fcc58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed200,e80fcc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85600,e80fcc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3310,e80fcc58,c596b100,0,c6120d80,...) at devfs_read_f+0x7e
dofileread(e80fcc58,ffffffff,ffffffff,0,c5ed3310,...) at dofileread+0x96
kern_readv(c6120d80,0,e80fcc58,e80fcc78,1,...) at kern_readv+0x58
read(c6120d80,e80fccf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(e80fcd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1507 tid 100085 td 0xc6121000
sched_switch(c6121000,0,104,191,51fdf8aa,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121000,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c6121000,c6121000,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59edc70,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59edc70,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59edc00,c59edc70,c57ffc58,1,0,...) at tty_wait+0x71
ttydisc_read(c59edc00,c57ffc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c86000,c57ffc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3e70,c57ffc58,c596b100,0,c6121000,...) at devfs_read_f+0x7e
dofileread(c57ffc58,ffffffff,ffffffff,0,c5ed3e70,...) at dofileread+0x96
kern_readv(c6121000,0,c57ffc58,c57ffc78,1,...) at kern_readv+0x58
read(c6121000,c57ffcf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(c57ffd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1506 tid 100084 td 0xc6121240
sched_switch(c6121240,0,104,191,50fa4d42,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121240,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,c6121240,c6121240,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed870,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed870,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed800,c59ed870,c57fcc58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed800,c57fcc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85a00,c57fcc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3a80,c57fcc58,c596b100,0,c6121240,...) at devfs_read_f+0x7e
dofileread(c57fcc58,ffffffff,ffffffff,0,c5ed3a80,...) at dofileread+0x96
kern_readv(c6121240,0,c57fcc58,c57fcc78,1,...) at kern_readv+0x58
read(c6121240,c57fccf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f
syscall(c57fcd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command cron pid 1455 tid 100065 td 0xc5ee0b40
sched_switch(c5ee0b40,0,104,191,153fd976,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c5ee0b40,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(ea61,c08b8560,c5ee0b40,0,100,...) at
sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c0dcae04,5c,c0c7759a,100,0,...) at
sleepq_timedwait_sig+0x1a
_sleep(c0dcae04,0,15c,c0c7759a,ea61,...) at _sleep+0x31e
kern_nanosleep(c5ee0b40,c57a1c64,c57a1c6c,3c,0,...) at kern_nanosleep+0xc1
nanosleep(c5ee0b40,c57a1cf8,8,c0c7d048,c0d5dd80,...) at nanosleep+0x6f
syscall(c57a1d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28179c2f, esp =
0xbfbfec8c, ebp = 0xbfbfecb8 ---

Tracing command sendmail pid 1449 tid 100064 td 0xc5ee0d80
sched_switch(c5ee0d80,0,104,191,22a85a16,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,68,...) at mi_switch+0x200
sleepq_switch(c5ee0d80,0,c0c7a69a,18b,68,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c59deaf8,68,c0c2a55d,100,0,...) at sleepq_wait_sig+0x17
_sleep(c59deaf8,c59deb28,168,c0c2a55d,0,...) at _sleep+0x354
kern_sigsuspend(c5ee0d80,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c5ee0d80,c579dcf8,4,c0c7cdfa,c0d5e88c,...) at sigsuspend+0x4d
syscall(c579dd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28332a3b, esp =
0xbfbfcf9c, ebp = 0xbfbfcfc8 ---

Tracing command sendmail pid 1443 tid 100067 td 0xc5ee06c0
sched_switch(c5ee06c0,0,104,191,ecb7afa2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee06c0,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c57b0a4c,c087111a,c5e5cb50,0,c5ee06c0,...) at
sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c5e5cb64,0,c57b0a7c,101,0,...) at
sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c5e5cb64,c5e5cb50,1389,5d1,c5ed3bd0,...) at
_cv_timedwait_sig+0x250
seltdwait(c57b0c28,c57b0c30,c60ff500,c5ee06c0,c188b014,...) at
seltdwait+0x8a
kern_select(c5ee06c0,5,bfbfc520,0,0,c57b0c70,20,5,0) at kern_select+0x4f4
select(c5ee06c0,c57b0cf8,14,c0c7d31e,c0d5cd6c,...) at select+0x66
syscall(c57b0d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283d81d3, esp =
0xbfbfc48c, ebp = 0xbfbfcfb8 ---

Tracing command sshd pid 1436 tid 100071 td 0xc5eded80
sched_switch(c5eded80,0,104,191,4d9d04fe,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5eded80,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c087111a,c5f2d850,0,c0c74a49,c5eded80,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2d864,0,c57c0a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2d864,c5f2d850,c0c7c95a,5d1,c5ed3850,...) at
_cv_wait_sig+0x240
seltdwait(c5ed3850,58,c6100b00,c5eded80,0,...) at seltdwait+0xa2
kern_select(c5eded80,5,286090ac,0,0,0,20,65,a) at kern_select+0x4f4
select(c5eded80,c57c0cf8,14,c0c7d770,c0d5cd6c,...) at select+0x66
syscall(c57c0d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp =
0xbfbfdf1c, ebp = 0xbfbfee38 ---

Tracing command syslogd pid 1206 tid 100074 td 0xc5ede6c0
sched_switch(c5ede6c0,0,104,191,b65fba92,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede6c0,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c087111a,c5e7bc10,0,c0c74a49,c5ede6c0,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5e7bc24,0,c57caa7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5e7bc24,c5e7bc10,c0c7c95a,5d1,c5ed32d8,...) at
_cv_wait_sig+0x240
seltdwait(c5ed32d8,58,c596b100,c5ede6c0,0,...) at seltdwait+0xa2
kern_select(c5ede6c0,9,282290a8,0,0,0,20,65,281a4ad8) at kern_select+0x4f4
select(c5ede6c0,c57cacf8,14,c0c7cefa,c0d5cd6c,...) at select+0x66
syscall(c57cad38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281921d3, esp =
0xbfbfde7c, ebp = 0xbfbfee28 ---

Tracing command dhclient pid 1157 tid 100068 td 0xc5ee0480
sched_switch(c5ee0480,0,104,191,125256de,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0480,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c57b4aa8,c087111a,c5e7cdd0,0,c5ee0480,...) at
sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c5e7cde4,0,c57b4ad8,101,0,...) at
sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c5e7cde4,c5e7cdd0,5265431,5d1,c57b4b64,...) at
_cv_timedwait_sig+0x250
seltdwait(c57b4c5c,c57b4c64,4df,c5ee0480,c57b4b5c,...) at seltdwait+0x8a
poll(c5ee0480,c57b4cf8,c,c0c15d47,c0d5da1c,...) at poll+0x300
syscall(c57b4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command dhclient pid 1120 tid 100073 td 0xc5ede900
sched_switch(c5ede900,0,104,191,8b5d5aba,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede900,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c087111a,c5f2e5d0,0,c0c74a49,c5ede900,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2e5e4,0,c57c7ad8,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2e5e4,c5f2e5d0,c0c7c95a,5d1,c57c7b5c,...) at
_cv_wait_sig+0x240
seltdwait(c5f6692c,c0c7c95a,4df,c5ede900,c57c7b5c,...) at seltdwait+0xa2
poll(c5ede900,c57c7cf8,c,c57c7cb8,c0d5da1c,...) at poll+0x300
syscall(c57c7d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp =
0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command devd pid 971 tid 100072 td 0xc5edeb40
sched_switch(c5edeb40,0,104,191,7c9b5d12,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5edeb40,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c087111a,c5f2d650,0,c0c74a49,c5edeb40,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2d664,0,c57c4a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2d664,c5f2d650,c0c7c95a,5d1,c5ed34d0,...) at
_cv_wait_sig+0x240
seltdwait(c5ed34d0,58,c596b100,c5edeb40,c0c6e5c4,...) at seltdwait+0xa2
kern_select(c5edeb40,6,bfbfe9b0,0,0,0,20,65,10) at kern_select+0x4f4
select(c5edeb40,c57c4cf8,14,c0c15d47,c0d5cd6c,...) at select+0x66
syscall(c57c4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x80880e3, esp =
0xbfbfe97c, ebp = 0xbfbfee58 ---

Tracing command moused pid 954 tid 100066 td 0xc5ee0900
sched_switch(c5ee0900,0,104,191,5e2c030e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0900,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c087111a,c5f2ed50,0,c0c74a49,c5ee0900,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2ed64,0,c57aca7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2ed64,c5f2ed50,c0c7c95a,5d1,c5ed3000,...) at
_cv_wait_sig+0x240
seltdwait(c5ed3000,58,c596b100,c5ee0900,0,...) at seltdwait+0xa2
kern_select(c5ee0900,7,bfbfea14,0,0,0,20,65,0) at kern_select+0x4f4
select(c5ee0900,c57accf8,14,c0c8e3e7,c0d5cd6c,...) at select+0x66
syscall(c57acd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281ac1d3, esp =
0xbfbfe99c, ebp = 0xbfbfeb38 ---

Tracing command flowcleaner pid 21 tid 100062 td 0xc5e56480
sched_switch(c5e56480,0,104,191,8f896f2e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e56480,0,c0c7a69a,26e,c5e56480,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f37888,0,c5791cc4,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c0f37888,c0f37890,2710,3ec,0,...) at _cv_timedwait+0x250
flowtable_cleaner(0,c5791d38,c0c717a4,343,c5ea1000,...) at
flowtable_cleaner+0x1af
fork_exit(c0927ec0,0,c5791d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5791d70, ebp = 0 ---

Tracing command softdepflush pid 20 tid 100061 td 0xc5e566c0
sched_switch(c5e566c0,0,104,191,a95525e6,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5e566c0,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f42f80,44,c0c9cf37,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f42f80,c0f42f24,44,c0c9cf37,3e8,...) at _sleep+0x339
softdep_flush(0,c578ed38,c0c717a4,343,c5ea12a8,...) at softdep_flush+0x244
fork_exit(c0ab1150,0,c578ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc578ed70, ebp = 0 ---

Tracing command vnlru pid 19 tid 100060 td 0xc5e56900
sched_switch(c5e56900,0,104,191,a954e79e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,50,...) at mi_switch+0x200
sleepq_switch(c5e56900,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c5ea1550,50,c0c847ac,0,0,...) at sleepq_timedwait+0x6b
_sleep(c5ea1550,c0f37654,250,c0c847ac,3e8,...) at _sleep+0x339
vnlru_proc(0,c578bd38,c0c717a4,343,c5ea1550,...) at vnlru_proc+0xe7
fork_exit(c09138c0,0,c578bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc578bd70, ebp = 0 ---

Tracing command syncer pid 18 tid 100059 td 0xc5e56b40
sched_switch(c5e56b40,0,104,191,a954bed6,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e56b40,0,c0c7a69a,26e,c5e56b40,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f37694,0,c5788c88,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c0f37694,c0f37680,3e8,6cc,4e20,...) at _cv_timedwait+0x250
sched_sync(0,c5788d38,c0c717a4,343,c5ea17f8,...) at sched_sync+0x502
fork_exit(c0912cf0,0,c5788d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5788d70, ebp = 0 ---

Tracing command bufdaemon pid 17 tid 100058 td 0xc5e56d80
sched_switch(c5e56d80,0,104,191,a954977e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5e56d80,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f373c8,44,c0c81d22,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f373c8,c0f373cc,44,c0c81d22,3e8,...) at _sleep+0x339
buf_daemon(0,c5785d38,c0c717a4,343,c5ea1aa0,...) at buf_daemon+0x138
fork_exit(c08fb060,0,c5785d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5785d70, ebp = 0 ---

Tracing command pagezero pid 16 tid 100057 td 0xc5e5b000
sched_switch(c5e5b000,0,104,191,8e2607de,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e5b000,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f43c54,0,c0ca2840,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f43c54,c0f43810,0,c0ca2840,493e0,...) at _sleep+0x339
vm_pagezero(0,c5782d38,c0c717a4,343,c5ea1d48,...) at vm_pagezero+0xdc
fork_exit(c0aee490,0,c5782d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5782d70, ebp = 0 ---

Tracing command vmdaemon pid 15 tid 100056 td 0xc5e5b240
sched_switch(c5e5b240,0,104,191,2a76132,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,68,...) at mi_switch+0x200
sleepq_switch(c5e5b240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0f4387c,68,c0c81d22,0,0,...) at sleepq_wait+0x63
_sleep(c0f4387c,c0f43880,68,c0c81d22,0,...) at _sleep+0x36b
vm_daemon(0,c577fd38,c0c717a4,343,c596e2a8,...) at vm_daemon+0x59
fork_exit(c0ae88e0,0,c577fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc577fd70, ebp = 0 ---

Tracing command pagedaemon pid 9 tid 100055 td 0xc5e5b480
sched_switch(c5e5b480,0,104,191,5b168876,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5e5b480,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f43844,44,c0c81d22,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f43844,c0f43810,44,c0c81d22,1388,...) at _sleep+0x339
vm_pageout(0,c577cd38,c0c717a4,343,c596e550,...) at vm_pageout+0x2bb
fork_exit(c0ae9780,0,c577cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc577cd70, ebp = 0 ---

Tracing command usb pid 14 tid 100054 td 0xc5e5b6c0
sched_switch(c5e5b6c0,0,104,191,5ff31610,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e5b6c0,0,c0c7a69a,24b,c5e5b6c0,...) at sleepq_switch+0x15f
sleepq_wait(c5aead0c,0,c5779cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aead0c,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5aead04,c5779d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5aead04,c5779d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5779d70, ebp = 0 ---

Tracing command usb pid 14 tid 100053 td 0xc5e5b900
sched_switch(c5e5b900,0,104,191,730c5dd2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e5b900,0,c0c7a69a,24b,c5e5b900,...) at sleepq_switch+0x15f
sleepq_wait(c5aeacdc,0,c5776cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aeacdc,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5aeacd4,c5776d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5aeacd4,c5776d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5776d70, ebp = 0 ---

Tracing command usb pid 14 tid 100052 td 0xc5e5bb40
sched_switch(c5e5bb40,0,104,191,a1274526,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e5bb40,0,c0c7a69a,24b,c5e5bb40,...) at sleepq_switch+0x15f
sleepq_wait(c5aeacac,0,c5773cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aeacac,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5aeaca4,c5773d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5aeaca4,c5773d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5773d70, ebp = 0 ---

Tracing command usb pid 14 tid 100051 td 0xc5e5bd80
sched_switch(c5e5bd80,0,104,191,5a5f3bf8,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e5bd80,0,c0c7a69a,24b,c5e5bd80,...) at sleepq_switch+0x15f
sleepq_wait(c5aeac7c,0,c5770cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aeac7c,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5aeac74,c5770d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5aeac74,c5770d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5770d70, ebp = 0 ---

Tracing command usb pid 14 tid 100050 td 0xc5b5f6c0
sched_switch(c5b5f6c0,0,104,191,5a5f14f4,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b5f6c0,0,c0c7a69a,24b,c5b5f6c0,...) at sleepq_switch+0x15f
sleepq_wait(c5ad7dac,0,c576dcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ad7dac,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ad7da4,c576dd38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ad7da4,c576dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc576dd70, ebp = 0 ---

Tracing command usb pid 14 tid 100049 td 0xc5b5f900
sched_switch(c5b5f900,0,104,191,6ef6c8da,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b5f900,0,c0c7a69a,24b,c5b5f900,...) at sleepq_switch+0x15f
sleepq_wait(c5ad7d7c,0,c576acb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ad7d7c,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ad7d74,c576ad38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ad7d74,c576ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc576ad70, ebp = 0 ---

Tracing command usb pid 14 tid 100048 td 0xc5b5fb40
sched_switch(c5b5fb40,0,104,191,544dcdcc,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b5fb40,0,c0c7a69a,24b,c5b5fb40,...) at sleepq_switch+0x15f
sleepq_wait(c5ad7d4c,0,c5767cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ad7d4c,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ad7d44,c5767d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ad7d44,c5767d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5767d70, ebp = 0 ---

Tracing command usb pid 14 tid 100047 td 0xc5b5fd80
sched_switch(c5b5fd80,0,104,191,544da8c4,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b5fd80,0,c0c7a69a,24b,c5b5fd80,...) at sleepq_switch+0x15f
sleepq_wait(c5ad7d1c,0,c5764cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ad7d1c,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ad7d14,c5764d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ad7d14,c5764d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5764d70, ebp = 0 ---

Tracing command usb pid 14 tid 100046 td 0xc5e55000
sched_switch(c5e55000,0,104,191,544d83f0,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e55000,0,c0c7a69a,24b,c5e55000,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5dac,0,c5761cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5dac,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ac5da4,c5761d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ac5da4,c5761d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5761d70, ebp = 0 ---

Tracing command usb pid 14 tid 100045 td 0xc5e55240
sched_switch(c5e55240,0,104,191,6e88eef6,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e55240,0,c0c7a69a,24b,c5e55240,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d7c,0,c575ecb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d7c,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ac5d74,c575ed38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ac5d74,c575ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc575ed70, ebp = 0 ---

Tracing command usb pid 14 tid 100044 td 0xc5e55480
sched_switch(c5e55480,0,104,191,4e39af0c,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e55480,0,c0c7a69a,24b,c5e55480,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d4c,0,c575bcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d4c,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ac5d44,c575bd38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ac5d44,c575bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc575bd70, ebp = 0 ---

Tracing command usb pid 14 tid 100043 td 0xc5e556c0
sched_switch(c5e556c0,0,104,191,4e398d00,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e556c0,0,c0c7a69a,24b,c5e556c0,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d1c,0,c5758cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d1c,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ac5d14,c5758d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ac5d14,c5758d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5758d70, ebp = 0 ---

Tracing command usb pid 14 tid 100042 td 0xc5e55900
sched_switch(c5e55900,0,104,191,6f1cda86,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e55900,0,c0c7a69a,24b,c5e55900,...) at sleepq_switch+0x15f
sleepq_wait(c5ab0dac,0,c5755cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ab0dac,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ab0da4,c5755d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ab0da4,c5755d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5755d70, ebp = 0 ---

Tracing command usb pid 14 tid 100041 td 0xc5e55b40
sched_switch(c5e55b40,0,104,191,6f1d1da2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e55b40,0,c0c7a69a,24b,c5e55b40,...) at sleepq_switch+0x15f
sleepq_wait(c5ab0d7c,0,c5752cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ab0d7c,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ab0d74,c5752d38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ab0d74,c5752d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5752d70, ebp = 0 ---

Tracing command usb pid 14 tid 100040 td 0xc5e55d80
sched_switch(c5e55d80,0,104,191,5e2bc3d2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e55d80,0,c0c7a69a,24b,c5e55d80,...) at sleepq_switch+0x15f
sleepq_wait(c5ab0d4c,0,c574fcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ab0d4c,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ab0d44,c574fd38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ab0d44,c574fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc574fd70, ebp = 0 ---

Tracing command usb pid 14 tid 100039 td 0xc5e56000
sched_switch(c5e56000,0,104,191,3a0de216,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5e56000,0,c0c7a69a,24b,c5e56000,...) at sleepq_switch+0x15f
sleepq_wait(c5ab0d1c,0,c574ccb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ab0d1c,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240
usb_process(c5ab0d14,c574cd38,c0c717a4,343,c596e7f8,...) at
usb_process+0x193
fork_exit(c07b7150,c5ab0d14,c574cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc574cd70, ebp = 0 ---

Tracing command sctp_iterator pid 8 tid 100038 td 0xc5a85d80
sched_switch(c5a85d80,0,104,191,804aa3ce,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5a85d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0f3917c,0,c0c8fb50,0,0,...) at sleepq_wait+0x63
_sleep(c0f3917c,c0f39090,0,c0c8fb50,0,...) at _sleep+0x36b
sctp_iterator_thread(0,c5749d38,c0c717a4,343,c596eaa0,...) at
sctp_iterator_thread+0x60
fork_exit(c099c420,0,c5749d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5749d70, ebp = 0 ---

Tracing command fdc0 pid 7 tid 100034 td 0xc5b5d6c0
sched_switch(c5b5d6c0,0,104,191,b1e97d3e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c5b5d6c0,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c59ec23c,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b
_sleep(c59ec23c,c59ec2f0,4c,c0c6c015,3e8,...) at _sleep+0x339
fdc_thread(c59ec200,c573cd38,c0c717a4,343,c596ed48,...) at fdc_thread+0x2be
fork_exit(c0b6ba00,c59ec200,c573cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc573cd70, ebp = 0 ---

Tracing command fw0_probe pid 6 tid 100031 td 0xc5b5dd80
sched_switch(c5b5dd80,0,104,191,7c72a4a,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c5b5dd80,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c5b62000,5c,c0c6c015,100,0,...) at sleepq_wait_sig+0x17
_sleep(c5b62000,c5b66488,15c,c0c6c015,0,...) at _sleep+0x354
fw_bus_probe_thread(c5b62000,c5558d38,c0c717a4,343,c59de000,...) at
fw_bus_probe_thread+0xa08
fork_exit(c0654500,c5b62000,c5558d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5558d70, ebp = 0 ---

Tracing command xpt_thrd pid 5 tid 100015 td 0xc59716c0
sched_switch(c59716c0,0,104,191,481a07e0,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c59716c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0d97a54,4c,c0c117a4,0,0,...) at sleepq_wait+0x63
_sleep(c0d97a54,c0d97a6c,4c,c0c117a4,0,...) at _sleep+0x36b
xpt_scanner_thread(0,c54f2d38,c0c717a4,343,c59de2a8,...) at
xpt_scanner_thread+0x4a
fork_exit(c0482f30,0,c54f2d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f2d70, ebp = 0 ---

Tracing command yarrow pid 13 tid 100011 td 0xc59b5000
sched_switch(c59b5000,0,104,191,f525b556,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5000,0,c0c7a69a,26e,2,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcac64,0,c0c6c015,2,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcac64,0,0,c0c6c015,64,...) at _sleep+0x339
pause(c0c6c015,64,c0c58f7a,111,0,...) at pause+0x47
random_kthread(0,c54e6d38,c0c717a4,343,c59de550,...) at random_kthread+0x1ef
fork_exit(c0730080,0,c54e6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e6d70, ebp = 0 ---

Tracing command g_down pid 4 tid 100009 td 0xc59b5480
sched_switch(c59b5480,0,104,191,f124675e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c59b5480,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dc8a24,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dc8a24,c0dc8988,24c,c0c6c015,64,...) at _sleep+0x339
g_io_schedule_down(c59b5480,0,c0c6d64b,74,0,...) at g_io_schedule_down+0x6b
g_down_procbody(0,c54e0d38,c0c717a4,343,c596d000,...) at
g_down_procbody+0x8d
fork_exit(c081e2f0,0,c54e0d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e0d70, ebp = 0 ---

Tracing command g_up pid 3 tid 100008 td 0xc596f000
sched_switch(c596f000,0,104,191,f1248b8e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c596f000,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dc8a20,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dc8a20,c0dc89a8,24c,c0c6c015,64,...) at _sleep+0x339
g_io_schedule_up(c596f000,0,c0c6d64b,5d,0,...) at g_io_schedule_up+0x133
g_up_procbody(0,c54ddd38,c0c717a4,343,c596d2a8,...) at g_up_procbody+0x8d
fork_exit(c081e380,0,c54ddd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54ddd70, ebp = 0 ---

Tracing command g_event pid 2 tid 100007 td 0xc596f240
sched_switch(c596f240,0,104,191,f525e4ee,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c596f240,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dc8a18,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dc8a18,0,4c,c0c6c015,64,...) at _sleep+0x339
g_event_procbody(0,c54dad38,c0c717a4,343,c596d550,...) at
g_event_procbody+0xcb
fork_exit(c081e410,0,c54dad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54dad70, ebp = 0 ---

Tracing command intr pid 12 tid 100037 td 0xc5b5d000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100036 td 0xc5b5d240
sched_switch(c5b5d240,0,109,191,f631f0a6,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c5c77170,...) at mi_switch+0x200
ithread_loop(c5c802f0,c5743d38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c5c802f0,c5743d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5743d70, ebp = 0 ---

Tracing command intr pid 12 tid 100035 td 0xc5b5d480
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100033 td 0xc5b5d900
sched_switch(c5b5d900,0,109,191,59d95806,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b42f0,...) at mi_switch+0x200
ithread_loop(c5c6a630,c572fd38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c5c6a630,c572fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc572fd70, ebp = 0 ---

Tracing command intr pid 12 tid 100032 td 0xc5b5db40
sched_switch(c5b5db40,0,109,191,479c27be,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b4370,...) at mi_switch+0x200
ithread_loop(c5c6a4a0,c56c4d38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c5c6a4a0,c56c4d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc56c4d70, ebp = 0 ---

Tracing command intr pid 12 tid 100029 td 0xc5b5f240
sched_switch(c5b5f240,0,109,191,47dd9b8c,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b41f0,...) at mi_switch+0x200
ithread_loop(c5b028e0,c554fd38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c5b028e0,c554fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc554fd70, ebp = 0 ---

Tracing command intr pid 12 tid 100027 td 0xc59b56c0
sched_switch(c59b56c0,0,109,191,e51b0c76,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b3ef0,...) at mi_switch+0x200
ithread_loop(c5b02980,c5546d38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c5b02980,c5546d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5546d70, ebp = 0 ---

Tracing command intr pid 12 tid 100026 td 0xc59b5900
sched_switch(c59b5900,0,109,191,a1270286,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b3df0,...) at mi_switch+0x200
ithread_loop(c5aa95c0,c552ed38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c5aa95c0,c552ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc552ed70, ebp = 0 ---

Tracing command intr pid 12 tid 100025 td 0xc59b5b40
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100024 td 0xc59b5d80
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100023 td 0xc5a85000
sched_switch(c5a85000,0,109,191,6f1c989a,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b4270,...) at mi_switch+0x200
ithread_loop(c5aac820,c5521d38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c5aac820,c5521d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5521d70, ebp = 0 ---

Tracing command intr pid 12 tid 100022 td 0xc5a85240
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100020 td 0xc5a856c0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100016 td 0xc5971480
sched_switch(c5971480,0,109,191,a12a276a,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c5a83970,...) at mi_switch+0x200
ithread_loop(c590a980,c54f5d38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c590a980,c54f5d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f5d70, ebp = 0 ---

Tracing command intr pid 12 tid 100013 td 0xc5971b40
sched_switch(c5971b40,0,109,191,8a3a33da,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b3070,...) at mi_switch+0x200
ithread_loop(c590a9b0,c54ecd38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c590a9b0,c54ecd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54ecd70, ebp = 0 ---

Tracing command intr pid 12 tid 100012 td 0xc5971d80
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100006 td 0xc596f480
sched_switch(c596f480,0,109,191,fc4c5766,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b3c70,...) at mi_switch+0x200
ithread_loop(c596c0c0,c54d7d38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c596c0c0,c54d7d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54d7d70, ebp = 0 ---

Tracing command intr pid 12 tid 100005 td 0xc596f6c0
sched_switch(c596f6c0,0,109,191,4b847aea,...) at sched_switch+0x406
mi_switch(109,0,c0c71a2b,4f6,c59b3cf0,...) at mi_switch+0x200
ithread_loop(c596c0d0,c54d4d38,c0c717a4,343,c596d7f8,...) at
ithread_loop+0x1f6
fork_exit(c0859f70,c596c0d0,c54d4d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54d4d70, ebp = 0 ---

Tracing command intr pid 12 tid 100004 td 0xc596f900
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 100003 td 0xc596fb40
kdb_enter(c0c1f62f,c0c5fa9d,80000000,c5afd380,0,...) at kdb_enter+0x3a
uart_intr(c5afd300,c596fb40,c5953cd0,c596ad80,4,...) at uart_intr+0x126
intr_event_handle(c596ad80,c54cdc34,0,1f4,c5c7d000,...) at
intr_event_handle+0x5c
intr_execute_handlers(c5953cd0,c54cdc34,0,c54cdc74,c0b96044,...) at
intr_execute_handlers+0x49
lapic_handle_intr(3b,c54cdc34) at lapic_handle_intr+0x4c
Xapic_isr1() at Xapic_isr1+0x34
--- interrupt, eip = 0xc0b8a525, esp = 0xc54cdc74, ebp = 0xc54cdc74 ---
acpi_cpu_c1(c54c0008,c54c0028,0,c596fb40,c0dceb40,...) at acpi_cpu_c1+0x5
acpi_cpu_idle(c54cdcb4,c0ba194b,1,c54cdcf8,c08a31be,...) at
acpi_cpu_idle+0x11c
cpu_idle_acpi(1,c54cdcf8,c08a31be,1,0,...) at cpu_idle_acpi+0x1b
cpu_idle(1,0,c0c77e8a,9e9,c596fb40,...) at cpu_idle+0x1b
sched_idletd(0,c54cdd38,c0c717a4,343,c596daa0,...) at sched_idletd+0x23e
fork_exit(c08a2f80,0,c54cdd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54cdd70, ebp = 0 ---

Tracing command init pid 1 tid 100002 td 0xc596fd80
sched_switch(c596fd80,0,104,191,6b6f0d6a,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c596fd80,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at
sleepq_catch_signals+0xb7
sleepq_wait_sig(c596dd48,5c,c0c7cf0b,100,0,...) at sleepq_wait_sig+0x17
_sleep(c596dd48,c596ddd0,15c,c0c7cf0b,0,...) at _sleep+0x354
kern_wait(c596fd80,ffffffff,c54c9c74,0,0,...) at kern_wait+0xb76
wait4(c596fd80,c54c9cf8,10,c0c7cd3b,c0d5c404,...) at wait4+0x3b
syscall(c54c9d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b8f, esp =
0xbfbfe90c, ebp = 0xbfbfe928 ---

Tracing command audit pid 10 tid 100001 td 0xc5971000
sched_switch(c5971000,0,104,191,48159710,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971000,0,c0c7a69a,24b,c5971000,...) at sleepq_switch+0x15f
sleepq_wait(c0f428a0,0,c54c6c9c,1,0,...) at sleepq_wait+0x63
_cv_wait(c0f428a0,c0f42884,c0c9a679,194,0,...) at _cv_wait+0x240
audit_worker(0,c54c6d38,c0c717a4,343,c596e000,...) at audit_worker+0x84
fork_exit(c0a84b10,0,c54c6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54c6d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100130 td 0xc673a000
sched_switch(c673a000,0,104,191,fea55740,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c673a000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(d4a446c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(d4a446c0,d4a446d8,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(ce9352f0,e8194d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,ce9352f0,e8194d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8194d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100115 td 0xc6122b40
sched_switch(c6122b40,0,104,191,9e7765e8,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6542280,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6542280,c6542298,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5b07c70,e8157d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5b07c70,e8157d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8157d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100113 td 0xc6545000
sched_switch(c6545000,0,104,191,a12b1c96,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6545000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541700,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541700,c6541718,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f3b0,e8151d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f3b0,e8151d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8151d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100112 td 0xc6545240
sched_switch(c6545240,0,104,191,96bcc54e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6545240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c65416c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c65416c0,c65416d8,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4e0,e814ed38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4e0,e814ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe814ed70, ebp = 0 ---

Tracing command kernel pid 0 tid 100111 td 0xc6545480
sched_switch(c6545480,0,104,191,96bca382,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6545480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541680,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541680,c6541698,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5e9a1d0,e814bd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5e9a1d0,e814bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe814bd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100110 td 0xc65456c0
sched_switch(c65456c0,0,104,191,96bc803e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c65456c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c65412c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c65412c0,c65412d8,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f470,e8148d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f470,e8148d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8148d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100109 td 0xc6546480
sched_switch(c6546480,0,104,191,96bc5daa,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6546480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541280,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541280,c6541298,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5e9a700,e8145d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5e9a700,e8145d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8145d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100108 td 0xc6120000
sched_switch(c6120000,0,104,191,96bc39da,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541240,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541240,c6541258,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f520,e8142d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f520,e8142d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8142d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100107 td 0xc6120240
sched_switch(c6120240,0,104,191,a0dc8922,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541800,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541800,c6541818,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e813fd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f500,e813fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe813fd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100106 td 0xc6545900
sched_switch(c6545900,0,104,191,9fc3328a,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6545900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e813cd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e813cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe813cd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100105 td 0xc6545b40
sched_switch(c6545b40,0,104,191,9fc30fca,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6545b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8139d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e8139d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8139d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100104 td 0xc6545d80
sched_switch(c6545d80,0,104,191,9fc38dc2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6545d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8136d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e8136d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8136d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100103 td 0xc6546000
sched_switch(c6546000,0,104,191,9fc3aa6e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6546000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8133d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e8133d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8133d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100102 td 0xc6546240
sched_switch(c6546240,0,104,191,9fc37132,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6546240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8130d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e8130d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8130d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100101 td 0xc6120480
sched_switch(c6120480,0,104,191,9fc2dec2,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e812dd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e812dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe812dd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100100 td 0xc61206c0
sched_switch(c61206c0,0,104,191,9fc3cc86,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c61206c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e811bd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e811bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe811bd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100099 td 0xc6120900
sched_switch(c6120900,0,104,191,9fc35486,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8118d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4f0,e8118d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8118d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100098 td 0xc6547d80
sched_switch(c6547d80,0,104,191,4b17e030,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6547d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e8115d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e8115d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8115d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100097 td 0xc6547b40
sched_switch(c6547b40,0,104,191,4af7fcd8,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6547b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e812ad38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e812ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe812ad70, ebp = 0 ---

Tracing command kernel pid 0 tid 100096 td 0xc65466c0
sched_switch(c65466c0,0,104,191,4adc90e8,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c65466c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e8127d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e8127d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8127d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100095 td 0xc6546900
sched_switch(c6546900,0,104,191,47f2818e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6546900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e8124d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e8124d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8124d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100094 td 0xc6546b40
sched_switch(c6546b40,0,104,191,47d25caa,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6546b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e8121d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e8121d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8121d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100093 td 0xc6546d80
sched_switch(c6546d80,0,104,191,4bdfd712,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6546d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e811ed38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e811ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe811ed70, ebp = 0 ---

Tracing command kernel pid 0 tid 100092 td 0xc6547000
sched_switch(c6547000,0,104,191,4ba7157e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6547000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e8112d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e8112d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8112d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100091 td 0xc6547240
sched_switch(c6547240,0,104,191,4b7c372e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6547240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e810fd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4d0,e810fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe810fd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100090 td 0xc6547480
sched_switch(c6547480,0,104,191,a4c861da,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6547480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c65418c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c65418c0,c65418d8,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f490,e810cd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f490,e810cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe810cd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100089 td 0xc65476c0
sched_switch(c65476c0,0,104,191,a128ad06,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c65476c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5e5d680,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c5e5d680,c5e5d698,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4c0,e815ad38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5f3f4c0,e815ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe815ad70, ebp = 0 ---

Tracing command kernel pid 0 tid 100088 td 0xc6547900
sched_switch(c6547900,0,104,191,a0e7711e,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6547900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5e7c100,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c5e7c100,c5e7c118,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5c4d360,e815dd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5c4d360,e815dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe815dd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100030 td 0xc5b5f000
sched_switch(c5b5f000,0,104,191,f94b86a6,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b5f000,0,c0c7a69a,24b,c5b5f000,...) at sleepq_switch+0x15f
sleepq_wait(c5abeb80,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63
msleep_spin(c5abeb80,c5abeb98,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d
taskqueue_thread_loop(c5b6649c,c5555d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0x94
fork_exit(c08baf50,c5b6649c,c5555d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5555d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100028 td 0xc5b5f480
sched_switch(c5b5f480,0,104,191,39b65fdc,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b5f480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5b05500,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c5b05500,c5b05518,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5b5b074,c554bd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c5b5b074,c554bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc554bd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100021 td 0xc5a85480
sched_switch(c5a85480,0,104,191,737dd5ea,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5a85480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5a66e80,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c5a66e80,c5a66e98,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0ddd5c8,c5504d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c0ddd5c8,c5504d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5504d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100019 td 0xc5a85900
sched_switch(c5a85900,0,104,191,609f5440,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5a85900,0,c0c7a69a,24b,c5a85900,...) at sleepq_switch+0x15f
sleepq_wait(c5a67240,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63
msleep_spin(c5a67240,c5a67258,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9a860,c54fed38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0x94
fork_exit(c08baf50,c0d9a860,c54fed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54fed70, ebp = 0 ---

Tracing command kernel pid 0 tid 100018 td 0xc5a85b40
sched_switch(c5a85b40,0,104,191,609f3204,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5a85b40,0,c0c7a69a,24b,c5a85b40,...) at sleepq_switch+0x15f
sleepq_wait(c5a67240,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63
msleep_spin(c5a67240,c5a67258,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9a860,c54fbd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0x94
fork_exit(c08baf50,c0d9a860,c54fbd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54fbd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100017 td 0xc5971240
sched_switch(c5971240,0,104,191,609f07ac,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971240,0,c0c7a69a,24b,c5971240,...) at sleepq_switch+0x15f
sleepq_wait(c5a67240,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63
msleep_spin(c5a67240,c5a67258,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9a860,c54f8d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0x94
fork_exit(c08baf50,c0d9a860,c54f8d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f8d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100014 td 0xc5971900
sched_switch(c5971900,0,104,191,609dbb5c,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5a673c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c5a673c0,c5a673d8,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0dc9384,c54efd38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c0dc9384,c54efd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54efd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100010 td 0xc59b5240
sched_switch(c59b5240,0,104,191,73b9f616,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5955c40,0,c0c6c015,0,0,...) at sleepq_wait+0x63
_sleep(c5955c40,c5955c58,0,c0c6c015,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0ddc060,c54e3d38,c0c717a4,343,c0dc8ae0,...) at
taskqueue_thread_loop+0xba
fork_exit(c08baf50,c0ddc060,c54e3d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e3d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100000 td 0xc0dc8d90
sched_switch(c0dc8d90,0,104,191,8f890d82,...) at sched_switch+0x406
mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200
sleepq_switch(c0dc8d90,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dc8ae0,44,c0c78703,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dc8ae0,0,44,c0c78703,2710,...) at _sleep+0x339
scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x23e
mi_startup() at mi_startup+0x96
begin() at begin+0x2c


----------------------------------------------------------------------
 TOP OF DMESG

KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RC1 #1: Sat Oct 24 18:12:41 UTC 2009
    root@box8.ws.local:/usr/obj/usr/src/sys/WITHDDB
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2400.10-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNXT-ID,xTPR>
real memory  = 2147483648 (2048 MB)
avail memory = 2086248448 (1989 MB)


----------------------------------------------------------------------
 KERNEL CONFIG

include GENERIC

ident   WITHDDB

options KDB
options DDB
options BREAK_TO_DEBUGGER

makeoptions        DEBUG=-g
options     INVARIANTS
options     INVARIANT_SUPPORT
options     WITNESS
options     DEBUG_LOCKS
options     DEBUG_VFS_LOCKS
options     DIAGNOSTIC
From rnoland at FreeBSD.org  Sun Oct 25 03:46:04 2009
From: rnoland at FreeBSD.org (Robert Noland)
Date: Sun Oct 25 03:46:42 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
 unavailable"
In-Reply-To: <4AE33CF9.3050308@buchlovice.org>
References: <4AD710D6.70404@buchlovice.org>
	<1255633430.2175.12.camel@balrog.2hip.net>
	<4AD779FC.1070204@buchlovice.org>
	<1256146709.2310.9.camel@balrog.2hip.net>
	<4AE33CF9.3050308@buchlovice.org>
Message-ID: <1256442352.3303.10.camel@balrog.2hip.net>

On Sat, 2009-10-24 at 19:44 +0200, Radek Val??ek wrote:
> Robert Noland napsal(a):
> > On Thu, 2009-10-15 at 21:37 +0200, Radek Val??ek wrote:
> >   
> >> Robert Noland napsal(a):
> >>     
> >>> On Thu, 2009-10-15 at 14:08 +0200, Radek Val??ek wrote:
> >>>   
> >>>       
> >>>> Hi,
> >>>>
> >>>> I want to ask if there is something new in adding support to 
> >>>> gptzfsboot/zfsboot for reading gang-blocks?
> >>>>    
> >>>>         
> >
> > I think that the gang block patch will work, though still haven't gotten
> > it tested.  However, I'm fairly confident that the issue is not gang
> > block related.  Right now, I have setup a disk like this:
> >
> > =>        34  1953525101  ada1  GPT  (932G)
> >           34         128     1  freebsd-boot  (64K)
> >          162     8388608     2  freebsd-swap  (4.0G)
> >      8388770   648019968     3  freebsd-zfs  (309G)
> >    656408738   648019968     4  freebsd-zfs  (309G)
> >   1304428706   648019968     5  freebsd-zfs  (309G)
> >   1952448674     1076461        - free -  (526M)
> >
> > Note that this is not a raidz pool right now.  It is just 3 toplevel
> > partitions setup as a single pool.  I finally have this configuration
> > working reliably.  At least in this case, the issue is due to all of the
> > partitions not being probed during early boot and so not being added to
> > the list of vdevs for the pool.  When zio_read finds a dva that points
> > to a device it doesn't know about, it gives up and whines.
> >
> > Can you detail for me how you have everything configured, so that I can
> > try to replicate it.  gpart show, zpool status and zpool get all <pool>
> > would be good.  I'm not sure that I have enough spare disks lying around
> > to do this properly, but maybe I can use virtual disks or something.
> >
> > robert.
> >
> >   
> 
> Sorry for not responding so long. Here are details you want from me:
> 
> # gpart show
> =>        34  1953525101  ad6  GPT  (932G)
>           34         128    1  freebsd-boot  (64K)
>          162  1953524973    2  freebsd-zfs  (932G)
> 
> =>        34  1953525101  ad8  GPT  (932G)
>           34         128    1  freebsd-boot  (64K)
>          162  1953524973    2  freebsd-zfs  (932G)
> 
> =>        34  1953525101  ad10  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162  1953524973     2  freebsd-zfs  (932G)
> 
> =>        34  1953525101  ad12  GPT  (932G)
>           34         128     1  freebsd-boot  (64K)
>          162  1953524973     2  freebsd-zfs  (932G)
> 
> # zpool status
>   pool: z
>  state: ONLINE
>  scrub: none requested
> config:
> 
>     NAME        STATE     READ WRITE CKSUM
>     z           ONLINE       0     0     0
>       raidz1    ONLINE       0     0     0
>         ad6p2   ONLINE       0     0     0
>         ad8p2   ONLINE       0     0     0
>         ad10p2  ONLINE       0     0     0
>         ad12p2  ONLINE       0     0     0
> 
> errors: No known data errors
> 
> # zpool get all z
> NAME  PROPERTY       VALUE       SOURCE
> z     size           3.62T       -
> z     used           4.62G       -
> z     available      3.62T       -
> z     capacity       0%          -
> z     altroot        -           default
> z     health         ONLINE      -
> z     guid           17857007133862981114  -
> z     version        13          default
> z     bootfs         z/system    local
> z     delegation     on          default
> z     autoreplace    off         default
> z     cachefile      -           default
> z     failmode       wait        default
> z     listsnapshots  off         default
> 
> I've tested your patches but it seems that you're right and it's not 
> gang related issue. I was able to discover these things on a fully 
> functional zfs pool (system compiled with your patches):
> 
> 1, If I overwrite the file /boot/loader.conf (with copy of itself, or 
> when upgrading kernel/world), next reboot comes with these messages:
> 
> BTX loader 1.00  BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> 
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root@ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> 
> Then I'm still able to boot the system, but I must set the boot 
> variables included in loader.conf by hand
> 
> 2, Next I overwrite the file /boot/loader (with copy of itself, or when 
> upgrading kernel/world) and reboot comes with these messages:
> 
> BTX loader 1.00  BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> 
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root@ztest, Thu Oct 22 22:27:22 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> ZFS: i/o error - all block copies unavailable
> Unable to load a kernel!
> 
> After that I'm no longer able to boot the system from zfs pool.
> 
> Hope you have some ideas...

Ok, can you retest with -CURRENT?  I just committed some fixes on
Friday.  I'm having real difficulty in reproducing these issues.  Most
of the problems that I've run into so far had to do with the system not
knowing about all of the vdevs when it wanted to read something.  In
your case, it looks like you are making it to boot3 and it appears to be
seeing all 4 of your disks.  Right now, I've been trying to track down
an issue wher the MOS can't be read, which basically means that we have
screwed up the root block pointer somehow.  I haven't been able to
reproduce that issue in qemu, I have been able to reproduce it with
VirtualBox, but it is really time consuming trying to work in vbox since
I have to reconvert all of the disk images every time I make a change.
I'm actually a bit concerned that it hinges on how many drives are
visible to the bios at various points in time.

robert.

> vaLin
> 
> >>> Ok, I can't figure out any way to test this... beyond the fact that it
> >>> builds and doesn't break my currently working setup.  Can you give this
> >>> a try?  It should still report if it finds gang blocks, but hopefully
> >>> now will read them as well.
> >>>
> >>> robert.
> >>>
> >>>   
> >>>       
> >> Big thanks for the patches Robert, I will definitely test them as soon 
> >> as possible (tomorrow) and report the results immediately to list. I can 
> >> repeat this issue probably at any time (up to cca 30 times tested with 
> >> the same result), so don't bother about the broken booting, I'm prepared 
> >> for it...
> >>
> >> vaLin
> >>     
> >>>>  From Sun's docs:
> >>>>
> >>>> Gang blocks
> >>>>
> >>>> When there is not enough contiguous space to write a complete block, the ZIO
> >>>> pipeline will break the I/O up into smaller 'gang blocks' which can later be
> >>>> assembled transparently to appear as complete blocks.
> >>>>
> >>>> Everything works fine for me, until I rewrite kernel/world after system 
> >>>> upgrade to latest one (releng_8). After this am I no longer able to boot 
> >>>> from zfs raidz1 pool with following messages:
> >>>>
> >>>>  >/ ZFS: i/o error - all block copies unavailable
> >>>> />/ ZFS: can't read MOS
> >>>> />/ ZFS: unexpected object set type lld
> >>>> />/ ZFS: unexpected object set type lld
> >>>> />/
> >>>> />/ FreeBSD/i386 boot
> >>>> />/ Default: z:/boot/kernel/kernel
> >>>> />/ boot:
> >>>> />/ ZFS: unexpected object set type lld
> >>>> />/
> >>>> />/ FreeBSD/i386 boot
> >>>> />/ Default: tank:/boot/kernel/kernel
> >>>> />/ boot:
> >>>> //
> >>>> /I presume it's the same issue as talked in june-2009 current mailing 
> >>>> list 
> >>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
> >>>>
> >>>> Any success in that matter?
> >>>>
> >>>> Thnx for answer.
> >>>>
> >>>> vaLin
> >>>> _______________________________________________
> >>>> freebsd-current@freebsd.org mailing list
> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
> >>>>     
> >>>>         
> 
-- 
Robert Noland <rnoland@FreeBSD.org>
FreeBSD

From carl at chave.us  Sun Oct 25 16:00:19 2009
From: carl at chave.us (Carl Chave)
Date: Sun Oct 25 16:01:19 2009
Subject: kern/139806: [zfs] [panic] Write attempt to file in ZFS
	snapshot dir causes panic
Message-ID: <200910251600.n9PG0IeL019956@freefall.freebsd.org>

The following reply was made to PR kern/139806; it has been noted by GNATS.

From: Carl Chave <carl@chave.us>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/139806: [zfs] [panic] Write attempt to file in ZFS snapshot 
	dir causes panic
Date: Sun, 25 Oct 2009 10:47:40 -0400

 Rebuilt zfs.ko using:
 
 head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
 (version 197861 + this patch)
 head/sys/cddl/compat/opensolaris/kern/opensolaris_policy.c (version 197861)
 head/sys/cddl/compat/opensolaris/sys/policy.h (version 197861)
 
 Resulting zfs.ko works as expected and does not panic when attempting
 to modify an existing snapshot file as described in the original PR.
 
 On Thu, Oct 22, 2009 at 3:50 PM, Jaakko Heinonen <jh@freebsd.org> wrote:
 >
 > Hi,
 >
 > On 2009-10-21, Carl Chave wrote:
 >> Fixit# echo hello >> test.txt
 >> panic: dirtying snapshot!
 >
 > The problem seems to be that in certain conditions zfs_freebsd_access()
 > uses only vaccess(9) for access check. However vaccess(9) doesn't handle
 > the read-only file system case.
 >
 > Could you try this patch?
 >
 > --- patch begins here ---
 > Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 > --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c =A0(revisi=
 on 198368)
 > +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c =A0(workin=
 g copy)
 > @@ -3989,7 +3989,12 @@
 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0struct thread *a_td;
 > =A0 =A0 =A0 =A0} */ *ap;
 > =A0{
 > + =A0 =A0 =A0 int error;
 >
 > + =A0 =A0 =A0 error =3D zfs_access(ap->a_vp, ap->a_accmode, 0, ap->a_cred=
 , NULL);
 > + =A0 =A0 =A0 if (error !=3D 0)
 > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return (error);
 > +
 > =A0 =A0 =A0 =A0/*
 > =A0 =A0 =A0 =A0 * ZFS itself only knowns about VREAD, VWRITE and VEXEC, t=
 he rest
 > =A0 =A0 =A0 =A0 * we have to handle by calling vaccess().
 > @@ -3999,11 +4004,11 @@
 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0znode_t *zp =3D VTOZ(vp);
 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0znode_phys_t *zphys =3D zp->z_phys;
 >
 > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 return (vaccess(vp->v_type, zphys->zp_mode,=
  zphys->zp_uid,
 > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 zphys->zp_gid, ap->a_accmode, ap->a=
 _cred, NULL));
 > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 error =3D vaccess(vp->v_type, zphys->zp_mod=
 e, zphys->zp_uid,
 > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 zphys->zp_gid, ap->a_accmode, ap->a=
 _cred, NULL);
 > =A0 =A0 =A0 =A0}
 >
 > - =A0 =A0 =A0 return (zfs_access(ap->a_vp, ap->a_accmode, 0, ap->a_cred, =
 NULL));
 > + =A0 =A0 =A0 return (error);
 > =A0}
 >
 > =A0static int
 > --- patch ends here ---
 >
 > --
 > Jaakko
 >
From gcorcoran at rcn.com  Sun Oct 25 21:23:44 2009
From: gcorcoran at rcn.com (Gary Corcoran)
Date: Sun Oct 25 21:23:50 2009
Subject: Western Digital RE4-GP WD2002FYPS 2TB hard disks
Message-ID: <4AE4BB46.2040608@rcn.com>

I'm thinking of building a ZFS-based home server using just the
motherboard ICH10R SATA connectors, and a few Western Digital RE4-GP WD2002FYPS 2TB
disk drives.  These are the "Raid Edition enterprise class" drives.  I've read of
problems with these drives when connected to mid-to-high end SATA RAID controllers
from Adaptec, Areca, and LSI.  But I was wondering if anyone has had any problems
just hooking them up to to the Intel motherboard ICH10R SATA connections as simple
(individual) disks? Or has anyone had any success in using these drives for a raidz
setup?

Thanks,
Gary
From rincebrain at gmail.com  Sun Oct 25 22:00:39 2009
From: rincebrain at gmail.com (Rich)
Date: Sun Oct 25 22:00:45 2009
Subject: extattr/xattr on 8.0-RC1 ZFS?
Message-ID: <5da0588e0910251500q79f72822w87581c2823360a71@mail.gmail.com>

Hello world,
I'm running 8.0-RC1 built from latest RELENG_8 on September 29th.

I found a use case for extended attrs in my day to day life - I wanted
to make everything in a given directory have a default mask of
group-writable.

I tried:
# setfacl -d -m u::rwx,g::rwx,mask::rwx /bukkit/home/rercola/oose
setfacl: acl_get_file() failed: Operation not supported
# zfs get xattr bukkit/home/rercola
NAME                 PROPERTY  VALUE                SOURCE
bukkit/home/rercola  xattr     off                  temporary
# zfs set xattr=on bukkit/home/rercola
# zfs get xattr bukkit/home/rercola
NAME                 PROPERTY  VALUE                SOURCE
bukkit/home/rercola  xattr     off                  temporary

What's going on? Wikipedia says, uncited, that FBSD ZFS has extended
attributes. The FBSD wiki page on ZFS says that extattr is "done", but
the description hasn't changed from before this was the case.
I tried, naively:

# extattrctl start /bukkit/home/rercola/
extattrctl start: Operation not supported

What's the correct thing to do here? Is it secretly unsupported and
the documentation is imprecise? Or am I just an idiot?

- Rich

-- 

Todo corpo mergulhado numa banheira faz tocar o telefone. -- Joseph Murphy
From josh at multipart-mixed.com  Mon Oct 26 00:38:12 2009
From: josh at multipart-mixed.com (Josh Carter)
Date: Mon Oct 26 00:38:19 2009
Subject: Western Digital RE4-GP WD2002FYPS 2TB hard disks
In-Reply-To: <4AE4BB46.2040608@rcn.com>
References: <4AE4BB46.2040608@rcn.com>
Message-ID: <08FA3881-AE46-441B-841E-F3E314713256@multipart-mixed.com>

Gary,

Just one data point, we've been evaluating 60 of these drives and  
we've had some problems. We've had a couple fall over, and WD said  
they had some production issues which are now worked out. We'll  
see. :) We had problems on both SAS and direct-attach SATA.

The drives have great specs, however, and we're going to continue  
working with WD. I'm pretty sure WD is making a serious effort to get  
into the enterprise space with these drives, so they'll get their  
stuff together. Right at the moment I'd have some reservations about  
putting these drives into a production environment.

Best regards,
Josh


On Oct 25, 2009, at 2:55 PM, Gary Corcoran wrote:

> I'm thinking of building a ZFS-based home server using just the
> motherboard ICH10R SATA connectors, and a few Western Digital RE4-GP  
> WD2002FYPS 2TB
> disk drives.  These are the "Raid Edition enterprise class" drives.   
> I've read of
> problems with these drives when connected to mid-to-high end SATA  
> RAID controllers
> from Adaptec, Areca, and LSI.  But I was wondering if anyone has had  
> any problems
> just hooking them up to to the Intel motherboard ICH10R SATA  
> connections as simple
> (individual) disks? Or has anyone had any success in using these  
> drives for a raidz
> setup?
>
> Thanks,
> Gary
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"

From solon at pyro.de  Mon Oct 26 01:30:39 2009
From: solon at pyro.de (Solon Lutz)
Date: Mon Oct 26 01:30:47 2009
Subject: raidz slowing down
In-Reply-To: <alpine.BSF.2.00.0910251941080.2854@ibyngvyr.purzvxnyf.bet>
References: <886802879.20091008113716@pyro.de>
	<alpine.BSF.2.00.0910251941080.2854@ibyngvyr.purzvxnyf.bet>
Message-ID: <1791999980.20091026023032@pyro.de>

> Did you ever get any response? I have a very similar sounding issue with 
> my raidz2. I've always assumed it was because the volume was nearly full 
> and maybe some fragmentation or something. All of my devices are on MPT 
> controllers, so I don't think that the highpoint device is an issue.

Nope, no responses...

Since I was working on a rescue operation, I didn't have the patience
to eliminated  all kinds of errors and so I swapped out da1
(maybe a little bit slow or buggy?) and used the forensics version of dd
'dcfldd'. It has a split option and I suspected that ZFS has problems when
writing huge amounts of continous data streams - so I split the 10TB in
100GB files, which took about 11 hours.

I don't know if this is general problem, or if this only happens when the
input id delivered at a much higher data-rate. In this case, the HW-RAID/zpool was
able to deliver data at 600MB/s while the RAIDZ/zpool could only write at 130MB/s.

The dynamics of this 'slow-down' that I could watch via gstat looked like the
whole access on the device level was desynchronizing completely.
In the end, before I quit the process, write-speed was down to 5MB/s !

But as I mentioned earlier, I had no nerves for bug-hunting, due to a
bigger (still unsolved) problem at hand.

Maybe somebody else likes to investigate? I'm busy with ZFS forensics...

solon

> On Thu, 8 Oct 2009, Solon Lutz wrote:

>> I built a 9x hdd 11TB raidz for some rescue purposes and started
>> copying an image from another partition via "dd if=/dev/da0..." to it.
>> It consists of: ad4 da1 da2 da3 da4 da5 da6 da7 da8, da1 to da8 are
>> connected via two highpoint controllers.

>> In the beginning write speeds were quite fair:

>> dT: 1.002s  w: 1.000s
>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>>    0    424      0      0    0.0    424  52483   33.9   84.6| ad4
>>    0      0      0      0    0.0      0      0    0.0    0.0| da0
>>   35    356      0      0    0.0    356  44584   76.4  124.5| da1
>>   35    296      0      0    0.0    296  36919   84.5  121.0| da2
>>   34    361      0      0    0.0    361  45111   75.5  124.7| da3
>>   35    346      0      0    0.0    346  43196   78.6  123.2| da4
>>   35    344      0      0    0.0    344  42940   80.0  124.7| da5
>>   35    343      0      0    0.0    343  42812   80.7  124.5| da6
>>   35    344      0      0    0.0    344  43051   79.8  123.9| da7
>>   34    342      0      0    0.0    342  42796   80.6  124.4| da8

>> Now, some 10 hours and 2.5TB later, it look like that most of the time:

>> dT: 1.002s  w: 1.000s
>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>>    0     10      0      0    0.0     10      6    0.8    0.2| ad4
>>    0      0      0      0    0.0      0      0    0.0    0.0| da0
>>    4     13      0      0    0.0     13      8  550.4  178.5| da1
>>    0     12      0      0    0.0     12      7    0.7    0.2| da2
>>    0     11      0      0    0.0     11      7    0.7    0.2| da3
>>    0     10      0      0    0.0     10      5    0.6    0.2| da4
>>    0     11      0      0    0.0     11      6    0.9    0.3| da5
>>    0     12      0      0    0.0     12      7    0.7    0.2| da6
>>    0     11      0      0    0.0     11      7    0.7    0.2| da7
>>    0      9      0      0    0.0      9      6    0.8    0.2| da8


>> da1 seems to be busy most of time and every few seconds all the other
>> devices write some data with nearly normal speed:

>> dT: 1.003s  w: 1.000s
>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>>    0    254      0      0    0.0    254  31331   34.9   35.4| ad4
>>    0      0      0      0    0.0      0      0    0.0    0.0| da0
>>    4      0      0      0    0.0      0      0    0.0    0.0| da1
>>    0    254      0      0    0.0    254  31346  107.4  104.5| da2
>>    0    256      0      0    0.0    256  31345  108.1  104.0| da3
>>    0    255      0      0    0.0    255  31345  110.2  105.1| da4
>>   35    200      0      0    0.0    200  24912  143.3  115.0| da5
>>   35    211      0      0    0.0    211  26303  137.8  114.9| da6
>>   35    210      0      0    0.0    210  26079  139.3  114.9| da7
>>   35    209      0      0    0.0    209  25952  135.2  113.7| da8

>> Sometimes it even gets back to 'normal' behaviour, but never reaches
>> the speeds it once had:

>> dT: 1.002s  w: 1.000s
>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>>   35    274      0      0    0.0    274  34334   44.2   66.6| ad4
>>    0   1166   1166 149243    0.1      0      0    0.0   14.3| da0
>>   35    120      0      0    0.0    120  14717   94.4   64.5| da1
>>   35     96      0      0    0.0     96  11665  113.9   64.3| da2
>>   35    100      0      0    0.0    100  12288   98.7   63.9| da3
>>   35    103      0      0    0.0    103  12496   93.4   59.4| da4
>>   34    112      0      0    0.0    112  13694  106.1   67.4| da5
>>   35     71      0      0    0.0     71   8596  115.3   66.8| da6
>>   35    116      0      0    0.0    116  14205  101.7   67.3| da7
>>   35     83      0      0    0.0     83  10066  112.2   65.9| da8

>> Syslog reports the following:

>> Oct  8 09:53:40 radium kernel: hptrr: start channel [0,0]
>> Oct  8 09:53:40 radium kernel: hptrr: channel [0,0] started successfully
>> Oct  8 09:57:44 radium kernel: hptrr: start channel [0,0]
>> Oct  8 09:57:45 radium kernel: hptrr: channel [0,0] started successfully
>> Oct  8 10:54:26 radium kernel: hptrr: start channel [0,0]
>> Oct  8 10:54:27 radium kernel: hptrr: channel [0,0] started successfully
>> Oct  8 11:10:29 radium kernel: hptrr: start channel [0,0]
>> Oct  8 11:10:30 radium kernel: hptrr: channel [0,0] started successfully
>> Oct  8 11:17:27 radium kernel: hptrr: start channel [0,0]
>> Oct  8 11:17:27 radium kernel: hptrr: channel [0,0] started successfully

>> Is this a problem of the hptrr device or is da1 failing?




































>> Mit freundlichen Gr??en
>> Best regards,

>> Solon Lutz


>> +-----------------------------------------------+
>> | Pyro.Labs Berlin -  Creativity for tomorrow   |
>> | Wasgenstrasse 75/13 - 14129 Berlin, Germany   |
>> | www.pyro.de - phone + 49 - 30 - 48 48 58 58   |
>> | info@pyro.de - fax + 49 - 30 - 80 94 03 52    |
>> +-----------------------------------------------+

>> _______________________________________________
>> freebsd-fs@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"



From bugmaster at FreeBSD.org  Mon Oct 26 11:06:58 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Oct 26 11:08:01 2009
Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org
Message-ID: <200910261106.n9QB6vE9043743@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o kern/139806  fs         [zfs] [panic] Write attempt to file in ZFS snapshot di
o kern/139725  fs         [zfs] zdb(1) dumps core on i386 when examining zpool c
o kern/139715  fs         [zfs] vfs.numvnodes leak on busy zfs
o bin/139651   fs         [nfs] mount(8): read-only remount of NFS volume does n
o kern/139597  fs         [patch] [tmpfs] tmpfs initializes va_gen but doesn't u
o kern/139564  fs         [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo
o kern/139407  fs         [smbfs] [panic] smb mount causes system crash if remot
o kern/139363  fs         [nfs] diskless root nfs mount from non FreeBSD server 
o kern/138790  fs         [zfs] ZFS ceases caching when mem demand is high
o kern/138524  fs         [msdosfs] disks and usb flashes/cards with Russian lab
o kern/138421  fs         [ufs] [patch] remove UFS label limitations
o kern/138367  fs         [tmpfs] [panic] 'panic: Assertion pages > 0 failed' wh
o kern/138202  fs         mount_msdosfs(1) see only 2Gb
o kern/138109  fs         [extfs] [patch] Minor cleanups to the sys/gnu/fs/ext2f
f kern/137037  fs         [zfs] [hang] zfs rollback on root causes FreeBSD to fr
o kern/136968  fs         [ufs] [lor] ufs/bufwait/ufs (open)
o kern/136945  fs         [ufs] [lor] filedesc structure/ufs (poll)
o kern/136944  fs         [ffs] [lor] bufwait/snaplk (fsync)
o kern/136873  fs         [ntfs] Missing directories/files on NTFS volume
o kern/136865  fs         [nfs] [patch] NFS exports atomic and on-the-fly atomic
o kern/136470  fs         [nfs] Cannot mount / in read-only, over NFS
o kern/135594  fs         [zfs] Single dataset unresponsive with Samba
o kern/135546  fs         [zfs] zfs.ko module doesn't ignore zpool.cache filenam
o kern/135469  fs         [ufs] [panic] kernel crash on md operation in ufs_dirb
o kern/135050  fs         [zfs] ZFS clears/hides disk errors on reboot
o kern/134491  fs         [zfs] Hot spares are rather cold...
o kern/133980  fs         [panic] [ffs] panic: ffs_valloc: dup alloc
o kern/133676  fs         [smbfs] [panic] umount -f'ing a vnode-based memory dis
o kern/133614  fs         [smbfs] [panic] panic: ffs_truncate: read-only filesys
o kern/133174  fs         [msdosfs] [patch] msdosfs must support utf-encoded int
f kern/133150  fs         [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w
o kern/132960  fs         [ufs] [panic] panic:ffs_blkfree: freeing free frag
o kern/132597  fs         [tmpfs] [panic] tmpfs-related panic while interrupting
o kern/132397  fs         reboot causes filesystem corruption (failure to sync b
o kern/132331  fs         [ufs] [lor] LOR ufs and syncer
o kern/132237  fs         [msdosfs] msdosfs has problems to read MSDOS Floppy
o kern/132145  fs         [panic] File System Hard Crashes
o kern/131995  fs         [nfs] Failure to mount NFSv4 server
o kern/131441  fs         [unionfs] [nullfs] unionfs and/or nullfs not combineab
o kern/131360  fs         [nfs] poor scaling behavior of the NFS server under lo
o kern/131342  fs         [nfs] mounting/unmounting of disks causes NFS to fail
o bin/131341   fs         makefs: error "Bad file descriptor"  on the mount poin
o kern/131086  fs         [ext2fs] [patch] mkfs.ext2 creates rotten partition
o kern/130979  fs         [smbfs] [panic] boot/kernel/smbfs.ko
o kern/130920  fs         [msdosfs] cp(1) takes 100% CPU time while copying file
o kern/130229  fs         [iconv] usermount fails on fs that need iconv
o kern/130210  fs         [nullfs] Error by check nullfs
o kern/129760  fs         [nfs] after 'umount -f' of a stale NFS share FreeBSD l
o kern/129488  fs         [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: 
o kern/129231  fs         [ufs] [patch] New UFS mount (norandom) option - mostly
o kern/129152  fs         [panic] non-userfriendly panic when trying to mount(8)
o kern/129059  fs         [zfs] [patch] ZFS bootloader whitelistable via WITHOUT
f kern/128829  fs         smbd(8) causes periodic panic on 7-RELEASE
f kern/128173  fs         [ext2fs] ls gives "Input/output error" on mounted ext3
o kern/127659  fs         [tmpfs] tmpfs memory leak
o kern/127420  fs         [gjournal] [panic] Journal overflow on gmirrored gjour
o kern/127029  fs         [panic] mount(8): trying to mount a write protected zi
o kern/126287  fs         [ufs] [panic] Kernel panics while mounting an UFS file
s kern/125738  fs         [zfs] [request] SHA256 acceleration in ZFS
f kern/125536  fs         [ext2fs] ext 2 mounts cleanly but fails on commands li
f kern/124621  fs         [ext3] [patch] Cannot mount ext2fs partition
f bin/124424   fs         [zfs] zfs(8): zfs list -r shows strange snapshots' siz
o kern/123939  fs         [msdosfs] corrupts new files
o kern/122380  fs         [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash
o bin/122172   fs         [fs]: amd(8) automount daemon dies on 6.3-STABLE i386,
o kern/122047  fs         [ext2fs] [patch] incorrect handling of UF_IMMUTABLE / 
o bin/121898   fs         [nullfs] pwd(1)/getcwd(2) fails with Permission denied
o bin/121779   fs         [ufs] snapinfo(8) (and related tools?) only work for t
o bin/121366   fs         [zfs] [patch] Automatic disk scrubbing from periodic(8
o bin/121072   fs         [smbfs] mount_smbfs(8) cannot normally convert the cha
f kern/120991  fs         [panic] [fs] [snapshot] System crashes when manipulati
o kern/120483  fs         [ntfs] [patch] NTFS filesystem locking changes
o kern/120482  fs         [ntfs] [patch] Sync style changes between NetBSD and F
f kern/119735  fs         [zfs] geli + ZFS + samba starting on boot panics 7.0-B
o kern/118912  fs         [2tb] disk sizing/geometry problem with large array
o kern/118713  fs         [minidump] [patch] Display media size required for a k
o bin/118249   fs         mv(1): moving a directory changes its mtime
o kern/118107  fs         [ntfs] [panic] Kernel panic when accessing a file at N
o bin/117315   fs         [smbfs] mount_smbfs(8) and related options can't mount
o kern/117314  fs         [ntfs] Long-filename only NTFS fs'es cause kernel pani
o kern/117158  fs         [zfs] zpool scrub causes panic if geli vdevs detach on
o bin/116980   fs         [msdosfs] [patch] mount_msdosfs(8) resets some flags f
o kern/116913  fs         [ffs] [panic] ffs_blkfree: freeing free block
p kern/116608  fs         [msdosfs] [patch] msdosfs fails to check mount options
o kern/116583  fs         [ffs] [hang] System freezes for short time when using 
o kern/116170  fs         [panic] Kernel panic when mounting /tmp
o kern/115645  fs         [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex
o bin/115361   fs         [zfs] mount(8) gets into a state where it won't set/un
o kern/114955  fs         [cd9660] [patch] [request] support for mask,dirmask,ui
o kern/114847  fs         [ntfs] [patch] [request] dirmask support for NTFS ala 
o kern/114676  fs         [ufs] snapshot creation panics: snapacct_ufs2: bad blo
o bin/114468   fs         [patch] [request] add -d option to umount(8) to detach
o kern/113852  fs         [smbfs] smbfs does not properly implement DFS referral
o bin/113838   fs         [patch] [request] mount(8): add support for relative p
o bin/113049   fs         [patch] [request] make quot(8) use getopt(3) and show 
o kern/112658  fs         [smbfs] [patch] smbfs and caching problems (resolves b
f usb/112640   fs         [ext2fs] [hang] Kernel freezes when writing a file to 
o kern/111843  fs         [msdosfs] Long Names of files are incorrectly created 
o kern/111782  fs         [ufs] dump(8) fails horribly for large filesystems
s bin/111146   fs         [2tb] fsck(8) fails on 6T filesystem
o kern/109024  fs         [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not 
o kern/109010  fs         [msdosfs] can't mv directory within fat32 file system
o bin/107829   fs         [2TB] fdisk(8): invalid boundary checking in fdisk / w
o kern/106030  fs         [ufs] [panic] panic in ufs from geom when a dead disk 
o kern/105093  fs         [ext2fs] [patch] ext2fs on read-only media cannot be m
o kern/104406  fs         [ufs] Processes get stuck in "ufs" state under persist
o kern/104133  fs         [ext2fs] EXT2FS module corrupts EXT2/3 filesystems
o kern/103035  fs         [ntfs] Directories in NTFS mounted disc images appear 
o kern/101324  fs         [smbfs] smbfs sometimes not case sensitive when it's s
o kern/99290   fs         [ntfs] mount_ntfs ignorant of cluster sizes
o kern/97377   fs         [ntfs] [patch] syntax cleanup for ntfs_ihash.c
o kern/95222   fs         [iso9660] File sections on ISO9660 level 3 CDs ignored
o kern/94849   fs         [ufs] rename on UFS filesystem is not atomic
o kern/94769   fs         [ufs] Multiple file deletions on multi-snapshotted fil
o kern/94733   fs         [smbfs] smbfs may cause double unlock
o kern/93942   fs         [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D
o kern/92272   fs         [ffs] [hang] Filling a filesystem while creating a sna
f kern/91568   fs         [ufs] [panic] writing to UFS/softupdates DVD media in 
o kern/91134   fs         [smbfs] [patch] Preserve access and modification time 
a kern/90815   fs         [smbfs] [patch] SMBFS with character conversions somet
o kern/89991   fs         [ufs] softupdates with mount -ur causes fs UNREFS
o kern/88657   fs         [smbfs] windows client hang when browsing a samba shar
o kern/88266   fs         [smbfs] smbfs does not implement UIO_NOCOPY and sendfi
o kern/87859   fs         [smbfs] System reboot while umount smbfs.
o kern/86587   fs         [msdosfs] rm -r /PATH fails with lots of small files
o kern/85326   fs         [smbfs] [panic] saving a file via samba to an overquot
o kern/84589   fs         [2TB] 5.4-STABLE unresponsive during background fsck 2
o kern/80088   fs         [smbfs] Incorrect file time setting on NTFS mounted vi
o kern/77826   fs         [ext2fs] ext2fs usb filesystem will not mount RW
o kern/73484   fs         [ntfs] Kernel panic when doing `ls` from the client si
o bin/73019    fs         [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino
o kern/71774   fs         [ntfs] NTFS cannot "see" files on a WinXP filesystem
o kern/68978   fs         [panic] [ufs] crashes with failing hard disk, loose po
o kern/65920   fs         [nwfs] Mounted Netware filesystem behaves strange
o kern/65901   fs         [smbfs] [patch] smbfs fails fsx write/truncate-down/tr
o kern/61503   fs         [smbfs] mount_smbfs does not work as non-root
o kern/55617   fs         [smbfs] Accessing an nsmb-mounted drive via a smb expo
o kern/51685   fs         [hang] Unbounded inode allocation causes kernel to loc
o kern/51583   fs         [nullfs] [patch] allow to work with devices and socket
o kern/36566   fs         [smbfs] System reboot with dead smb mount and umount
o kern/18874   fs         [2TB] 32bit NFS servers export wrong negative values t

141 problems total.

From avg at icyb.net.ua  Mon Oct 26 12:01:30 2009
From: avg at icyb.net.ua (Andriy Gapon)
Date: Mon Oct 26 12:01:37 2009
Subject: extattr/xattr on 8.0-RC1 ZFS?
In-Reply-To: <5da0588e0910251500q79f72822w87581c2823360a71@mail.gmail.com>
References: <5da0588e0910251500q79f72822w87581c2823360a71@mail.gmail.com>
Message-ID: <4AE58F93.3040300@icyb.net.ua>

on 26/10/2009 00:00 Rich said the following:
> Hello world,
> I'm running 8.0-RC1 built from latest RELENG_8 on September 29th.
> 
> I found a use case for extended attrs in my day to day life - I wanted
> to make everything in a given directory have a default mask of
> group-writable.
> 
> I tried:
> # setfacl -d -m u::rwx,g::rwx,mask::rwx /bukkit/home/rercola/oose
> setfacl: acl_get_file() failed: Operation not supported
> # zfs get xattr bukkit/home/rercola
> NAME                 PROPERTY  VALUE                SOURCE
> bukkit/home/rercola  xattr     off                  temporary
> # zfs set xattr=on bukkit/home/rercola
> # zfs get xattr bukkit/home/rercola
> NAME                 PROPERTY  VALUE                SOURCE
> bukkit/home/rercola  xattr     off                  temporary
> 
> What's going on? Wikipedia says, uncited, that FBSD ZFS has extended
> attributes. The FBSD wiki page on ZFS says that extattr is "done", but
> the description hasn't changed from before this was the case.
> I tried, naively:
> 
> # extattrctl start /bukkit/home/rercola/
> extattrctl start: Operation not supported
> 
> What's the correct thing to do here? Is it secretly unsupported and
> the documentation is imprecise? Or am I just an idiot?

I believe that only NFSv4-style ACLs are supported for ZFS.
AFAIK, "default ACL" is a feature of POSIX-style ACLs.
BTW, ACLs and extended attributes might be related things but they are not the same.
And your attempt to use extattrctl was very na?ve indeed, just open its manual page.
setfacl(1), lsextattr(1) are mandatory reading :-)

-- 
Andriy Gapon
From valin at buchlovice.org  Mon Oct 26 19:51:21 2009
From: valin at buchlovice.org (=?UTF-8?B?UmFkZWsgVmFsw6HFoWVr?=)
Date: Mon Oct 26 19:51:28 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
	unavailable"
In-Reply-To: <1256571299.2502.219.camel@balrog.2hip.net>
References: <op.u2dqyh1o4534sa@localhost>	<1256517106.2502.205.camel@balrog.2hip.net>	<op.u2ehpmzo4534sa@localhost>
	<1256571299.2502.219.camel@balrog.2hip.net>
Message-ID: <4AE5FDB4.20101@buchlovice.org>

Robert Noland napsal(a):
> On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
>   
>> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@freebsd.org>  
>> wrote:
>>     
>>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
>>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
>>>> kernel
>>>> booting from the ZFS fails. The initial installation I did just this, on
>>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
>>>> adX" on all disks before rebooting to see if that had any effect. The  
>>>> end
>>>> result is the same. After rebooting the machine I get the following
>>>> prompt(s):
>>>>
>>>>          ZFS: i/o error - all block copies unavailable
>>>>          Invalid format
>>>>
>>>>          FreeBSD/i386 boot
>>>>          Default: tank:/boot/kernel/kernel
>>>>          boot:
>>>>         
>>> Could you type "status" at this point and tell what it shows?
>>>       
>> If I type status at this point I get:
>>
>>    pool: tank
>>    config:
>>        NAME    STATE
>>        tank   ONLINE
>>      raidz1   ONLINE
>>       ad4p3   ONLINE
>>       ad6p3   ONLINE
>>       ad8p3   ONLINE
>>      ad10p3   ONLINE
>>
>> Which seems odd, since that's all the drives there are. So if it finds  
>> these it's already found all drives. My optimistic "Oh! I'll try and boot  
>> again" spirit was however crushed since it just results in the same error.
>>     
>
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
>
> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.
>  
> robert.
>   
So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001  err=00000000  efl=00000087 eip=0018d27d
eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
       fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
       00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status  pool: z
config:
NAME  STATE
z  ONLINE
raidz1  ONLINE
ad6p2  ONLINE
ad8p2  ONLINE
ad10p2  ONLINE
ad12p2  ONLINE

radek.

>  
>   
>> Kind regards,
>> Merijn
>> _______________________________________________
>> freebsd-current@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
>>     

From arno at heho.snv.jussieu.fr  Tue Oct 27 00:01:31 2009
From: arno at heho.snv.jussieu.fr (Arno J. Klaassen)
Date: Tue Oct 27 00:01:38 2009
Subject: ZFS and 'traditional' nfs-export
Message-ID: <wpbpjtwwg7.fsf@heho.snv.jussieu.fr>


Hello,

I googled a bit on this question but could not find
a clear answer :

is there any risk/inconvenience/advantage in exporting
a ZFS-fs by just putting it in /etc/exports the old
way and leaving the 'sharenfs' option on the filesystem off?

I'd like to replace a UFS-based server serving mostly
linux-clients which work well now with a ZFS-fs, and somehow
am a bit waterfearing changing the nfs?options which worked
great till now.

Thank you in advance,

regards, Arno
From valin at buchlovice.org  Tue Oct 27 10:15:24 2009
From: valin at buchlovice.org (=?UTF-8?B?UmFkZWsgVmFsw6HFoWVr?=)
Date: Tue Oct 27 10:15:36 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
	unavailable"
In-Reply-To: <1256571299.2502.219.camel@balrog.2hip.net>
References: <op.u2dqyh1o4534sa@localhost>	<1256517106.2502.205.camel@balrog.2hip.net>	<op.u2ehpmzo4534sa@localhost>
	<1256571299.2502.219.camel@balrog.2hip.net>
Message-ID: <4AE6C836.2070708@buchlovice.org>

Robert Noland napsal(a):
> On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
>   
>> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland <rnoland@freebsd.org>  
>> wrote:
>>     
>>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
>>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
>>>> kernel
>>>> booting from the ZFS fails. The initial installation I did just this, on
>>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
>>>> adX" on all disks before rebooting to see if that had any effect. The  
>>>> end
>>>> result is the same. After rebooting the machine I get the following
>>>> prompt(s):
>>>>
>>>>          ZFS: i/o error - all block copies unavailable
>>>>          Invalid format
>>>>
>>>>          FreeBSD/i386 boot
>>>>          Default: tank:/boot/kernel/kernel
>>>>          boot:
>>>>         
>>> Could you type "status" at this point and tell what it shows?
>>>       
>> If I type status at this point I get:
>>
>>    pool: tank
>>    config:
>>        NAME    STATE
>>        tank   ONLINE
>>      raidz1   ONLINE
>>       ad4p3   ONLINE
>>       ad6p3   ONLINE
>>       ad8p3   ONLINE
>>      ad10p3   ONLINE
>>
>> Which seems odd, since that's all the drives there are. So if it finds  
>> these it's already found all drives. My optimistic "Oh! I'll try and boot  
>> again" spirit was however crushed since it just results in the same error.
>>     
>
> Ok, that is both good and frustrating... I haven't produced any boot
> failures with all of the drives visible.  Do, note that I just added
> support for reading gang blocks to the loader. (basically untested,
> since I haven't managed to create them at will)  You will need to update
> your partition boot code for it to be supported during early boot.  i.e.
> gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
>
> The "all block copies unavailable" is a frustrating error, since all it
> means is a failed read, but we don't get a clue what failed or why.
> With the code that is in -CURRENT it will report gang blocks if found,
> even if it fails to read them.
>  
> robert.
>   
So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001  err=00000000  efl=00000087 eip=0018d27d
eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
       fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
       00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status  pool: z
config:
NAME  STATE
z  ONLINE
raidz1  ONLINE
ad6p2  ONLINE
ad8p2  ONLINE
ad10p2  ONLINE
ad12p2  ONLINE

radek.

>  
>   
>> Kind regards,
>> Merijn
>> _______________________________________________
>> freebsd-current@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
>>     


From ler at lerctr.org  Tue Oct 27 19:32:35 2009
From: ler at lerctr.org (Larry Rosenman)
Date: Tue Oct 27 19:32:47 2009
Subject: ZFS and 'traditional' nfs-export
In-Reply-To: <wpbpjtwwg7.fsf@heho.snv.jussieu.fr>
References: <wpbpjtwwg7.fsf@heho.snv.jussieu.fr>
Message-ID: <00b301ca5739$8e9b3850$abd1a8f0$@org>

ZFS makes its own version of the exports file.

Just do it that way, and be safe.

You can pass the full set of NFS options in the sharenfs parameter....



-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 512-248-2683                E-Mail: ler@lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893


-----Original Message-----
From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org]
On Behalf Of Arno J. Klaassen
Sent: Monday, October 26, 2009 6:46 PM
To: freebsd-fs@freebsd.org
Cc: freebsd-net@freebsd.org
Subject: ZFS and 'traditional' nfs-export


Hello,

I googled a bit on this question but could not find
a clear answer :

is there any risk/inconvenience/advantage in exporting
a ZFS-fs by just putting it in /etc/exports the old
way and leaving the 'sharenfs' option on the filesystem off?

I'd like to replace a UFS-based server serving mostly
linux-clients which work well now with a ZFS-fs, and somehow
am a bit waterfearing changing the nfs?options which worked
great till now.

Thank you in advance,

regards, Arno
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"

From patpro at patpro.net  Wed Oct 28 16:08:47 2009
From: patpro at patpro.net (Patrick Proniewski)
Date: Wed Oct 28 16:08:55 2009
Subject: ZFS licensing question
Message-ID: <1D0AE4B4-278A-4325-BFED-07B04BE316E1@patpro.net>

Hello,

Well, I understand this list is mainly interested in implementation  
and tuning, but I'm sure some of you are well informed about licensing  
issues.
Recently, Apple giving up on ZFS has made the headlines. *It appears*  
to be related to the NetApp vs. Sun case (<http://www.netapp.com/us/company/news/press-releases/news_rel_20070905.html 
 >). And *it appears* Sun tried to impose a license that would make  
Apple responsible for IP infringement in case Sun would lose against  
NetApp.

It makes me wonder about the FreeBSD port of ZFS. Do you have further  
informations about the licensing of ZFS technology, and about the case  
of FreeBSD port ?


regards,
patpro

From borjam at sarenet.es  Wed Oct 28 16:24:50 2009
From: borjam at sarenet.es (Borja Marcos)
Date: Wed Oct 28 16:24:58 2009
Subject: 8.0-RC2: ZFS deadlock with zfs receive
Message-ID: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es>

	
Hello,

I've been sending some alltraces to pjd about this easy to reproduce  
problem. I am using zfs send/zfs receive to replicate a dataset from  
one server to another. At 1 minute intervals, an incremental snapshot  
is sent to update the dataset copy. If there is reading activity on  
the dataset copy, a deadlock can happen rendering ZFS and all the FS  
subsystem unusable. I've tried with 8.0RC2 and it still happens.

How to reproduce:

For example, while doing a make buildworld, replicate the dataset  
containing /usr/obj to the second machine. On the second machine a  
script just runs an endless loop copying all the contents of /usr/obj  
(which, remember, are being copied from the first machine using zfs  
receive) to another location, thus generating quite a lot of read  
activity. The time to deadlock is random, but soon the deadlock appears.

The problem can be triggered by a backup accessing the destination  
dataset or when the security periodic scripts run at night.

I tried to mitigate the problem by cloning a given snapshot at the  
destination and doing the backup from it, but the problem persists.

Any ideas? Any way to mitigate this problem?




Borja.



This is the alltrace after breaking into ddb:

>> PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU
>> COMMAND
>> 1914 root        1  62    0 11932K  2564K zfsvfs  0   0:51  0.00%  
>> bsdtar
>> 1093 borjam      1  44    0  8304K  2464K CPU1    1   0:32  0.00% top
>> 1913 root        1  54    0 11932K  2600K rrl->r  0   0:19  0.00%  
>> bsdtar
>> 1019 root        1  44    0 25108K  4812K select  0   0:05  0.00%  
>> sshd
>> 2008 root        1  76    0 13600K  1904K tx->tx  0   0:04  0.00% zfs
>> 1089 borjam      1  44    0 37040K  5216K select  1   0:04  0.00%  
>> sshd
>>  995 root        1  76    0  8252K  2652K pause   0   0:02  0.00% csh
>>  840 root        1  44    0 11044K  3828K select  1   0:02  0.00%


debug.ddb.capture.data: db> alltrace


Tracing command sysctl pid 1373 tid 100183 td 0xffffff00177e5ab0
kdb_enter() at kdb_enter+0x3d
kdb_sysctl_enter() at kdb_sysctl_enter+0x89
sysctl_root() at sysctl_root+0x113
userland_sysctl() at userland_sysctl+0x158
__sysctl() at __sysctl+0xaa
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x80073d9dc, rsp =  
0x7fffffffe2a8, rbp = 0x7fffffffe37c ---

Tracing command csh pid 1369 tid 100181 td 0xffffff00177e6390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe638, rbp = 0x800c59400 ---

Tracing command su pid 1368 tid 100159 td 0xffffff001284aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x559 ---

Tracing command unlink pid 1363 tid 100162 td 0xffffff001284a000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1361 tid 100100 td 0xffffff000270bab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x551 ---

Tracing command cron pid 1360 tid 100122 td 0xffffff0002af0390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1355 tid 100127 td 0xffffff0002aea000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1353 tid 100171 td 0xffffff00177e8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x549 ---

Tracing command cron pid 1352 tid 100155 td 0xffffff001284d000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1347 tid 100157 td 0xffffff001284c720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1345 tid 100108 td 0xffffff0002916ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x541 ---

Tracing command cron pid 1344 tid 100101 td 0xffffff000270b720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1337 tid 100172 td 0xffffff00177e8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1334 tid 100098 td 0xffffff000270e390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x536 ---

Tracing command newsyslog pid 1333 tid 100065 td 0xffffff00026d6ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
ffs_vgetf() at ffs_vgetf+0xdf
ufs_lookup_() at ufs_lookup_+0xb46
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
stat() at stat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x80073251c, rsp =  
0x7fffffffc2b8, rbp = 0x8009075e0 ---

Tracing command cron pid 1331 tid 100147 td 0xffffff0002cb9000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command cron pid 1330 tid 100178 td 0xffffff00177e7000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1329 tid 100177 td 0xffffff00177e7390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1326 tid 100176 td 0xffffff00177e7720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x52e ---

Tracing command cron pid 1324 tid 100115 td 0xffffff0002915000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1319 tid 100059 td 0xffffff000256a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1317 tid 100058 td 0xffffff0002650000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x525 ---

Tracing command cron pid 1316 tid 100055 td 0xffffff0002650ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1309 tid 100112 td 0xffffff0002915ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1307 tid 100167 td 0xffffff0002cb8000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x51b ---

Tracing command cron pid 1306 tid 100166 td 0xffffff0002af0720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command unlink pid 1301 tid 100164 td 0xffffff00026e7ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
ffs_lock() at ffs_lock+0x9c
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
lookup() at lookup+0x5f0
namei() at namei+0x4a9
kern_unlinkat() at kern_unlinkat+0xab
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (10, FreeBSD ELF64, unlink), rip = 0x80072be5c, rsp =  
0x7fffffffec68, rbp = 0 ---

Tracing command sh pid 1299 tid 100103 td 0xffffff000270a000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800935edc, rsp =  
0x7fffffffe618, rbp = 0x513 ---

Tracing command cron pid 1298 tid 100160 td 0xffffff001284a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command sh pid 1289 tid 100161 td 0xffffff001284a390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
ffs_vgetf() at ffs_vgetf+0xdf
ufs_lookup_() at ufs_lookup_+0xb46
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
stat() at stat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (188, FreeBSD ELF64, stat), rip = 0x8009ae51c, rsp =  
0x7fffffffe798, rbp = 0x800c06b80 ---

Tracing command cron pid 1288 tid 100063 td 0xffffff000264f720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80095a7ac, rsp =  
0x7fffffffe4b8, rbp = 0x800a9e700 ---

Tracing command zfs pid 1285 tid 100143 td 0xffffff0002cbb000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
txg_wait_synced() at txg_wait_synced+0x7c
zfsvfs_teardown() at zfsvfs_teardown+0x1b2
zfs_suspend_fs() at zfs_suspend_fs+0x2b
zfs_ioc_recv() at zfs_ioc_recv+0x28b
zfsdev_ioctl() at zfsdev_ioctl+0x8d
devfs_ioctl_f() at devfs_ioctl_f+0x76
kern_ioctl() at kern_ioctl+0xc5
ioctl() at ioctl+0xfd
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe874c, rsp =  
0x7fffffffbbf8, rbp = 0x7fffffffc930 ---

Tracing command csh pid 1283 tid 100163 td 0xffffff0009979ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe598, rbp = 0x800c1d300 ---

Tracing command sshd pid 1281 tid 100124 td 0xffffff0002aeaab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command zpool pid 1256 tid 100109 td 0xffffff0002916720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x23f
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8010ce92c, rsp =  
0x7fffffffaac8, rbp = 0 ---

Tracing command csh pid 1252 tid 100158 td 0xffffff001284c000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe638, rbp = 0x800c59400 ---

Tracing command su pid 1248 tid 100156 td 0xffffff001284cab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x4e4 ---

Tracing command csh pid 1243 tid 100137 td 0xffffff0002917720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f700 ---

Tracing command sshd pid 1242 tid 100154 td 0xffffff001284d390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 1234 tid 100060 td 0xffffff000264fab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command bsdtar pid 1203 tid 100153 td 0xffffff000270aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1202 tid 100057 td 0xffffff0002650390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1200 tid 100152 td 0xffffff000270a720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1199 tid 100151 td 0xffffff0002af0ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1198 tid 100105 td 0xffffff000264c000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
rrw_enter() at rrw_enter+0xb1
zfs_freebsd_read() at zfs_freebsd_read+0x55
VOP_READ_APV() at VOP_READ_APV+0xaf
vn_read() at vn_read+0x254
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe898, rbp = 0x1a87 ---

Tracing command csh pid 1196 tid 100056 td 0xffffff0002650720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1195 tid 100067 td 0xffffff00026d6720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_sx_slock_hard() at _sx_slock_hard+0x1b7
_sx_slock() at _sx_slock+0xc1
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5
vgonel() at vgonel+0x119
vnlru_free() at vnlru_free+0x345
getnewvnode() at getnewvnode+0x24f
zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x2c
zfs_znode_alloc() at zfs_znode_alloc+0x39
zfs_mknode() at zfs_mknode+0x205
zfs_freebsd_create() at zfs_freebsd_create+0x583
VOP_CREATE_APV() at VOP_CREATE_APV+0xb3
vn_open_cred() at vn_open_cred+0x473
kern_openat() at kern_openat+0x179
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (5, FreeBSD ELF64, open), rip = 0x800e3557c, rsp =  
0x7fffffffe758, rbp = 0x1a4 ---

Tracing command bsdtar pid 1194 tid 100150 td 0xffffff0002cb8390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1192 tid 100138 td 0xffffff0002cbc390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1191 tid 100113 td 0xffffff0002915720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1190 tid 100118 td 0xffffff0002914390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1188 tid 100144 td 0xffffff0002cb9ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1187 tid 100139 td 0xffffff0002cbc000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
pipe_read() at pipe_read+0x4a3
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e447ac, rsp =  
0x7fffffffe7a8, rbp = 0x80101a240 ---

Tracing command bsdtar pid 1186 tid 100123 td 0xffffff0002af0000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1184 tid 100146 td 0xffffff0002cb9390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command bsdtar pid 1183 tid 100126 td 0xffffff0002aea390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x7b2
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vget() at vget+0x7b
cache_lookup() at cache_lookup+0x4e0
vfs_cache_lookup() at vfs_cache_lookup+0xc0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x2eb
namei() at namei+0x4a9
kern_statat_vnhook() at kern_statat_vnhook+0x8f
kern_statat() at kern_statat+0x15
lstat() at lstat+0x2a
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (190, FreeBSD ELF64, lstat), rip = 0x800e354fc, rsp =  
0x7fffffffe778, rbp = 0x801023380 ---

Tracing command bsdtar pid 1182 tid 100066 td 0xffffff000264f000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
__lockmgr_args() at __lockmgr_args+0x79b
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x57
vacl_get_acl() at vacl_get_acl+0x55
__acl_get_file() at __acl_get_file+0xcf
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (347, FreeBSD ELF64, __acl_get_file), rip = 0x800dbde0c,  
rsp = 0x7fffffffe1d8, rbp = 0x2 ---

Tracing command csh pid 1180 tid 100120 td 0xffffff0002714ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe3a8, rbp = 0x800c43200 ---

Tracing command csh pid 1147 tid 100142 td 0xffffff0002cbb390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp =  
0x7fffffffe658, rbp = 0x1 ---

Tracing command sshd pid 1146 tid 100111 td 0xffffff0002916000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 1133 tid 100116 td 0xffffff0002914ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command csh pid 959 tid 100149 td 0xffffff0002cb8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8009ec7ac, rsp =  
0x7fffffffe738, rbp = 0x1 ---

Tracing command su pid 958 tid 100148 td 0xffffff0002cb8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_wait() at kern_wait+0x3f7
wait4() at wait4+0x35
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e8edc, rsp =  
0x7fffffffe598, rbp = 0x3bf ---

Tracing command csh pid 952 tid 100125 td 0xffffff0002aea720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f500 ---

Tracing command sshd pid 951 tid 100130 td 0xffffff0002ae8390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command csh pid 946 tid 100141 td 0xffffff0002cbb720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094e75c, rsp =  
0x7fffffffe558, rbp = 0x800c1f500 ---

Tracing command sshd pid 945 tid 100140 td 0xffffff0002cbbab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffdcb8, rbp = 0x7fffffffdd40 ---

Tracing command sshd pid 942 tid 100128 td 0xffffff0002ae8ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command sshd pid 939 tid 100132 td 0xffffff0002ae7ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
soreceive_generic() at soreceive_generic+0xf99
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8013d77ac, rsp =  
0x7fffffffdcd8, rbp = 0 ---

Tracing command getty pid 908 tid 100117 td 0xffffff0002914720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 907 tid 100129 td 0xffffff0002ae8720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 906 tid 100106 td 0xffffff0002917390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 905 tid 100121 td 0xffffff0002714720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 904 tid 100136 td 0xffffff0002917ab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 903 tid 100135 td 0xffffff0002ae7000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 902 tid 100134 td 0xffffff0002ae7390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 901 tid 100133 td 0xffffff0002ae7720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
tty_wait() at tty_wait+0x48
ttydisc_read() at ttydisc_read+0x2f1
ttydev_read() at ttydev_read+0xab
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084f7ac, rsp =  
0x7fffffffecc8, rbp = 0 ---

Tracing command cron pid 853 tid 100119 td 0xffffff0002914000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x23f
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093d92c, rsp =  
0x7fffffffeb28, rbp = 0x3c ---

Tracing command sendmail pid 847 tid 100061 td 0xffffff000256a390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_sleep() at _sleep+0x309
kern_sigsuspend() at kern_sigsuspend+0xca
sigsuspend() at sigsuspend+0x34
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d3975c, rsp =  
0x7fffffffcd68, rbp = 0x1 ---

Tracing command sendmail pid 841 tid 100107 td 0xffffff0002917000
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x18c
seltdwait() at seltdwait+0x56
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800dd772c, rsp =  
0x7fffffffc1e8, rbp = 0x7fffffffc280 ---

Tracing command sshd pid 834 tid 100110 td 0xffffff0002916390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013d772c, rsp =  
0x7fffffffddd8, rbp = 0x2 ---

Tracing command syslogd pid 610 tid 100064 td 0xffffff000264f390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80085272c, rsp =  
0x7fffffffdcb8, rbp = 0x1 ---

Tracing command devd pid 503 tid 100114 td 0xffffff0002915390
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_catch_signals() at sleepq_catch_signals+0x2af
sleepq_wait_sig() at sleepq_wait_sig+0x16
_cv_wait_sig() at _cv_wait_sig+0x17e
seltdwait() at seltdwait+0xac
kern_select() at kern_select+0x620
select() at select+0x5d
syscall() at syscall+0x1d0
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x440cfc, rsp =  
0x7fffffffe898, rbp = 0x7fffffffe8b0 ---

Tracing command flowcleaner pid 22 tid 100054 td 0xffffff000256aab0
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_timedwait() at sleepq_timedwait+0x4d
_cv_timedwait() at _cv_timedwait+0x18c
flowtable_
debug.ddb.capture.bufsize: 49152
debug.ddb.capture.inprogress: 0
debug.ddb.capture.maxbufsize: 5242880
debug.ddb.capture.bufoff: 49152
debug.ddb.scripting.unscript:
debug.ddb.scripting.scripts:
debug.ddb.textdump.do_version: 1
debug.ddb.textdump.do_panic: 1
debug.ddb.textdump.do_msgbuf: 1
debug.ddb.textdump.do_ddb: 1
debug.ddb.textdump.pending: 0

From kris at FreeBSD.org  Wed Oct 28 16:30:33 2009
From: kris at FreeBSD.org (Kris Kennaway)
Date: Wed Oct 28 16:30:40 2009
Subject: ZFS licensing question
In-Reply-To: <1D0AE4B4-278A-4325-BFED-07B04BE316E1@patpro.net>
References: <1D0AE4B4-278A-4325-BFED-07B04BE316E1@patpro.net>
Message-ID: <4AE871B7.4000807@FreeBSD.org>

Patrick Proniewski wrote:
> Hello,
> 
> Well, I understand this list is mainly interested in implementation and 
> tuning, but I'm sure some of you are well informed about licensing issues.
> Recently, Apple giving up on ZFS has made the headlines. *It appears* to 
> be related to the NetApp vs. Sun case 
> (<http://www.netapp.com/us/company/news/press-releases/news_rel_20070905.html>). 
> And *it appears* Sun tried to impose a license that would make Apple 
> responsible for IP infringement in case Sun would lose against NetApp.
> 
> It makes me wonder about the FreeBSD port of ZFS. Do you have further 
> informations about the licensing of ZFS technology, and about the case 
> of FreeBSD port ?

I don't think anyone from FreeBSD has additional details to contribute 
about Apple's decision, and as far as we're aware nothing has changed 
with respect to ZFS licensing.

Kris
From jh at FreeBSD.org  Wed Oct 28 17:32:24 2009
From: jh at FreeBSD.org (jh@FreeBSD.org)
Date: Wed Oct 28 17:32:30 2009
Subject: usb/112640: [ext2fs] [hang] Kernel freezes when writing a file
	to an ex2fs filesystem on a usb disk
Message-ID: <200910281732.n9SHWMMe050016@freefall.freebsd.org>

Synopsis: [ext2fs] [hang] Kernel freezes when writing a file to an ex2fs filesystem on a usb disk

State-Changed-From-To: feedback->closed
State-Changed-By: jh
State-Changed-When: Wed Oct 28 17:28:55 UTC 2009
State-Changed-Why: 
Feedback timeout. Possibly fixed by r186740 / r187386.

http://www.freebsd.org/cgi/query-pr.cgi?pr=112640
From patpro at patpro.net  Wed Oct 28 17:45:27 2009
From: patpro at patpro.net (Patrick Proniewski)
Date: Wed Oct 28 17:45:33 2009
Subject: ZFS licensing question
In-Reply-To: <4AE871B7.4000807@FreeBSD.org>
References: <1D0AE4B4-278A-4325-BFED-07B04BE316E1@patpro.net>
	<4AE871B7.4000807@FreeBSD.org>
Message-ID: <573B5C38-C31D-47CB-8158-41B4B7DFEB6C@patpro.net>

On 28 oct. 2009, at 17:30, Kris Kennaway wrote:

> Patrick Proniewski wrote:
>> Hello,
>> Well, I understand this list is mainly interested in implementation  
>> and tuning, but I'm sure some of you are well informed about  
>> licensing issues.
>> Recently, Apple giving up on ZFS has made the headlines. *It  
>> appears* to be related to the NetApp vs. Sun case (<http://www.netapp.com/us/company/news/press-releases/news_rel_20070905.html 
>> >). And *it appears* Sun tried to impose a license that would make  
>> Apple responsible for IP infringement in case Sun would lose  
>> against NetApp.
>> It makes me wonder about the FreeBSD port of ZFS. Do you have  
>> further informations about the licensing of ZFS technology, and  
>> about the case of FreeBSD port ?
>
> I don't think anyone from FreeBSD has additional details to  
> contribute about Apple's decision, and as far as we're aware nothing  
> has changed with respect to ZFS licensing.

Well, I think some people are working for Apple and FreeBSD ;)
May be I should have made my point clearer. My concern is: do you  
think this kind of problem can impact FreeBSD too. Unfortunately, I'm  
not familiar with licenses subtleties and I'm not a lawyer...

regards,
patpro
From pjd at FreeBSD.org  Wed Oct 28 17:54:43 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Wed Oct 28 17:55:22 2009
Subject: ZFS and 'traditional' nfs-export
In-Reply-To: <wpbpjtwwg7.fsf@heho.snv.jussieu.fr>
References: <wpbpjtwwg7.fsf@heho.snv.jussieu.fr>
Message-ID: <20091028175433.GA1671@garage.freebsd.pl>

On Tue, Oct 27, 2009 at 12:46:00AM +0100, Arno J. Klaassen wrote:
> 
> Hello,
> 
> I googled a bit on this question but could not find
> a clear answer :
> 
> is there any risk/inconvenience/advantage in exporting
> a ZFS-fs by just putting it in /etc/exports the old
> way and leaving the 'sharenfs' option on the filesystem off?
> 
> I'd like to replace a UFS-based server serving mostly
> linux-clients which work well now with a ZFS-fs, and somehow
> am a bit waterfearing changing the nfs?options which worked
> great till now.

You can safely use /etc/exports without touching sharenfs property.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091028/091db385/attachment.pgp
From matt at corp.spry.com  Wed Oct 28 18:43:01 2009
From: matt at corp.spry.com (Matt Simerson)
Date: Wed Oct 28 18:43:08 2009
Subject: ZFS licensing question
In-Reply-To: <4AE871B7.4000807@FreeBSD.org>
References: <1D0AE4B4-278A-4325-BFED-07B04BE316E1@patpro.net>
	<4AE871B7.4000807@FreeBSD.org>
Message-ID: <168A0140-B91E-4F9E-8088-A69A0E17C56A@spry.com>


There is far more too it than has been made public, and all the  
conjecture in the world isn't likely to clear it up for us.

If you post a link to one side of the story, you should at least post  
a link to the other side. Here's a couple updates from Sun's legal  
counsel.

   Oct 2008: http://www.sun.com/lawsuit/zfs/
   Jun 2008: http://blogs.sun.com/dillon/entry/netapp_draft
   Oct 2008: http://blogs.sun.com/dillon/entry/more_on_the_netapp_litigation
   Oct 2008: http://blogs.netapp.com/dave/2008/10/current-status.html

It appears that Sun is besting NetApp. Sun has gotten numerous NetApp  
patent claims invalidated, including all claims for one of NetApp's  
self described 'core' patents.  Sun has also had a few of its patent  
claims dismissed, but they have a larger arsenal which which they are  
countersuing. If I were to put some skin in the game, I'd put money on  
Sun.

But what about Apple?

Apple needs a next generation file system, and they need it last year.  
Or 2-3 years ago. Hence Apple's interest in ZFS. But the case is  
already 2 years old and Sun is filing motions to stay the suit for  
even longer, giving the US PTO office time to review and  invalidate  
even more patent claims. If Apple committed to and deployed ZFS, they  
could easily be on the hook for many millions of dollars in licensing  
fees down the road. It could end up costing them far more than  
building a next generation FS for Mac OS X in house.

And what about FreeBSD?

This is what MLP had to say at http://blogs.sun.com/jonathan/entry/harvesting_from_a_troll

> First, the basics. Sun indemnifies all its customers against IP  
> claims like this. That is, we've always protected our markets from  
> trolls, so customers can continue to use ZFS without concern for  
> spurious patent and copyright issues. We stand behind our  
> innovation, and our customers.

Matt

On Oct 28, 2009, at 9:30 AM, Kris Kennaway wrote:

> Patrick Proniewski wrote:
>> Hello,
>> Well, I understand this list is mainly interested in implementation  
>> and tuning, but I'm sure some of you are well informed about  
>> licensing issues.
>> Recently, Apple giving up on ZFS has made the headlines. *It  
>> appears* to be related to the NetApp vs. Sun case (<http://www.netapp.com/us/company/news/press-releases/news_rel_20070905.html 
>> >). And *it appears* Sun tried to impose a license that would make  
>> Apple responsible for IP infringement in case Sun would lose  
>> against NetApp.
>> It makes me wonder about the FreeBSD port of ZFS. Do you have  
>> further informations about the licensing of ZFS technology, and  
>> about the case of FreeBSD port ?
>
> I don't think anyone from FreeBSD has additional details to  
> contribute about Apple's decision, and as far as we're aware nothing  
> has changed with respect to ZFS licensing.
>
> Kris
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"

From phk at phk.freebsd.dk  Wed Oct 28 19:15:37 2009
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed Oct 28 19:15:44 2009
Subject: ZFS licensing question 
In-Reply-To: Your message of "Wed, 28 Oct 2009 11:42:51 MST."
	<168A0140-B91E-4F9E-8088-A69A0E17C56A@spry.com> 
Message-ID: <3171.1256756201@critter.freebsd.dk>

In message <168A0140-B91E-4F9E-8088-A69A0E17C56A@spry.com>, Matt Simerson write
s:

>> First, the basics. Sun indemnifies all its customers against IP  
>> claims like this. That is, we've always protected our markets [...]

The quality of the protection hinges on conditions:

	1. Sun has to exist and stand by its word.

	2. Sun must have enough money to do so (see 1.)

None of these have 1.0 probability, but what the actual probability
is, is anyones guess, and you will have to make up your own mind.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
From james-freebsd-fs2 at jrv.org  Wed Oct 28 20:24:37 2009
From: james-freebsd-fs2 at jrv.org (James R. Van Artsdalen)
Date: Wed Oct 28 20:24:44 2009
Subject: ZFS licensing question
In-Reply-To: <3171.1256756201@critter.freebsd.dk>
References: <3171.1256756201@critter.freebsd.dk>
Message-ID: <4AE8A881.7050203@jrv.org>

There are lies, damn lies, and Corporate Press Releases.

Poul-Henning Kamp wrote:
> The quality of the protection hinges on conditions:
>
> 	1. Sun has to exist and stand by its word.
>
> 	2. Sun must have enough money to do so (see 1.)

phk is exactly right; in my experience it has always been easier to get
a billion dollar's worth of guarantees from a vendor than to get a
million dollar's worth.

Matt Simerson wrote:
> Apple needs a next generation file system, and they need it last year.

No they don't, not unless there was something in last quarter's results
I missed.

The benefit to FreeBSD is clear: ZFS, coupled with the SIIS driver and
SATA port multipliers, will make it possible to build cheap storage
arrays in the 50 TB and above range using FreeBSD.  The other things are
k00l but it's the huge cost savings I'll pitch to clients once the SIIS
driver is finished.

PS. Even when patent litigation is settled it isn't easy to figure out
who "won" until the 10-K's can be analyzed: post-settlement statements
by everyone are often part of the agreement.  That this has something of
a Motorola/Hitachi feel to it is as far as I'd speculate.

From kickbsd at ya.ru  Wed Oct 28 20:49:59 2009
From: kickbsd at ya.ru (kickbsd kickbsd)
Date: Wed Oct 28 20:50:32 2009
Subject: GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies
	unavailable"
In-Reply-To: <4AE6C836.2070708@buchlovice.org>
References: <op.u2dqyh1o4534sa@localhost>
	<1256517106.2502.205.camel@balrog.2hip.net>
	<op.u2ehpmzo4534sa@localhost>
	<1256571299.2502.219.camel@balrog.2hip.net>
	<4AE6C836.2070708@buchlovice.org>
Message-ID: <54151256762996@webmail84.yandex.ru>

Hi!

I've just tried to follow 
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ2
with RC2
and got the same error as below.

27.10.09, 11:15, "Radek Val??ek" <valin@buchlovice.org>:

> Robert Noland napsal(a):
> > On Mon, 2009-10-26 at 10:23 +0100, Merijn Verstraaten wrote:
> >   
> >> On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland   
> >> wrote:
> >>     
> >>>> After installing 8.0-RC1 (amd64) from USB stick this installation works
> >>>> fine. If I csup to RELENG_8 (amd64) and compile + install world and  
> >>>> kernel
> >>>> booting from the ZFS fails. The initial installation I did just this, on
> >>>> another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
> >>>> adX" on all disks before rebooting to see if that had any effect. The  
> >>>> end
> >>>> result is the same. After rebooting the machine I get the following
> >>>> prompt(s):
> >>>>
> >>>>          ZFS: i/o error - all block copies unavailable
> >>>>          Invalid format
> >>>>
> >>>>          FreeBSD/i386 boot
> >>>>          Default: tank:/boot/kernel/kernel
> >>>>          boot:
> >>>>         
> >>> Could you type "status" at this point and tell what it shows?
> >>>       
> >> If I type status at this point I get:
> >>
> >>    pool: tank
> >>    config:
> >>        NAME    STATE
> >>        tank   ONLINE
> >>      raidz1   ONLINE
> >>       ad4p3   ONLINE
> >>       ad6p3   ONLINE
> >>       ad8p3   ONLINE
> >>      ad10p3   ONLINE
> >>
> >> Which seems odd, since that's all the drives there are. So if it finds  
> >> these it's already found all drives. My optimistic "Oh! I'll try and boot  
> >> again" spirit was however crushed since it just results in the same error.
> >>     
> >
> > Ok, that is both good and frustrating... I haven't produced any boot
> > failures with all of the drives visible.  Do, note that I just added
> > support for reading gang blocks to the loader. (basically untested,
> > since I haven't managed to create them at will)  You will need to update
> > your partition boot code for it to be supported during early boot.  i.e.
> > gpart bootcode -p /boot/gptzfsboot -i  
> >
> > The "all block copies unavailable" is a frustrating error, since all it
> > means is a failed read, but we don't get a clue what failed or why.
> > With the code that is in -CURRENT it will report gang blocks if found,
> > even if it fails to read them.
> >  
> > robert.
> >   
> So I switched to -CURRENT:
> 1, overwriting /boot/loader.conf results with:
> BTX loader 1.00 BTX version is 1.02
> Consoles: internal video/keyboard
> BIOS drive C: is disk0
> BIOS drive D: is disk1
> BIOS drive E: is disk2
> BIOS drive F: is disk3
> BIOS 627kB/3405248kB available memory
> FreeBSD/i386 bootstrap loader, Revision 1.1
> (root@ztest, Mon Oct 26 14:01:44 CEST 2009)
> Loading /boot/defaults/loader.conf
> ZFS: i/o error - all block copies unavailable
> Warning: error reading file /boot/loader.conf
> so basically the same as in RELENG_8
> 2, + overwriting /boot/loader results with:
> ZFS: i/o error - all block copies unavailable
> Invalid format
> FreeBSD/i386 boot
> Default: z:/boot/kernel/kernel
> boot:
> \
> int=00000001  err=00000000  efl=00000087 eip=0018d27d
> eax=0018d2af  ebx=18bf9925  ecx=540d8ef2  edx=00000000
> esi=00009401  edi=000919d0  ebp=36571125  esp=80000000
> cs=0008  ds=0010  es=0010  fs=0010  gs=0010  ss=0010
> cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
>        fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
> ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
>        00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
> BTX halted
> 3, I also try the 'status' as you told to Merijn before BTX halted:
> ZFS: i/o error - all block copies unavailable
> Invalid format
> FreeBSD/i386 boot
> Default: z:/boot/kernel/kernel
> boot: status  pool: z
> config:
> NAME  STATE
> z  ONLINE
> raidz1  ONLINE
> ad6p2  ONLINE
> ad8p2  ONLINE
> ad10p2  ONLINE
> ad12p2  ONLINE
> radek.
> >  
> >   
> >> Kind regards,
> >> Merijn
> >> _______________________________________________
> >> freebsd-current@freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
> >>     
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"

-- 
?????????? ????? ????????? ?????: http://mail.yandex.ru/promo/new/search
From ronald-freebsd8 at klop.yi.org  Wed Oct 28 21:04:34 2009
From: ronald-freebsd8 at klop.yi.org (Ronald Klop)
Date: Wed Oct 28 21:04:46 2009
Subject: Fwd: Re: zfs receive gives: internal error: Argument list too long
In-Reply-To: <op.u2dgfwap8527sy@82-170-177-25.ip.telfort.nl>
References: <op.u2df9msz8527sy@82-170-177-25.ip.telfort.nl>
	<op.u2dgfwap8527sy@82-170-177-25.ip.telfort.nl>
Message-ID: <op.u2i2wknf8527sy@82-170-177-25.ip.telfort.nl>

Hi,

I'm forwarding this, because there was no answer on freebsd-stable.

Does anybody know about this and have some tips on how to solve it?

Ronald.

------- Forwarded message -------
From: "Ronald Klop" <ronald-freebsd8@klop.yi.org>
To: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>
Cc:
Subject: Re: zfs receive gives: internal error: Argument list too long
Date: Sun, 25 Oct 2009 20:58:34 +0100

On Sun, 25 Oct 2009 20:54:48 +0100, Ronald Klop
<ronald-freebsd8@klop.yi.org> wrote:

> Hi,
>
> Making a backup to my external USB-disk now fails with the following  
> output.
>
> [root@sjakie ~]# zfs send -vi tank/home@repl-20090919_195900  
> tank/home@repl-20090921_195900 > /bla.snapshot
>
> # ls -lh /bla*
> -rw-r--r--  1 root  wheel   547M Oct 25 20:44 /bla.snapshot
>
> [root@sjakie ~]# zfs receive -F extern/home < /bla.snapshot
> internal error: Argument list too long
> Abort trap: 6 (core dumped)
>
> # uname -a
> FreeBSD sjakie.klop.ws 8.0-RC1 FreeBSD 8.0-RC1 #6: Wed Oct 21 00:57:07  
> CEST 2009     root@sjakie.klop.ws:/usr/obj/usr/src/sys/GENERIC  amd64
>
> I have two pools 'tank' and 'extern'. I send/recieve several volumes  
> from tank to extern.
> zpool is version 13 and zfs is version 3
>
> Searching the internet I found this link:
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6573681

I meant: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6801979

> Is this a known bug for freebsd also?
> Can anybody help me in continuing my snapshotting process? (Except  
> making a new fresh snapshot, which is my last option.)
>
> Ronald.
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

From linimon at FreeBSD.org  Thu Oct 29 07:08:55 2009
From: linimon at FreeBSD.org (linimon@FreeBSD.org)
Date: Thu Oct 29 07:09:02 2009
Subject: kern/140068: [smbfs] [patch] smbfs does not allow semicolon in
	filenames, but should
Message-ID: <200910290708.n9T78sVE053473@freefall.freebsd.org>

Old Synopsis: [patch] smbfs does not allow semicolon in filenames, but should
New Synopsis: [smbfs] [patch] smbfs does not allow semicolon in filenames, but should

Responsible-Changed-From-To: freebsd-bugs->freebsd-fs
Responsible-Changed-By: linimon
Responsible-Changed-When: Thu Oct 29 07:08:38 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=140068
From pluknet at gmail.com  Thu Oct 29 11:39:13 2009
From: pluknet at gmail.com (pluknet)
Date: Thu Oct 29 11:39:19 2009
Subject: ZFS licensing question
In-Reply-To: <4AE8A881.7050203@jrv.org>
References: <3171.1256756201@critter.freebsd.dk> <4AE8A881.7050203@jrv.org>
Message-ID: <a31046fc0910290439r66a6f050sacf53c25e06954e3@mail.gmail.com>

2009/10/28 James R. Van Artsdalen <james-freebsd-fs2@jrv.org>:
> There are lies, damn lies, and Corporate Press Releases.
>
> Poul-Henning Kamp wrote:
>> The quality of the protection hinges on conditions:
>>
>> ? ? ? 1. Sun has to exist and stand by its word.
>>
>> ? ? ? 2. Sun must have enough money to do so (see 1.)
>
> phk is exactly right; in my experience it has always been easier to get
> a billion dollar's worth of guarantees from a vendor than to get a
> million dollar's worth.
>
> Matt Simerson wrote:
>> Apple needs a next generation file system, and they need it last year.
>
> No they don't, not unless there was something in last quarter's results
> I missed.

At least Apple is looking for a file system engineer.
http://jobs.apple.com/index.ajs?method=mExternal.showJob&RID=42559


-- 
wbr,
pluknet
From patpro at patpro.net  Thu Oct 29 16:04:19 2009
From: patpro at patpro.net (Patrick Proniewski)
Date: Thu Oct 29 16:04:25 2009
Subject: ZFS licensing question
In-Reply-To: <4AE871B7.4000807@FreeBSD.org>
References: <1D0AE4B4-278A-4325-BFED-07B04BE316E1@patpro.net>
	<4AE871B7.4000807@FreeBSD.org>
Message-ID: <126A246B-9FC3-4B12-BC5C-FA1871A8B256@patpro.net>

Hi,

Thanks all for your replies (I'm reading the daily digest).

> If you post a link to one side of the story, you should at least post
> a link to the other side. Here's a couple updates from Sun's legal
> counsel.

Thanks for the links. I only mean to provide fe bits of background,  
but more info is always welcome.


regards,
patpro
From jh at FreeBSD.org  Thu Oct 29 17:10:03 2009
From: jh at FreeBSD.org (Jaakko Heinonen)
Date: Thu Oct 29 17:10:09 2009
Subject: kern/133614: panic: ffs_truncate: read-only filesystem
Message-ID: <200910291710.n9THA3lC003871@freefall.freebsd.org>

The following reply was made to PR kern/133614; it has been noted by GNATS.

From: Jaakko Heinonen <jh@FreeBSD.org>
To: "Mikhail T." <mi@aldan.algebra.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/133614: panic: ffs_truncate: read-only filesystem
Date: Thu, 29 Oct 2009 19:02:00 +0200

 The bug has been described in this message:
 
 http://docs.freebsd.org/cgi/mid.cgi?20090910091329.GA2726
 
 There is a workaround patch available in this message:
 
 http://docs.freebsd.org/cgi/mid.cgi?20090914173208.GA4273
From mattjreimer at gmail.com  Thu Oct 29 19:07:45 2009
From: mattjreimer at gmail.com (Matt Reimer)
Date: Thu Oct 29 19:07:51 2009
Subject: Bogus malloc in zfsboot.c?
Message-ID: <f383264b0910291207t24a922aaxf03032b68786c7f6@mail.gmail.com>

I'm trying to debug why I suddenly can't boot an amd64 machine off a
raidz2 pool, after using freebsd-update to go from -rc1 to rc2. I'm
getting an error, "ZFS: out of temporary buffer space."

Is zfsboot.c's malloc really correct in the way it sets up its heap?

    heap_next = (char *) dmadat + sizeof(*dmadat);
    heap_end = (char *) (640*1024);

If I'm reading the code correctly, it assumes that dmadat is the last
item in bss, and that it can use all the memory from the end of dmadat
to 640KB.

But dmadat is not the last item in bss, as zfsimpl.c gets included and
it defines its own variables that end up in bss, with the result that
malloc could overwrite ZFS variables.

Am I reading this correctly?

Matt
From pjd at FreeBSD.org  Thu Oct 29 20:51:31 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Thu Oct 29 20:51:37 2009
Subject: Fwd: Re: zfs receive gives: internal error: Argument list too
	long
In-Reply-To: <op.u2i2wknf8527sy@82-170-177-25.ip.telfort.nl>
References: <op.u2df9msz8527sy@82-170-177-25.ip.telfort.nl>
	<op.u2dgfwap8527sy@82-170-177-25.ip.telfort.nl>
	<op.u2i2wknf8527sy@82-170-177-25.ip.telfort.nl>
Message-ID: <20091029205121.GB3418@garage.freebsd.pl>

On Wed, Oct 28, 2009 at 09:51:46PM +0100, Ronald Klop wrote:
> Hi,
> 
> I'm forwarding this, because there was no answer on freebsd-stable.
> 
> Does anybody know about this and have some tips on how to solve it?

Could you try this patch:

	http://people.freebsd.org/~pjd/patches/zfs_recv_E2BIG.patch

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091029/652767d0/attachment.pgp
From stef-list at memberwebs.com  Thu Oct 29 21:52:54 2009
From: stef-list at memberwebs.com (Stef Walter)
Date: Thu Oct 29 21:53:04 2009
Subject: 8.0-RC2: ZFS deadlock with zfs receive
In-Reply-To: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es>
References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es>
Message-ID: <4AEA0EAD.1050302@memberwebs.com>

Borja Marcos wrote:
> I've been sending some alltraces to pjd about this easy to reproduce
> problem. I am using zfs send/zfs receive to replicate a dataset from one
> server to another. At 1 minute intervals, an incremental snapshot is
> sent to update the dataset copy. If there is reading activity on the
> dataset copy, a deadlock can happen rendering ZFS and all the FS
> subsystem unusable. I've tried with 8.0RC2 and it still happens.

FWIW, another (or the same) zfs recv deadlock I've been trying to get to
the bottom of:

http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html

Cheers,

Stef

From tom at uffner.com  Thu Oct 29 22:50:04 2009
From: tom at uffner.com (Tom Uffner)
Date: Thu Oct 29 22:50:10 2009
Subject: bin/117315: [smbfs] mount_smbfs(8) and related options can't
 mount administration mounts (c$, d$, etc) on Windows PCs
Message-ID: <200910292250.n9TMo3FW099644@freefall.freebsd.org>

The following reply was made to PR bin/117315; it has been noted by GNATS.

From: Tom Uffner <tom@uffner.com>
To: bug-followup@FreeBSD.org, gcooper@FreeBSD.org
Cc:  
Subject: Re: bin/117315: [smbfs] mount_smbfs(8) and related options can't
 mount administration mounts (c$, d$, etc) on Windows PCs
Date: Thu, 29 Oct 2009 18:17:24 -0400

 I have reproduced this recently, but it seems to be a change in the
 way Windows behaves and not a bug.
 
 Microsoft seems to have finally realized that the {drive}$ shares are
 a huge security hole and decided that only members of the Administrators
 group are allowed to mount them.
From mattjreimer at gmail.com  Thu Oct 29 23:12:01 2009
From: mattjreimer at gmail.com (Matt Reimer)
Date: Thu Oct 29 23:12:16 2009
Subject: Bogus malloc in zfsboot.c?
In-Reply-To: <f383264b0910291207t24a922aaxf03032b68786c7f6@mail.gmail.com>
References: <f383264b0910291207t24a922aaxf03032b68786c7f6@mail.gmail.com>
Message-ID: <f383264b0910291612q4442a1f9x196a34c346698412@mail.gmail.com>

On Thu, Oct 29, 2009 at 12:07 PM, Matt Reimer <mattjreimer@gmail.com> wrote:
> I'm trying to debug why I suddenly can't boot an amd64 machine off a
> raidz2 pool, after using freebsd-update to go from -rc1 to rc2. I'm
> getting an error, "ZFS: out of temporary buffer space."
>
> Is zfsboot.c's malloc really correct in the way it sets up its heap?
>
> ? ?heap_next = (char *) dmadat + sizeof(*dmadat);
> ? ?heap_end = (char *) (640*1024);
>
> If I'm reading the code correctly, it assumes that dmadat is the last
> item in bss, and that it can use all the memory from the end of dmadat
> to 640KB.
>
> But dmadat is not the last item in bss, as zfsimpl.c gets included and
> it defines its own variables that end up in bss, with the result that
> malloc could overwrite ZFS variables.
>
> Am I reading this correctly?

Probably not; I missed this:

    dmadat = (void *)(roundup2(__base + (int32_t)&_end, 0x10000) - __base);

Matt
From alexz at visp.ru  Fri Oct 30 07:06:16 2009
From: alexz at visp.ru (Alexander Zagrebin)
Date: Fri Oct 30 07:06:23 2009
Subject: ZFS statfs problem
Message-ID: <656C9ACD24DB41898D89F64DC7A2E3E1@vosz.local>

Hi!

I have noticed, that statfs called for ZFS file systems,
returns the value of fs's recordsize property in both f_bsize and f_iosize.
It's a problem for some software.
For example, squid uses block size of cache's file system to calculate the
space
occupied by file.
So it considers that any small file uses 128KB of a cache (when default
value of recordsize is used),
though really this file may use 512B only.
This miscalculation generates unjustified cleaning of a cache.
There are the some possible solutions:
- to set recordsize to lower value (for example, 2K), but it's not optimal
for file system.
- to force squid to use 512 as block size (via patch)
- to change the zfs code (zfs_vfsops.c) to return 512 as f_bsize and
recordsize as f_iosize
(see attached patch for 8.0-RC1)
I think that the last solution is more correct.
The patch seems to be working, but i'm not sure that it's correct.
Would you comment this?

-- 
Alexander Zagrebin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch-zfs_vfsops.c
Type: application/octet-stream
Size: 1122 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091030/1b5fd8bc/patch-zfs_vfsops.obj
From jh at FreeBSD.org  Fri Oct 30 19:59:28 2009
From: jh at FreeBSD.org (jh@FreeBSD.org)
Date: Fri Oct 30 19:59:34 2009
Subject: kern/131086: [ext2fs] [patch] mkfs.ext2 creates rotten partition
Message-ID: <200910301959.n9UJxSok036883@freefall.freebsd.org>

Synopsis: [ext2fs] [patch] mkfs.ext2 creates rotten partition

State-Changed-From-To: open->closed
State-Changed-By: jh
State-Changed-When: Fri Oct 30 19:55:35 UTC 2009
State-Changed-Why: 
Duplicate of ports/131556. The change has been committed to sysutils/e2fsprogs port.

http://www.freebsd.org/cgi/query-pr.cgi?pr=131086
From zbeeble at gmail.com  Fri Oct 30 22:00:21 2009
From: zbeeble at gmail.com (Zaphod Beeblebrox)
Date: Fri Oct 30 22:00:27 2009
Subject: Reading allegedly bad ZFS blocks
Message-ID: <5f67a8c40910301500t4b3d3434ha70cc32b66842795@mail.gmail.com>

I have a system I suspect of not being entirely sane.  It has accused some
files in my ZFS RAIDZ1 array of being corrupt --- ie: not enough information
to recover them.

However, I suspect strongly that the data may be OK and simply the system
itself (reading those blocks) was bad.

Now... I attach the ZFS array to a new system ... and the new system refuses
to check those files again --- they are "known" bad when I first import the
filesystem.

I would like the ability to read the underlying data again --- possibly
re-check the checksum ... or just get the whole file.

Current ZFS behaviour is to return EIO for any of the blocks it doesn't wish
to read.  Someone in the archives mentioned patching ZFS to ignore it's
checksums.  Strikes me this should be sysctl.

Could someone post that patch?
From hali at datapipe.net  Fri Oct 30 22:42:42 2009
From: hali at datapipe.net (Hussain Ali)
Date: Fri Oct 30 22:42:49 2009
Subject: Plans for Logged/Journaled UFS
Message-ID: <20091030223225.GI5120@datapipe.com>


Hello,

Was just wondering if any of the LOG or Journal based UFS implementations
are being worked on. I know of three that could make thier way into
FreeBSD :

 - bluffs :: IIRC ups@ was working on this
 - wabl :: Wasabi's FFS extentions for NetBSD's UFS2
 - solaris ufs :: that has journaling support - this is probably the
   most work though

ZFS doesnt suffice for may use cases - so just wondering if this is in
the works.

Thanks,

--
-hussain

This message may contain confidential or privileged information.  If you are not the intended recipient, please advise us immediately and delete this message.  See http://www.datapipe.com/emaildisclaimer.aspx for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.
From mj at feral.com  Fri Oct 30 22:49:43 2009
From: mj at feral.com (Matthew Jacob)
Date: Fri Oct 30 22:49:50 2009
Subject: Plans for Logged/Journaled UFS
In-Reply-To: <20091030223225.GI5120@datapipe.com>
References: <20091030223225.GI5120@datapipe.com>
Message-ID: <4AEB6D79.5070703@feral.com>

Hussain Ali wrote:
>
> ZFS doesnt suffice for may use cases - so just wondering if this is in
> the works.
>
>   
Which use cases can you name?

From pjd at FreeBSD.org  Fri Oct 30 23:28:14 2009
From: pjd at FreeBSD.org (Pawel Jakub Dawidek)
Date: Fri Oct 30 23:28:21 2009
Subject: 8.0-RC2: ZFS deadlock with zfs receive
In-Reply-To: <4AEA0EAD.1050302@memberwebs.com>
References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es>
	<4AEA0EAD.1050302@memberwebs.com>
Message-ID: <20091030232805.GA2996@garage.freebsd.pl>

On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
> Borja Marcos wrote:
> > I've been sending some alltraces to pjd about this easy to reproduce
> > problem. I am using zfs send/zfs receive to replicate a dataset from one
> > server to another. At 1 minute intervals, an incremental snapshot is
> > sent to update the dataset copy. If there is reading activity on the
> > dataset copy, a deadlock can happen rendering ZFS and all the FS
> > subsystem unusable. I've tried with 8.0RC2 and it still happens.

I was able to reproduce it, but I don't have fix yet.

> FWIW, another (or the same) zfs recv deadlock I've been trying to get to
> the bottom of:
> 
> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html

Could you guys recompile your kernel after uncommenting line:

#CFLAGS+=-DDEBUG=1

in sys/modules/zfs/Makefile?

You should see panic on assertion instead of deadlock, I believe.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd@FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20091030/e958d884/attachment.pgp
From pjd at FreeBSD.org  Fri Oct 30 23:34:23 2009
From: pjd at FreeBSD.org (pjd@FreeBSD.org)
Date: Fri Oct 30 23:34:29 2009
Subject: kern/139806: [zfs] [panic] Write attempt to file in ZFS
	snapshot dir causes panic
Message-ID: <200910302334.n9UNYMB5028217@freefall.freebsd.org>

Synopsis: [zfs] [panic] Write attempt to file in ZFS snapshot dir causes panic

State-Changed-From-To: open->patched
State-Changed-By: pjd
State-Changed-When: ptk 30 pa¼ 2009 23:33:39 UTC
State-Changed-Why: 
Fix committed to HEAD. Thank you for the report!


Responsible-Changed-From-To: freebsd-fs->pjd
Responsible-Changed-By: pjd
Responsible-Changed-When: ptk 30 pa¼ 2009 23:33:39 UTC
Responsible-Changed-Why: 
I'll take this one.

http://www.freebsd.org/cgi/query-pr.cgi?pr=139806
From stef-list at memberwebs.com  Fri Oct 30 23:50:44 2009
From: stef-list at memberwebs.com (Stef Walter)
Date: Fri Oct 30 23:50:57 2009
Subject: Plans for Logged/Journaled UFS
In-Reply-To: <4AEB6D79.5070703@feral.com>
References: <20091030223225.GI5120@datapipe.com> <4AEB6D79.5070703@feral.com>
Message-ID: <4AEB7BCC.5000804@memberwebs.com>

Matthew Jacob wrote:
> Hussain Ali wrote:
>>
>> ZFS doesnt suffice for may use cases - so just wondering if this is in
>> the works.
>>
>>   
> Which use cases can you name?

Low memory or 'embedded' devices (which could have spinning hard disks):

http://soekris.com/

http://www.pcengines.ch/alix.htm

Cheers,

Stef

From mj at feral.com  Sat Oct 31 01:22:25 2009
From: mj at feral.com (Matthew Jacob)
Date: Sat Oct 31 01:22:33 2009
Subject: Plans for Logged/Journaled UFS
In-Reply-To: <4AEB7BCC.5000804@memberwebs.com>
References: <20091030223225.GI5120@datapipe.com> <4AEB6D79.5070703@feral.com>
	<4AEB7BCC.5000804@memberwebs.com>
Message-ID: <4AEB9144.9050204@feral.com>


   Stef Walter wrote:

Matthew Jacob wrote:
  

Hussain Ali wrote:
    

ZFS doesnt suffice for may use cases - so just wondering if this is in
the works.

  
      

Which use cases can you name?
    

Low memory or 'embedded' devices (which could have spinning hard disks):

[1]http://soekris.com/

[2]http://www.pcengines.ch/alix.htm



   Any others?

References

   1. http://soekris.com/
   2. http://www.pcengines.ch/alix.htm
From stef-list at memberwebs.com  Sat Oct 31 01:29:52 2009
From: stef-list at memberwebs.com (Stef Walter)
Date: Sat Oct 31 01:31:13 2009
Subject: 8.0-RC2: ZFS deadlock with zfs receive
In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl>
References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es>	<4AEA0EAD.1050302@memberwebs.com>
	<20091030232805.GA2996@garage.freebsd.pl>
Message-ID: <4AEB9306.5070501@memberwebs.com>

Pawel Jakub Dawidek wrote:
> On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
>> Borja Marcos wrote:
>>> I've been sending some alltraces to pjd about this easy to reproduce
>>> problem. I am using zfs send/zfs receive to replicate a dataset from one
>>> server to another. At 1 minute intervals, an incremental snapshot is
>>> sent to update the dataset copy. If there is reading activity on the
>>> dataset copy, a deadlock can happen rendering ZFS and all the FS
>>> subsystem unusable. I've tried with 8.0RC2 and it still happens.
> 
> I was able to reproduce it, but I don't have fix yet.

Thanks for taking your time to look at it. Super appreciated.

>> FWIW, another (or the same) zfs recv deadlock I've been trying to get to
>> the bottom of:
>>
>> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html
> 
> Could you guys recompile your kernel after uncommenting line:
> 
> #CFLAGS+=-DDEBUG=1
> 
> in sys/modules/zfs/Makefile?
> 
> You should see panic on assertion instead of deadlock, I believe.

Rebuilt with that change... You're right, now I get this assertion:

login: panic: solaris assert: list_head(&osi->os_dnodes) ==
osi->os_meta_dnode (0xc9060488 == 0xca922000), file:
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c,
line: 459

Full details and backtraces attached.

Thanks again,

Stef
-------------- next part --------------
http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html

----------------------------------------------------------------------
 The Assertion

login: panic: solaris assert: list_head(&osi->os_dnodes) == osi->os_meta_dnode (0xc9060488 == 0xca922000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c, line: 459
cpuid = 0
KDB: enter: panic
[thread pid 1552 tid 100121 ]
Stopped at      kdb_enter+0x3a: movl    $0,kdb_why


db> bt
Tracing pid 1552 tid 100121 td 0xc658f480
kdb_enter(c0c79511,c0c79511,c629f3cf,e81794d8,0,...) at kdb_enter+0x3a
panic(c629f3cf,c62a3486,c9060488,0,c629f660,...) at panic+0x136
dmu_objset_evict(c7346200,c72b0200,1cb,0,c629cc75,...) at dmu_objset_evict+0x299
dsl_dataset_destroy(c7346200,c62a2ebd,16,e8179984,e8179944,...) at dsl_dataset_destroy+0x24c
dmu_recv_abort_cleanup(e8179984,c62ad24a,1cb,c629c778,e817992c,...) at dmu_recv_abort_cleanup+0x2f
dmu_recv_stream(e8179984,c5ed4690,e817999c,0,0,...) at dmu_recv_stream+0xa0e
zfs_ioc_recv(caab8000,0,0,0,0,...) at zfs_ioc_recv+0x2d4
zfsdev_ioctl(c5f67300,cc285a1c,caab8000,3,c658f480,...) at zfsdev_ioctl+0x7d
devfs_ioctl_f(c5ed4dc8,cc285a1c,caab8000,c981c280,c658f480,...) at devfs_ioctl_f+0xf8
kern_ioctl(c658f480,3,cc285a1c,caab8000,1000000,...) at kern_ioctl+0x1fd
ioctl(c658f480,e8179cf8,c,c0c916cc,c0d5ff48,...) at ioctl+0x134
syscall(e8179d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfbcfc, ebp = 0xbfbfbd28 ---


db> show pcpu
cpuid        = 0
dynamic pcpu    = 0x68ac00
curthread    = 0xc658f480: pid 1552 "zfs"
curpcb       = 0xe8179d90
fpcurthread  = none
idlethread   = 0xc596fb40: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:


db> show allpcpu
Current CPU: 0

cpuid        = 0
dynamic pcpu    = 0x68ac00
curthread    = 0xc658f480: pid 1552 "zfs"
curpcb       = 0xe8179d90
fpcurthread  = none
idlethread   = 0xc596fb40: pid 11 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:


db> show locks
exclusive sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7346294) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:745
shared sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7a29494) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:519
exclusive sx zfsvfs->z_online_recv_lock (zfsvfs->z_online_recv_lock) r = 0 (0xd2940198) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2484


db> show alllocks
Process 1552 (zfs) thread 0xc658f480 (100121)
exclusive sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7346294) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:745
shared sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7a29494) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:519
exclusive sx zfsvfs->z_online_recv_lock (zfsvfs->z_online_recv_lock) r = 0 (0xd2940198) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2484


db> show lockedvnods
Locked vnodes


db> alltrace

Tracing command zfs pid 1552 tid 100121 td 0xc658f480
kdb_enter(c0c79511,c0c79511,c629f3cf,e81794d8,0,...) at kdb_enter+0x3a
panic(c629f3cf,c62a3486,c9060488,0,c629f660,...) at panic+0x136
dmu_objset_evict(c7346200,c72b0200,1cb,0,c629cc75,...) at dmu_objset_evict+0x299
dsl_dataset_destroy(c7346200,c62a2ebd,16,e8179984,e8179944,...) at dsl_dataset_destroy+0x24c
dmu_recv_abort_cleanup(e8179984,c62ad24a,1cb,c629c778,e817992c,...) at dmu_recv_abort_cleanup+0x2f
dmu_recv_stream(e8179984,c5ed4690,e817999c,0,0,...) at dmu_recv_stream+0xa0e
zfs_ioc_recv(caab8000,0,0,0,0,...) at zfs_ioc_recv+0x2d4
zfsdev_ioctl(c5f67300,cc285a1c,caab8000,3,c658f480,...) at zfsdev_ioctl+0x7d
devfs_ioctl_f(c5ed4dc8,cc285a1c,caab8000,c981c280,c658f480,...) at devfs_ioctl_f+0xf8
kern_ioctl(c658f480,3,cc285a1c,caab8000,1000000,...) at kern_ioctl+0x1fd
ioctl(c658f480,e8179cf8,c,c0c916cc,c0d5ff48,...) at ioctl+0x134
syscall(e8179d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfbcfc, ebp = 0xbfbfbd28 ---

Tracing command zfskern pid 1527 tid 100117 td 0xc61226c0
sched_switch(c61226c0,0,104,191,b9d041d4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c61226c0,0,c0c7d8d5,26e,c61226c0,...) at sleepq_switch+0x15f
sleepq_timedwait(c669addc,0,e8166c08,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c669addc,c669ad84,7530,1,c669addc,...) at _cv_timedwait+0x250
txg_thread_wait(7530,0,3b9aca00,0,202c7066,...) at txg_thread_wait+0xab
txg_sync_thread(c669ac00,e8166d38,c0c749df,343,c611e000,...) at txg_sync_thread+0x480
fork_exit(c61fd6c0,c669ac00,e8166d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8166d70, ebp = 0 ---

Tracing command zfskern pid 1527 tid 100116 td 0xc6122900
sched_switch(c6122900,0,104,191,b9d06478,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122900,0,c0c7d8d5,24b,c6122900,...) at sleepq_switch+0x15f
sleepq_wait(c669adec,0,e8162c38,1,0,...) at sleepq_wait+0x63
_cv_wait(c669adec,c669ad84,c669adf4,1,c669adec,...) at _cv_wait+0x240
txg_thread_wait(0,0,c62ad24a,18d,c08c441b,...) at txg_thread_wait+0x4f
txg_quiesce_thread(c669ac00,e8162d38,c0c749df,343,c611e000,...) at txg_quiesce_thread+0xa0
fork_exit(c61fce20,c669ac00,e8162d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8162d70, ebp = 0 ---
        
Tracing command zfskern pid 1527 tid 100114 td 0xc6122d80
sched_switch(c6122d80,0,104,191,b9c5a414,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c6122d80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c65882c8,4c,c62b6b3a,0,0,...) at sleepq_wait+0x63
_sleep(c65882c8,c65882dc,24c,c62b6b3a,0,...) at _sleep+0x36b
vdev_geom_worker(c65882c0,e8156d38,c0c749df,343,c611e000,...) at vdev_geom_worker+0x124
fork_exit(c6251040,c65882c0,e8156d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8156d70, ebp = 0 ---

Tracing command zfskern pid 1527 tid 100087 td 0xc6120b40
sched_switch(c6120b40,0,104,191,7710eb92,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120b40,0,c0c7d8d5,26e,c6120b40,...) at sleepq_switch+0x15f
sleepq_timedwait(c62c2ff4,0,e810bc24,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c62c2ff4,c62c2fe0,3e8,1131,0,...) at _cv_timedwait+0x250
l2arc_feed_thread(0,e810bd38,c0c749df,343,c611e000,...) at l2arc_feed_thread+0x149
fork_exit(c61a4fa0,0,e810bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe810bd70, ebp = 0 ---

Tracing command zfskern pid 1527 tid 100078 td 0xc6122000
sched_switch(c6122000,0,104,191,7711309a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122000,0,c0c7d8d5,26e,c6122000,...) at sleepq_switch+0x15f
sleepq_timedwait(c62babcc,0,c57eac80,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c62babcc,c62babb8,3e8,0,3e8,...) at _cv_timedwait+0x250
arc_reclaim_thread(0,c57ead38,c0c749df,343,c611e000,...) at arc_reclaim_thread+0x3b7
fork_exit(c61a8ac0,0,c57ead38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc57ead70, ebp = 0 ---

Tracing command bash pid 1525 tid 100076 td 0xc5ede240
sched_switch(c5ede240,0,104,191,788fbd72,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2ae04,0,c0c77c84,c5ede240,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2ae70,0,c57e4b0c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2ae70,c5f2ae04,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c5f2ae00,c5f2ae70,c57e4c58,1,0,...) at tty_wait+0x71
ttydisc_read(c5f2ae00,c57e4c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c6137000,c57e4c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed4738,c57e4c58,c5f40300,0,c5ede240,...) at devfs_read_f+0x7e
dofileread(c57e4c58,ffffffff,ffffffff,0,c5ed4738,...) at dofileread+0x96
kern_readv(c5ede240,0,c57e4c58,c57e4c78,1,...) at kern_readv+0x58
read(c5ede240,c57e4cf8,c,c5ede240,c0d5f9b4,...) at read+0x4f
syscall(c57e4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x8127b93, esp = 0xbfbfe03c, ebp = 0xbfbfe068 ---

Tracing command sshd pid 1521 tid 100079 td 0xc6121d80
sched_switch(c6121d80,0,104,191,788df0ca,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5e58710,0,c0c77c84,c6121d80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5e58724,0,c57eda7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5e58724,c5e58710,c0c7fb95,5d1,c5ed4770,...) at _cv_wait_sig+0x240
seltdwait(c5ed4770,58,c60ff600,c6121d80,200246,...) at seltdwait+0xa2
kern_select(c6121d80,9,286030c8,286030cc,0,0,20,65,280fdd50) at kern_select+0x4f4
select(c6121d80,c57edcf8,14,c0c61635,c0d6038c,...) at select+0x66
syscall(c57edd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfde5c, ebp = 0xbfbfdea8 ---

Tracing command getty pid 1518 tid 100071 td 0xc5eded80
sched_switch(c5eded80,0,104,191,d92742d4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5eded80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5c7f404,0,c0c77c84,c5eded80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5c7f470,0,c57c0b0c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5c7f470,c5c7f404,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c5c7f400,c5c7f470,c57c0c58,1,0,...) at tty_wait+0x71
ttydisc_read(c5c7f400,c57c0c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5b00a00,c57c0c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3850,c57c0c58,c596b100,0,c5eded80,...) at devfs_read_f+0x7e
dofileread(c57c0c58,ffffffff,ffffffff,0,c5ed3850,...) at dofileread+0x96
kern_readv(c5eded80,0,c57c0c58,c57c0c78,1,...) at kern_readv+0x58
read(c5eded80,c57c0cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57c0d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedbc, ebp = 0xbfbfede8 ---

Tracing command getty pid 1517 tid 100083 td 0xc6121480
sched_switch(c6121480,0,104,191,914789b2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121480,c6121480,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59eca70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59eca70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59eca00,c59eca70,c57f9c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59eca00,c57f9c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85a00,c57f9c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3498,c57f9c58,c596b100,0,c6121480,...) at devfs_read_f+0x7e
dofileread(c57f9c58,ffffffff,ffffffff,0,c5ed3498,...) at dofileread+0x96
kern_readv(c6121480,0,c57f9c58,c57f9c78,1,...) at kern_readv+0x58
read(c6121480,c57f9cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57f9d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1516 tid 100077 td 0xc5ede000
sched_switch(c5ede000,0,104,191,9126c18e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede000,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c5ede000,c5ede000,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed270,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed270,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed200,c59ed270,c57e7c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed200,c57e7c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5b00100,c57e7c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3af0,c57e7c58,c596b100,0,c5ede000,...) at devfs_read_f+0x7e
dofileread(c57e7c58,ffffffff,ffffffff,0,c5ed3af0,...) at dofileread+0x96
kern_readv(c5ede000,0,c57e7c58,c57e7c78,1,...) at kern_readv+0x58
read(c5ede000,c57e7cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57e7d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1515 tid 100074 td 0xc5ede6c0
sched_switch(c5ede6c0,0,104,191,9105c10a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede6c0,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c5ede6c0,c5ede6c0,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59edc70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59edc70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59edc00,c59edc70,c57cac58,1,0,...) at tty_wait+0x71
ttydisc_read(c59edc00,c57cac58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85800,c57cac58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3930,c57cac58,c596b100,0,c5ede6c0,...) at devfs_read_f+0x7e
dofileread(c57cac58,ffffffff,ffffffff,0,c5ed3930,...) at dofileread+0x96
kern_readv(c5ede6c0,0,c57cac58,c57cac78,1,...) at kern_readv+0x58
read(c5ede6c0,c57cacf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57cad38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1514 tid 100081 td 0xc6121900
sched_switch(c6121900,0,104,191,90e4c0de,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121900,c6121900,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed670,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed670,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed600,c59ed670,c57f3c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed600,c57f3c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85700,c57f3c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3ce8,c57f3c58,c596b100,0,c6121900,...) at devfs_read_f+0x7e
dofileread(c57f3c58,ffffffff,ffffffff,0,c5ed3ce8,...) at dofileread+0x96
kern_readv(c6121900,0,c57f3c58,c57f3c78,1,...) at kern_readv+0x58
read(c6121900,c57f3cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57f3d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1513 tid 100063 td 0xc5b06240
sched_switch(c5b06240,0,104,191,90c3f93e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c5b06240,c5b06240,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ede70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ede70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ede00,c59ede70,c5799c58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ede00,c5799c58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85300,c5799c58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3620,c5799c58,c596b100,0,c5b06240,...) at devfs_read_f+0x7e
dofileread(c5799c58,ffffffff,ffffffff,0,c5ed3620,...) at dofileread+0x96
kern_readv(c5b06240,0,c5799c58,c5799c78,1,...) at kern_readv+0x58
read(c5b06240,c5799cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c5799d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1512 tid 100086 td 0xc6120d80
sched_switch(c6120d80,0,104,191,90a2bcba,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6120d80,c6120d80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed470,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed470,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed400,c59ed470,e80fcc58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed400,e80fcc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85e00,e80fcc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3bd0,e80fcc58,c596b100,0,c6120d80,...) at devfs_read_f+0x7e
dofileread(e80fcc58,ffffffff,ffffffff,0,c5ed3bd0,...) at dofileread+0x96
kern_readv(c6120d80,0,e80fcc58,e80fcc78,1,...) at kern_readv+0x58
read(c6120d80,e80fccf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(e80fcd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1511 tid 100085 td 0xc6121000
sched_switch(c6121000,0,104,191,908225fa,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121000,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121000,c6121000,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c59ed870,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c59ed870,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c59ed800,c59ed870,c57ffc58,1,0,...) at tty_wait+0x71
ttydisc_read(c59ed800,c57ffc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c85900,c57ffc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3508,c57ffc58,c596b100,0,c6121000,...) at devfs_read_f+0x7e
dofileread(c57ffc58,ffffffff,ffffffff,0,c5ed3508,...) at dofileread+0x96
kern_readv(c6121000,0,c57ffc58,c57ffc78,1,...) at kern_readv+0x58
read(c6121000,c57ffcf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57ffd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command getty pid 1510 tid 100084 td 0xc6121240
sched_switch(c6121240,0,104,191,918e02aa,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6121240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,c6121240,c6121240,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5c7f070,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5c7f070,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240
tty_wait(c5c7f000,c5c7f070,c57fcc58,1,0,...) at tty_wait+0x71
ttydisc_read(c5c7f000,c57fcc58,0,9e,0,...) at ttydisc_read+0x22c
ttydev_read(c5c86000,c57fcc58,0,0,1,...) at ttydev_read+0xaa
devfs_read_f(c5ed3700,c57fcc58,c596b100,0,c6121240,...) at devfs_read_f+0x7e
dofileread(c57fcc58,ffffffff,ffffffff,0,c5ed3700,...) at dofileread+0x96
kern_readv(c6121240,0,c57fcc58,c57fcc78,1,...) at kern_readv+0x58
read(c6121240,c57fccf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f
syscall(c57fcd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command cron pid 1459 tid 100067 td 0xc5ee06c0
sched_switch(c5ee06c0,0,104,191,7c0f71e0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c5ee06c0,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c351,c08bad00,c5ee06c0,0,100,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c0dceac4,5c,c0c7a7d5,100,0,...) at sleepq_timedwait_sig+0x1a
_sleep(c0dceac4,0,15c,c0c7a7d5,c351,...) at _sleep+0x31e
kern_nanosleep(c5ee06c0,c57b0c64,c57b0c6c,32,0,...) at kern_nanosleep+0xc1
nanosleep(c5ee06c0,c57b0cf8,8,c0c80283,c0d613a0,...) at nanosleep+0x6f
syscall(c57b0d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28179c2f, esp = 0xbfbfec8c, ebp = 0xbfbfecb8 ---
        
Tracing command sendmail pid 1453 tid 100069 td 0xc5ee0240
sched_switch(c5ee0240,0,104,191,61ca0cea,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,68,...) at mi_switch+0x200
sleepq_switch(c5ee0240,0,c0c7d8d5,18b,68,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f1a058,68,c0c2d13b,100,0,...) at sleepq_wait_sig+0x17
_sleep(c5f1a058,c5f1a088,168,c0c2d13b,0,...) at _sleep+0x354
kern_sigsuspend(c5ee0240,0,0,0,0,...) at kern_sigsuspend+0xe4
sigsuspend(c5ee0240,c57b8cf8,4,c0c80035,c0d61eac,...) at sigsuspend+0x4d
syscall(c57b8d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28332a3b, esp = 0xbfbfcf9c, ebp = 0xbfbfcfc8 ---

Tracing command sendmail pid 1447 tid 100066 td 0xc5ee0900
sched_switch(c5ee0900,0,104,191,54c1b16c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c57aca4c,c08738da,c5f2d650,0,c5ee0900,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c5f2d664,0,c57aca7c,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c5f2d664,c5f2d650,1389,5d1,c5ed32a0,...) at _cv_timedwait_sig+0x250
seltdwait(c57acc28,c57acc30,c6100600,c5ee0900,c188b014,...) at seltdwait+0x8a
kern_select(c5ee0900,5,bfbfc520,0,0,c57acc70,20,5,0) at kern_select+0x4f4
select(c5ee0900,c57accf8,14,c0c80559,c0d6038c,...) at select+0x66
syscall(c57acd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283d81d3, esp = 0xbfbfc48c, ebp = 0xbfbfcfb8 ---

Tracing command sshd pid 1440 tid 100068 td 0xc5ee0480
sched_switch(c5ee0480,0,104,191,7cd515d2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2d010,0,c0c77c84,c5ee0480,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2d024,0,c57b4a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2d024,c5f2d010,c0c7fb95,5d1,c5ed34d0,...) at _cv_wait_sig+0x240
seltdwait(c5ed34d0,58,c6100000,c5ee0480,0,...) at seltdwait+0xa2
kern_select(c5ee0480,5,286090ac,0,0,0,20,65,a) at kern_select+0x4f4
select(c5ee0480,c57b4cf8,14,c0c809ab,c0d6038c,...) at select+0x66
syscall(c57b4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfdf1c, ebp = 0xbfbfee38 ---

Tracing command syslogd pid 1210 tid 100072 td 0xc5edeb40
sched_switch(c5edeb40,0,104,191,b0d575ee,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5edeb40,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5c4bd50,0,c0c77c84,c5edeb40,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5c4bd64,0,c57c4a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5c4bd64,c5c4bd50,c0c7fb95,5d1,c5ed38c0,...) at _cv_wait_sig+0x240
seltdwait(c5ed38c0,58,c596b100,c5edeb40,0,...) at seltdwait+0xa2
kern_select(c5edeb40,9,282290a8,0,0,0,20,65,281a4ad8) at kern_select+0x4f4
select(c5edeb40,c57c4cf8,14,c0c80135,c0d6038c,...) at select+0x66
syscall(c57c4d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281921d3, esp = 0xbfbfde7c, ebp = 0xbfbfee28 ---

Tracing command dhclient pid 1161 tid 100075 td 0xc5ede480
sched_switch(c5ede480,0,104,191,5075fdca,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c57cdaa8,c08738da,c5f2d4d0,0,c5ede480,...) at sleepq_catch_signals+0xb7
sleepq_timedwait_sig(c5f2d4e4,0,c57cdad8,101,0,...) at sleepq_timedwait_sig+0x1a
_cv_timedwait_sig(c5f2d4e4,c5f2d4d0,5265819,5d1,c57cdb64,...) at _cv_timedwait_sig+0x250
seltdwait(c57cdc5c,c57cdc64,4df,c5ede480,c57cdb5c,...) at seltdwait+0x8a
poll(c5ede480,c57cdcf8,c,c0c18927,c0d6103c,...) at poll+0x300
syscall(c57cdd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command dhclient pid 1120 tid 100064 td 0xc5ee0d80
sched_switch(c5ee0d80,0,104,191,c880666e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2e950,0,c0c77c84,c5ee0d80,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2e964,0,c579dad8,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2e964,c5f2e950,c0c7fb95,5d1,c579db5c,...) at _cv_wait_sig+0x240
seltdwait(c5f6692c,c0c7fb95,4df,c5ee0d80,c579db5c,...) at seltdwait+0xa2
poll(c5ee0d80,c579dcf8,c,c579dcb8,c0d6103c,...) at poll+0x300
syscall(c579dd38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 ---

Tracing command devd pid 971 tid 100065 td 0xc5ee0b40
sched_switch(c5ee0b40,0,104,191,1d25f46e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ee0b40,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2de10,0,c0c77c84,c5ee0b40,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2de24,0,c57a1a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2de24,c5f2de10,c0c7fb95,5d1,c5ed3380,...) at _cv_wait_sig+0x240
seltdwait(c5ed3380,58,c596b100,c5ee0b40,c0c717ff,...) at seltdwait+0xa2
kern_select(c5ee0b40,6,bfbfe9b0,0,0,0,20,c0dd2800,b) at kern_select+0x4f4
select(c5ee0b40,c57a1cf8,14,c57a1ca0,c0d6038c,...) at select+0x66
syscall(c57a1d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x80880e3, esp = 0xbfbfe97c, ebp = 0xbfbfee58 ---

Tracing command moused pid 954 tid 100073 td 0xc5ede900
sched_switch(c5ede900,0,104,191,b66153a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ede900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f
sleepq_catch_signals(c08738da,c5f2d810,0,c0c77c84,c5ede900,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5f2d824,0,c57c7a7c,101,0,...) at sleepq_wait_sig+0x17
_cv_wait_sig(c5f2d824,c5f2d810,c0c7fb95,5d1,c5ed3b28,...) at _cv_wait_sig+0x240
seltdwait(c5ed3b28,58,c596b100,c5ede900,0,...) at seltdwait+0xa2
kern_select(c5ede900,7,bfbfea14,0,0,0,20,65,0) at kern_select+0x4f4
select(c5ede900,c57c7cf8,14,c0c916cc,c0d6038c,...) at select+0x66
syscall(c57c7d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (93, FreeBSD ELF32, select), eip = 0x281ac1d3, esp = 0xbfbfe99c, ebp = 0xbfbfeb38 ---

Tracing command flowcleaner pid 21 tid 100062 td 0xc5b06480
sched_switch(c5b06480,0,104,191,8776d332,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06480,0,c0c7d8d5,26e,c5b06480,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f3b548,0,c5791cc4,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c0f3b548,c0f3b550,2710,3f0,0,...) at _cv_timedwait+0x250
flowtable_cleaner(0,c5791d38,c0c749df,343,c5e9b000,...) at flowtable_cleaner+0x1bf
fork_exit(c092a660,0,c5791d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5791d70, ebp = 0 ---
        
Tracing command softdepflush pid 20 tid 100061 td 0xc5b066c0
sched_switch(c5b066c0,0,104,191,9a91c59a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5b066c0,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f46c40,44,c0ca021c,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f46c40,c0f46be4,44,c0ca021c,3e8,...) at _sleep+0x339
softdep_flush(0,c578ed38,c0c749df,343,c5e9b2a8,...) at softdep_flush+0x244
fork_exit(c0ab3a60,0,c578ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc578ed70, ebp = 0 ---

Tracing command vnlru pid 19 tid 100060 td 0xc5b06900
sched_switch(c5b06900,0,104,191,9a9157a6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,50,...) at mi_switch+0x200
sleepq_switch(c5b06900,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c5e9b550,50,c0c879e7,0,0,...) at sleepq_timedwait+0x6b
_sleep(c5e9b550,c0f3b314,250,c0c879e7,3e8,...) at _sleep+0x339
vnlru_proc(0,c578bd38,c0c749df,343,c5e9b550,...) at vnlru_proc+0xe7
fork_exit(c0916060,0,c578bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc578bd70, ebp = 0 ---

Tracing command syncer pid 18 tid 100059 td 0xc5b06b40
sched_switch(c5b06b40,0,104,191,77355a82,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06b40,0,c0c7d8d5,26e,c5b06b40,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f3b354,0,c5788c88,1,0,...) at sleepq_timedwait+0x6b
_cv_timedwait(c0f3b354,c0f3b340,3e8,6cc,4e20,...) at _cv_timedwait+0x250
sched_sync(0,c5788d38,c0c749df,343,c5e9b7f8,...) at sched_sync+0x502
fork_exit(c0915490,0,c5788d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5788d70, ebp = 0 ---

Tracing command bufdaemon pid 17 tid 100058 td 0xc5b06d80
sched_switch(c5b06d80,0,104,191,9a911762,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5b06d80,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f3b088,44,c0c84f5d,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f3b088,c0f3b08c,44,c0c84f5d,3e8,...) at _sleep+0x339
buf_daemon(0,c5785d38,c0c749df,343,c5e9baa0,...) at buf_daemon+0x138
fork_exit(c08fd800,0,c5785d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5785d70, ebp = 0 ---

Tracing command pagezero pid 16 tid 100057 td 0xc5c82000
sched_switch(c5c82000,0,104,191,e6d3fcae,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5c82000,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f47914,0,c0ca5b25,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f47914,c0f474d0,0,c0ca5b25,493e0,...) at _sleep+0x339
vm_pagezero(0,c5782d38,c0c749df,343,c5e9bd48,...) at vm_pagezero+0xdc
fork_exit(c0af0df0,0,c5782d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5782d70, ebp = 0 ---

Tracing command vmdaemon pid 15 tid 100056 td 0xc5c82240
sched_switch(c5c82240,0,104,191,794357b6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,68,...) at mi_switch+0x200
sleepq_switch(c5c82240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0f4753c,68,c0c84f5d,0,0,...) at sleepq_wait+0x63
_sleep(c0f4753c,c0f47540,68,c0c84f5d,0,...) at _sleep+0x36b
vm_daemon(0,c577fd38,c0c749df,343,c596e2a8,...) at vm_daemon+0x59
fork_exit(c0aeb240,0,c577fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc577fd70, ebp = 0 ---

Tracing command pagedaemon pid 9 tid 100055 td 0xc5c82480
sched_switch(c5c82480,0,104,191,41e649ea,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c5c82480,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0f47504,44,c0c84f5d,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0f47504,c0f474d0,44,c0c84f5d,1388,...) at _sleep+0x339
vm_pageout(0,c577cd38,c0c749df,343,c596e550,...) at vm_pageout+0x2bb
fork_exit(c0aec0e0,0,c577cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc577cd70, ebp = 0 ---

Tracing command sctp_iterator pid 8 tid 100054 td 0xc5c826c0
sched_switch(c5c826c0,0,104,191,be4bf2fa,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5c826c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0f3ce3c,0,c0c92e35,0,0,...) at sleepq_wait+0x63
_sleep(c0f3ce3c,c0f3cd50,0,c0c92e35,0,...) at _sleep+0x36b
sctp_iterator_thread(0,c5779d38,c0c749df,343,c596e7f8,...) at sctp_iterator_thread+0x60
fork_exit(c099ed00,0,c5779d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5779d70, ebp = 0 ---

Tracing command fdc0 pid 7 tid 100050 td 0xc5ad66c0
sched_switch(c5ad66c0,0,104,191,76ec9a8a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c5ad66c0,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c59ec23c,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c59ec23c,c59ec2f0,4c,c0c6f250,3e8,...) at _sleep+0x339
fdc_thread(c59ec200,c576cd38,c0c749df,343,c596eaa0,...) at fdc_thread+0x2be
fork_exit(c0b6e360,c59ec200,c576cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc576cd70, ebp = 0 ---

Tracing command fw0_probe pid 6 tid 100047 td 0xc5ad6d80
sched_switch(c5ad6d80,0,104,191,397c48de,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c5ad6d80,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c5b6e000,5c,c0c6f250,100,0,...) at sleepq_wait_sig+0x17
_sleep(c5b6e000,c5b72488,15c,c0c6f250,0,...) at _sleep+0x354
fw_bus_probe_thread(c5b6e000,c5588d38,c0c749df,343,c596ed48,...) at fw_bus_probe_thread+0xa08
fork_exit(c0654740,c5b6e000,c5588d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5588d70, ebp = 0 ---

Tracing command usb pid 14 tid 100042 td 0xc5b05900
sched_switch(c5b05900,0,104,191,ab8bccb2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05900,0,c0c7d8d5,24b,c5b05900,...) at sleepq_switch+0x15f
sleepq_wait(c5aedd0c,0,c555ecb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedd0c,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedd04,c555ed38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedd04,c555ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc555ed70, ebp = 0 ---

Tracing command usb pid 14 tid 100041 td 0xc5b05b40
sched_switch(c5b05b40,0,104,191,1e3c062e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05b40,0,c0c7d8d5,24b,c5b05b40,...) at sleepq_switch+0x15f
sleepq_wait(c5aedcdc,0,c555bcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedcdc,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedcd4,c555bd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedcd4,c555bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc555bd70, ebp = 0 ---

Tracing command usb pid 14 tid 100040 td 0xc5b05d80
sched_switch(c5b05d80,0,104,191,b9c1c834,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05d80,0,c0c7d8d5,24b,c5b05d80,...) at sleepq_switch+0x15f
sleepq_wait(c5aedcac,0,c5558cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedcac,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedca4,c5558d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedca4,c5558d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5558d70, ebp = 0 ---

Tracing command usb pid 14 tid 100039 td 0xc5b06000
sched_switch(c5b06000,0,104,191,8b9b48a0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b06000,0,c0c7d8d5,24b,c5b06000,...) at sleepq_switch+0x15f
sleepq_wait(c5aedc7c,0,c5555cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aedc7c,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aedc74,c5555d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aedc74,c5555d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5555d70, ebp = 0 ---

Tracing command usb pid 14 tid 100037 td 0xc5ad4000
sched_switch(c5ad4000,0,104,191,8b9b1c18,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4000,0,c0c7d8d5,24b,c5ad4000,...) at sleepq_switch+0x15f
sleepq_wait(c5adadac,0,c554dcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adadac,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adada4,c554dd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adada4,c554dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc554dd70, ebp = 0 ---

Tracing command usb pid 14 tid 100036 td 0xc5ad4240
sched_switch(c5ad4240,0,104,191,ddca875a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4240,0,c0c7d8d5,24b,c5ad4240,...) at sleepq_switch+0x15f
sleepq_wait(c5adad7c,0,c554acb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adad7c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adad74,c554ad38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adad74,c554ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc554ad70, ebp = 0 ---

Tracing command usb pid 14 tid 100035 td 0xc5ad4480
sched_switch(c5ad4480,0,104,191,8589b4b4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4480,0,c0c7d8d5,24b,c5ad4480,...) at sleepq_switch+0x15f
sleepq_wait(c5adad4c,0,c5547cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adad4c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adad44,c5547d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adad44,c5547d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5547d70, ebp = 0 ---

Tracing command usb pid 14 tid 100034 td 0xc5ad46c0
sched_switch(c5ad46c0,0,104,191,85898ec8,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad46c0,0,c0c7d8d5,24b,c5ad46c0,...) at sleepq_switch+0x15f
sleepq_wait(c5adad1c,0,c5544cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5adad1c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5adad14,c5544d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5adad14,c5544d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5544d70, ebp = 0 ---

Tracing command usb pid 14 tid 100032 td 0xc5ad4b40
sched_switch(c5ad4b40,0,104,191,8589586c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4b40,0,c0c7d8d5,24b,c5ad4b40,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5dac,0,c553dcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5dac,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5da4,c553dd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5da4,c553dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc553dd70, ebp = 0 ---

Tracing command usb pid 14 tid 100031 td 0xc5ad4d80
sched_switch(c5ad4d80,0,104,191,dd5c7726,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad4d80,0,c0c7d8d5,24b,c5ad4d80,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d7c,0,c553acb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d7c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5d74,c553ad38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5d74,c553ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc553ad70, ebp = 0 ---

Tracing command usb pid 14 tid 100030 td 0xc5ad6000
sched_switch(c5ad6000,0,104,191,7f759cb0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad6000,0,c0c7d8d5,24b,c5ad6000,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d4c,0,c5537cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d4c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5d44,c5537d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5d44,c5537d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5537d70, ebp = 0 ---

Tracing command usb pid 14 tid 100029 td 0xc5ad6240
sched_switch(c5ad6240,0,104,191,7f757954,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5ad6240,0,c0c7d8d5,24b,c5ad6240,...) at sleepq_switch+0x15f
sleepq_wait(c5ac5d1c,0,c5534cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5ac5d1c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5ac5d14,c5534d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5ac5d14,c5534d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5534d70, ebp = 0 ---

Tracing command usb pid 14 tid 100027 td 0xc59b56c0
sched_switch(c59b56c0,0,104,191,de33bc1e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b56c0,0,c0c7d8d5,24b,c59b56c0,...) at sleepq_switch+0x15f
sleepq_wait(c5aaedac,0,c552dcb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaedac,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaeda4,c552dd38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaeda4,c552dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc552dd70, ebp = 0 ---

Tracing command usb pid 14 tid 100026 td 0xc59b5900
sched_switch(c59b5900,0,104,191,de342e32,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5900,0,c0c7d8d5,24b,c59b5900,...) at sleepq_switch+0x15f
sleepq_wait(c5aaed7c,0,c552acb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaed7c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaed74,c552ad38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaed74,c552ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc552ad70, ebp = 0 ---

Tracing command usb pid 14 tid 100025 td 0xc59b5b40
sched_switch(c59b5b40,0,104,191,72ec5146,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5b40,0,c0c7d8d5,24b,c59b5b40,...) at sleepq_switch+0x15f
sleepq_wait(c5aaed4c,0,c5527cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaed4c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaed44,c5527d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaed44,c5527d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5527d70, ebp = 0 ---

Tracing command usb pid 14 tid 100024 td 0xc59b5d80
sched_switch(c59b5d80,0,104,191,72edde2a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5d80,0,c0c7d8d5,24b,c59b5d80,...) at sleepq_switch+0x15f
sleepq_wait(c5aaed1c,0,c5524cb8,1,0,...) at sleepq_wait+0x63
_cv_wait(c5aaed1c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240
usb_process(c5aaed14,c5524d38,c0c749df,343,c59d9000,...) at usb_process+0x193
fork_exit(c07b7c80,c5aaed14,c5524d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5524d70, ebp = 0 ---

Tracing command xpt_thrd pid 5 tid 100018 td 0xc5a85b40
sched_switch(c5a85b40,0,104,191,795df7fc,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c5a85b40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c0d9b0d4,4c,c0c14384,0,0,...) at sleepq_wait+0x63
_sleep(c0d9b0d4,c0d9b0ec,4c,c0c14384,0,...) at _sleep+0x36b
xpt_scanner_thread(0,c54fbd38,c0c749df,343,c59d92a8,...) at xpt_scanner_thread+0x4a
fork_exit(c0483010,0,c54fbd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54fbd70, ebp = 0 ---

Tracing command yarrow pid 13 tid 100011 td 0xc59b5000
sched_switch(c59b5000,0,104,191,d456c068,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5000,0,c0c7d8d5,26e,2,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dce924,0,c0c6f250,2,0,...) at sleepq_timedwait+0x6b
_sleep(c0dce924,0,0,c0c6f250,64,...) at _sleep+0x339
pause(c0c6f250,64,c0c5bcd5,111,0,...) at pause+0x47
random_kthread(0,c54e6d38,c0c749df,343,c59d9550,...) at random_kthread+0x1ef
fork_exit(c07308f0,0,c54e6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e6d70, ebp = 0 ---

Tracing command g_down pid 4 tid 100009 td 0xc59b5480
sched_switch(c59b5480,0,104,191,d6a10fd4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c59b5480,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc6e4,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc6e4,c0dcc648,24c,c0c6f250,64,...) at _sleep+0x339
g_io_schedule_down(c59b5480,0,c0c70886,74,0,...) at g_io_schedule_down+0x6b
g_down_procbody(0,c54e0d38,c0c749df,343,c596d000,...) at g_down_procbody+0x8d
fork_exit(c0820a20,0,c54e0d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e0d70, ebp = 0 ---

Tracing command g_up pid 3 tid 100008 td 0xc596f000
sched_switch(c596f000,0,104,191,d6a139cc,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c596f000,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc6e0,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc6e0,c0dcc668,24c,c0c6f250,64,...) at _sleep+0x339
g_io_schedule_up(c596f000,0,c0c70886,5d,0,...) at g_io_schedule_up+0x133
g_up_procbody(0,c54ddd38,c0c749df,343,c596d2a8,...) at g_up_procbody+0x8d
fork_exit(c0820ab0,0,c54ddd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54ddd70, ebp = 0 ---

Tracing command g_event pid 2 tid 100007 td 0xc596f240
sched_switch(c596f240,0,104,191,d4571590,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200
sleepq_switch(c596f240,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc6d8,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc6d8,0,4c,c0c6f250,64,...) at _sleep+0x339
g_event_procbody(0,c54dad38,c0c749df,343,c596d550,...) at g_event_procbody+0xcb
fork_exit(c0820b40,0,c54dad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54dad70, ebp = 0 ---

Tracing command intr pid 12 tid 100053 td 0xc5c82900
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100052 td 0xc5c82b40
sched_switch(c5c82b40,0,109,191,d922a5cc,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c5c79070,...) at mi_switch+0x200
ithread_loop(c5c842e0,c5773d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5c842e0,c5773d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5773d70, ebp = 0 ---

Tracing command intr pid 12 tid 100051 td 0xc5c82d80
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100049 td 0xc5ad6900
sched_switch(c5ad6900,0,109,191,8409d254,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b42f0,...) at mi_switch+0x200
ithread_loop(c5c6c620,c575fd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5c6c620,c575fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc575fd70, ebp = 0 ---

Tracing command intr pid 12 tid 100048 td 0xc5ad6b40
sched_switch(c5ad6b40,0,109,191,766e7c92,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b4370,...) at mi_switch+0x200
ithread_loop(c5c6c490,c56f4d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5c6c490,c56f4d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc56f4d70, ebp = 0 ---

Tracing command intr pid 12 tid 100045 td 0xc5b05240
sched_switch(c5b05240,0,109,191,79566bac,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b41f0,...) at mi_switch+0x200
ithread_loop(c5b078e0,c557fd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5b078e0,c557fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc557fd70, ebp = 0 ---

Tracing command intr pid 12 tid 100043 td 0xc5b056c0
sched_switch(c5b056c0,0,109,191,78975ef2,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3ef0,...) at mi_switch+0x200
ithread_loop(c5b07980,c5576d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5b07980,c5576d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5576d70, ebp = 0 ---

Tracing command intr pid 12 tid 100038 td 0xc5a85d80
sched_switch(c5a85d80,0,109,191,b9c1897c,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3df0,...) at mi_switch+0x200
ithread_loop(c5aaa740,c5552d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5aaa740,c5552d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5552d70, ebp = 0 ---

Tracing command intr pid 12 tid 100033 td 0xc5ad4900
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100028 td 0xc5ad6480
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100023 td 0xc5a85000
sched_switch(c5a85000,0,109,191,de33447e,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b4270,...) at mi_switch+0x200
ithread_loop(c5aa9820,c5521d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c5aa9820,c5521d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5521d70, ebp = 0 ---

Tracing command intr pid 12 tid 100022 td 0xc5a85240
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100020 td 0xc5a856c0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100019 td 0xc5a85900
sched_switch(c5a85900,0,109,191,b9c50404,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c5a838f0,...) at mi_switch+0x200
ithread_loop(c590aa40,c54fed38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c590aa40,c54fed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54fed70, ebp = 0 ---

Tracing command intr pid 12 tid 100013 td 0xc5971b40
sched_switch(c5971b40,0,109,191,387ac886,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3070,...) at mi_switch+0x200
ithread_loop(c590a9b0,c54ecd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c590a9b0,c54ecd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54ecd70, ebp = 0 ---

Tracing command intr pid 12 tid 100012 td 0xc5971d80
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 100006 td 0xc596f480
sched_switch(c596f480,0,109,191,d7c526fc,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3c70,...) at mi_switch+0x200
ithread_loop(c596c0c0,c54d7d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c596c0c0,c54d7d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54d7d70, ebp = 0 ---

Tracing command intr pid 12 tid 100005 td 0xc596f6c0
sched_switch(c596f6c0,0,109,191,7b7ba1ea,...) at sched_switch+0x406
mi_switch(109,0,c0c74c66,4f6,c59b3cf0,...) at mi_switch+0x200
ithread_loop(c596c0d0,c54d4d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6
fork_exit(c085c730,c596c0d0,c54d4d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54d4d70, ebp = 0 ---

Tracing command intr pid 12 tid 100004 td 0xc596f900
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 100003 td 0xc596fb40
sched_switch(c596fb40,0,608,18c,b9c14e64,...) at sched_switch+0x406
mi_switch(608,0,c0c79c0b,cf,0,...) at mi_switch+0x200
critical_exit(c5953f7c,c596fb40,c5953f7c,c59b3d80,17,...) at critical_exit+0xa8
intr_event_handle(c59b3d80,c54cdc7c,c54cdc74,2710,c0dd2800,...) at intr_event_handle+0xba
intr_execute_handlers(c5953f7c,c54cdc7c,0,c54cdcf8,c0b98a14,...) at intr_execute_handlers+0x49
lapic_handle_intr(34,c54cdc7c) at lapic_handle_intr+0x4c
Xapic_isr1() at Xapic_isr1+0x34
--- interrupt, eip = 0xc08a594c, esp = 0xc54cdcbc, ebp = 0xc54cdcf8 ---
sched_idletd(0,c54cdd38,c0c749df,343,c596daa0,...) at sched_idletd+0x21c
fork_exit(c08a5730,0,c54cdd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54cdd70, ebp = 0 ---

Tracing command init pid 1 tid 100002 td 0xc596fd80
sched_switch(c596fd80,0,104,191,7ccb36e6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200
sleepq_switch(c596fd80,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f
sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7
sleepq_wait_sig(c596dd48,5c,c0c80146,100,0,...) at sleepq_wait_sig+0x17
_sleep(c596dd48,c596ddd0,15c,c0c80146,0,...) at _sleep+0x354
kern_wait(c596fd80,ffffffff,c54c9c74,0,0,...) at kern_wait+0xb76
wait4(c596fd80,c54c9cf8,10,c0c7ff76,c0d5fa24,...) at wait4+0x3b
syscall(c54c9d38) at syscall+0x2b4
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b8f, esp = 0xbfbfe90c, ebp = 0xbfbfe928 ---

Tracing command audit pid 10 tid 100001 td 0xc5971000
sched_switch(c5971000,0,104,191,7959b734,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971000,0,c0c7d8d5,24b,c5971000,...) at sleepq_switch+0x15f
sleepq_wait(c0f46560,0,c54c6c9c,1,0,...) at sleepq_wait+0x63
_cv_wait(c0f46560,c0f46544,c0c9d95e,194,c0d67918,...) at _cv_wait+0x240
audit_worker(0,c54c6d38,c0c749df,343,c596e000,...) at audit_worker+0x84
fork_exit(c0a87420,0,c54c6d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54c6d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100118 td 0xc6122240
sched_switch(c6122240,0,104,191,c19cab6c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(d34c0cc0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(d34c0cc0,d34c0cd8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c72d5c80,e8170d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c72d5c80,e8170d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8170d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100115 td 0xc6122b40
sched_switch(c6122b40,0,104,191,7673f806,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6122b40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587940,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587940,c6587958,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5e6cc40,e8159d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5e6cc40,e8159d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8159d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100113 td 0xc658c000
sched_switch(c658c000,0,104,191,b9c61070,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588240,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588240,c6588258,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f390,e8153d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f390,e8153d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8153d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100112 td 0xc658c240
sched_switch(c658c240,0,104,191,6f062614,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588200,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588200,c6588218,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5b0c390,e8150d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5b0c390,e8150d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8150d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100111 td 0xc658c480
sched_switch(c658c480,0,104,191,6f0602b0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588180,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588180,c6588198,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f450,e814dd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f450,e814dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe814dd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100110 td 0xc658c6c0
sched_switch(c658c6c0,0,104,191,6f05dd6c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588140,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588140,c6588158,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4c0,e814ad38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4c0,e814ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe814ad70, ebp = 0 ---

Tracing command kernel pid 0 tid 100109 td 0xc658d240
sched_switch(c658d240,0,104,191,6f0351a8,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588100,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588100,c6588118,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f460,e8147d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f460,e8147d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8147d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100108 td 0xc658d480
sched_switch(c658d480,0,104,191,6f032cf4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c65880c0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c65880c0,c65880d8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f570,e8144d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f570,e8144d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8144d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100107 td 0xc6120000
sched_switch(c6120000,0,104,191,b97558e6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588080,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588080,c6588098,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f510,e8141d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f510,e8141d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8141d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100106 td 0xc6120240
sched_switch(c6120240,0,104,191,b85eff06,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e813ed38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e813ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe813ed70, ebp = 0 ---

Tracing command kernel pid 0 tid 100105 td 0xc658c900
sched_switch(c658c900,0,104,191,b85e93c2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658c900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e813bd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e813bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe813bd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100104 td 0xc658cb40
sched_switch(c658cb40,0,104,191,b85f7bde,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658cb40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8138d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8138d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8138d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100103 td 0xc658cd80
sched_switch(c658cd80,0,104,191,b85f588e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658cd80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8135d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8135d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8135d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100102 td 0xc658d000
sched_switch(c658d000,0,104,191,b85f3b6a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8132d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8132d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8132d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100101 td 0xc6120480
sched_switch(c6120480,0,104,191,b85f1fb2,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e812fd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e812fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe812fd70, ebp = 0 ---
        
Tracing command kernel pid 0 tid 100100 td 0xc61206c0
sched_switch(c61206c0,0,104,191,b85ec786,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c61206c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e8120d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e8120d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8120d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100099 td 0xc6120900
sched_switch(c6120900,0,104,191,b85f9ab6,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c6120900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f500,e811dd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f500,e811dd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe811dd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100098 td 0xc658ed80
sched_switch(c658ed80,0,104,191,89d1724a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658ed80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e811ad38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e811ad38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe811ad70, ebp = 0 ---

Tracing command kernel pid 0 tid 100097 td 0xc658eb40
sched_switch(c658eb40,0,104,191,89a3cffe,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658eb40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8117d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8117d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8117d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100096 td 0xc658d6c0
sched_switch(c658d6c0,0,104,191,89838012,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e812cd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e812cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe812cd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100095 td 0xc658d900
sched_switch(c658d900,0,104,191,8af49d86,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658d900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8129d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8129d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8129d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100094 td 0xc658db40
sched_switch(c658db40,0,104,191,8ab29942,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658db40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8126d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8126d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8126d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100093 td 0xc658dd80
sched_switch(c658dd80,0,104,191,8a832bb8,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658dd80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8123d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8123d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8123d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100092 td 0xc658e000
sched_switch(c658e000,0,104,191,8a1a880e,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8114d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8114d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8114d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100091 td 0xc658e240
sched_switch(c658e240,0,104,191,89f70072,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4f0,e8111d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4f0,e8111d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe8111d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100090 td 0xc658e480
sched_switch(c658e480,0,104,191,7c37d874,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e40,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e40,c6587e58,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f4d0,e810ed38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f4d0,e810ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe810ed70, ebp = 0 ---

Tracing command kernel pid 0 tid 100089 td 0xc658e6c0
sched_switch(c658e6c0,0,104,191,b9c3495c,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587e00,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587e00,c6587e18,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f490,e815cd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f490,e815cd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe815cd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100088 td 0xc658e900
sched_switch(c658e900,0,104,191,b9838a5a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c658e900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c6587dc0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c6587dc0,c6587dd8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5f3f2b0,e815fd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5f3f2b0,e815fd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe815fd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100046 td 0xc5b05000
sched_switch(c5b05000,0,104,191,d30d63e4,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05000,0,c0c7d8d5,24b,c5b05000,...) at sleepq_switch+0x15f
sleepq_wait(c5b08540,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5b08540,c5b08558,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c5b7249c,c5585d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c5b7249c,c5585d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5585d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100044 td 0xc5b05480
sched_switch(c5b05480,0,104,191,6b34f380,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5b05480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5abec80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5abec80,c5abec98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c5b61074,c557bd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c5b61074,c557bd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc557bd70, ebp = 0 ---

Tracing command kernel pid 0 tid 100021 td 0xc5a85480
sched_switch(c5a85480,0,104,191,aa033aea,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5a85480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5a5ee80,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5a5ee80,c5a5ee98,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0de1288,c5504d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c0de1288,c5504d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5504d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100017 td 0xc5971240
sched_switch(c5971240,0,104,191,91e1f390,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971240,0,c0c7d8d5,24b,c5971240,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9dee0,c54f8d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c0d9dee0,c54f8d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f8d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100016 td 0xc5971480
sched_switch(c5971480,0,104,191,91e1d060,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971480,0,c0c7d8d5,24b,c5971480,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9dee0,c54f5d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c0d9dee0,c54f5d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f5d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100015 td 0xc59716c0
sched_switch(c59716c0,0,104,191,91e1ab74,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59716c0,0,c0c7d8d5,24b,c59716c0,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63
msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d
taskqueue_thread_loop(c0d9dee0,c54f2d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94
fork_exit(c08bd6f0,c0d9dee0,c54f2d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54f2d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100014 td 0xc5971900
sched_switch(c5971900,0,104,191,91e061e0,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c5971900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5a5f3c0,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5a5f3c0,c5a5f3d8,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0dcd044,c54efd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c0dcd044,c54efd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54efd70, ebp = 0 ---
        
Tracing command kernel pid 0 tid 100010 td 0xc59b5240
sched_switch(c59b5240,0,104,191,aa3f995a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200
sleepq_switch(c59b5240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f
sleepq_wait(c5955c40,0,c0c6f250,0,0,...) at sleepq_wait+0x63
_sleep(c5955c40,c5955c58,0,c0c6f250,0,...) at _sleep+0x36b
taskqueue_thread_loop(c0ddfd20,c54e3d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba
fork_exit(c08bd6f0,c0ddfd20,c54e3d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc54e3d70, ebp = 0 ---

Tracing command kernel pid 0 tid 100000 td 0xc0dcca50
sched_switch(c0dcca50,0,104,191,8776839a,...) at sched_switch+0x406
mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200
sleepq_switch(c0dcca50,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f
sleepq_timedwait(c0dcc7a0,44,c0c7b93e,0,0,...) at sleepq_timedwait+0x6b
_sleep(c0dcc7a0,0,44,c0c7b93e,2710,...) at _sleep+0x339
scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x23e
mi_startup() at mi_startup+0x96
begin() at begin+0x2c


----------------------------------------------------------------------
 Head of dmesg

WARNING: console at 0xc0d5a000 has no name
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RC2 #4: Sat Oct 31 02:28:58 UTC 2009
    root@box8.ws.local:/usr/obj/usr/src/sys/WITHDDB
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2400.10-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf27  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4400<CNXT-ID,xTPR>
real memory  = 2147483648 (2048 MB)
avail memory = 2086248448 (1989 MB)


----------------------------------------------------------------------
 Kernel Config

include GENERIC

ident   WITHDDB

options KDB
options DDB
options BREAK_TO_DEBUGGER

makeoptions        DEBUG=-g
options     INVARIANTS
options     INVARIANT_SUPPORT
options     WITNESS
options     DEBUG_LOCKS
options     DEBUG_VFS_LOCKS
options     DIAGNOSTIC

From ronald-freebsd8 at klop.yi.org  Sat Oct 31 11:01:05 2009
From: ronald-freebsd8 at klop.yi.org (Ronald Klop)
Date: Sat Oct 31 11:01:16 2009
Subject: Plans for Logged/Journaled UFS
In-Reply-To: <20091030223225.GI5120@datapipe.com>
References: <20091030223225.GI5120@datapipe.com>
Message-ID: <op.u2nvjxto8527sy@82-170-177-25.ip.telfort.nl>

On Fri, 30 Oct 2009 23:32:25 +0100, Hussain Ali <hali@datapipe.net> wrote:

>
> Hello,
>
> Was just wondering if any of the LOG or Journal based UFS implementations
> are being worked on. I know of three that could make thier way into
> FreeBSD :
>
>  - bluffs :: IIRC ups@ was working on this
>  - wabl :: Wasabi's FFS extentions for NetBSD's UFS2
>  - solaris ufs :: that has journaling support - this is probably the
>    most work though
>
> ZFS doesnt suffice for may use cases - so just wondering if this is in
> the works.
>
> Thanks,
>
> --
> -hussain

FFS with gjournal? It is already in FreeBSD and is production ready.

Ronald.
From gavin at FreeBSD.org  Sat Oct 31 12:59:24 2009
From: gavin at FreeBSD.org (gavin@FreeBSD.org)
Date: Sat Oct 31 12:59:30 2009
Subject: kern/140134: [msdosfs] write and fsck destroy filesystem
	integrity
Message-ID: <200910311259.n9VCxN7Q069973@freefall.freebsd.org>

Old Synopsis: msdosfs write and fsck destroy filesystem integrity
New Synopsis: [msdosfs] write and fsck destroy filesystem integrity

Responsible-Changed-From-To: freebsd-amd64->freebsd-fs
Responsible-Changed-By: gavin
Responsible-Changed-When: Sat Oct 31 12:57:34 UTC 2009
Responsible-Changed-Why: 
Over to maintainer(s).

http://www.freebsd.org/cgi/query-pr.cgi?pr=140134
From jh at FreeBSD.org  Sat Oct 31 14:39:30 2009
From: jh at FreeBSD.org (jh@FreeBSD.org)
Date: Sat Oct 31 14:39:37 2009
Subject: kern/125536: [ext2fs] ext 2 mounts cleanly but fails on
	commands like ls
Message-ID: <200910311439.n9VEdT5A057332@freefall.freebsd.org>

Synopsis: [ext2fs] ext 2 mounts cleanly but fails on commands like ls

State-Changed-From-To: feedback->closed
State-Changed-By: jh
State-Changed-When: Sat Oct 31 14:38:17 UTC 2009
State-Changed-Why: 
Duplicate of kern/124621.

http://www.freebsd.org/cgi/query-pr.cgi?pr=125536
From jh at FreeBSD.org  Sat Oct 31 14:40:33 2009
From: jh at FreeBSD.org (jh@FreeBSD.org)
Date: Sat Oct 31 14:40:39 2009
Subject: kern/128173: [ext2fs] ls gives "Input/output error" on mounted
	ext3 filesystem
Message-ID: <200910311440.n9VEeX0R061318@freefall.freebsd.org>

Synopsis: [ext2fs] ls gives "Input/output error" on mounted ext3 filesystem

State-Changed-From-To: feedback->closed
State-Changed-By: jh
State-Changed-When: Sat Oct 31 14:39:41 UTC 2009
State-Changed-Why: 
Duplicate of kern/124621.

http://www.freebsd.org/cgi/query-pr.cgi?pr=128173
From jh at FreeBSD.org  Sat Oct 31 14:44:34 2009
From: jh at FreeBSD.org (jh@FreeBSD.org)
Date: Sat Oct 31 14:44:41 2009
Subject: kern/124621: [ext3] [patch] Cannot mount ext2fs partition
Message-ID: <200910311444.n9VEiXA2065482@freefall.freebsd.org>

Synopsis: [ext3] [patch] Cannot mount ext2fs partition

State-Changed-From-To: feedback->patched
State-Changed-By: jh
State-Changed-When: Sat Oct 31 14:41:11 UTC 2009
State-Changed-Why: 
Patched in head / stable/8 by r187395 and in stable/7 by r194495.

http://www.freebsd.org/cgi/query-pr.cgi?pr=124621
From ronald-freebsd8 at klop.yi.org  Sat Oct 31 17:53:37 2009
From: ronald-freebsd8 at klop.yi.org (Ronald Klop)
Date: Sat Oct 31 17:53:49 2009
Subject: Fwd: Re: zfs receive gives: internal error: Argument list too
 long
In-Reply-To: <20091029205121.GB3418@garage.freebsd.pl>
References: <op.u2df9msz8527sy@82-170-177-25.ip.telfort.nl>
	<op.u2dgfwap8527sy@82-170-177-25.ip.telfort.nl>
	<op.u2i2wknf8527sy@82-170-177-25.ip.telfort.nl>
	<20091029205121.GB3418@garage.freebsd.pl>
Message-ID: <op.u2oenhk88527sy@82-170-177-25.ip.telfort.nl>

On Thu, 29 Oct 2009 21:51:21 +0100, Pawel Jakub Dawidek <pjd@freebsd.org>  
wrote:

> On Wed, Oct 28, 2009 at 09:51:46PM +0100, Ronald Klop wrote:
>> Hi,
>>
>> I'm forwarding this, because there was no answer on freebsd-stable.
>>
>> Does anybody know about this and have some tips on how to solve it?
>
> Could you try this patch:
>
> 	http://people.freebsd.org/~pjd/patches/zfs_recv_E2BIG.patch
>

Thank you. The patch works on 8.0-RC2. My backups are up-to-date again.

Ronald.