cp from NFS to ZFS hung in "fifoor"

[Mikhail T.]

I was copying /home from an old server (narawntapu) to a new one
(aldan). The narawntapu:/home is mounted on aldan as /mnt with flags
ro,intr. On narawntapu /home was simply located on an SSD, but on aldan
I created a ZFS filesystem for it.

The copying was started thus:

    root at aldan:/home (435) cp -Rpn /mnt/* .

for a while this was proceeding at a decent clip with cp making
newnfsreq-uests:

    load: 0.78  cmd: cp 38711 [newnfsreq] 802.84r 1.57u 140.63s 20% 10768k
    /mnt/mi/.kde/share/apps/kmail/dimap/.42838394.directory/sent/cur/1219621413.32392.hd8cl:2,S
    ->
    ./mi/.kde/share/apps/kmail/dimap/.42838394.directory/sent/cur/1219621413.32392.hd8cl:2,S
    100%
    load: 1.23  cmd: cp 38711 [newnfsreq] 874.19r 1.66u 154.74s 17% 4576k
    /mnt/mi/.kde/share/apps/kmail/dimap/.42838394.directory/ML/cur/1219595347.32392.rMDFf:2,S
    ->
    ./mi/.kde/share/apps/kmail/dimap/.42838394.directory/ML/cur/1219595347.32392.rMDFf:2,S
    100%

ZFS on the destination compressing and writing stuff out and the traffic
between the two ranging from 30 to 50Mb/s (according to systat), but
then something happened and the cp-process is now hung:

    load: 0.55  cmd: cp 38711 [fifoor] 1107.67r 2.09u 194.12s 0% 3300k
    load: 0.50  cmd: cp 38711 [fifoor] 1112.66r 2.09u 194.12s 0% 3300k
    load: 0.22  cmd: cp 38711 [fifoor] 1642.37r 2.09u 194.12s 0% 3300k

There is nothing in the logs on the new system, but the old one has a
number of entries like:

    Nov 28 10:28:45 narawntapu kernel: sonewconn: pcb
    0xfffff80086231930: Listen queue overflow: 8 already in queue
    awaiting acceptance (62 occurrences)
    Nov 28 10:29:45 narawntapu kernel: sonewconn: pcb
    0xfffff80086231930: Listen queue overflow: 8 already in queue
    awaiting acceptance (50 occurrences)
    Nov 28 10:30:46 narawntapu kernel: sonewconn: pcb
    0xfffff80086231930: Listen queue overflow: 8 already in queue
    awaiting acceptance (59 occurrences)
    Nov 28 10:31:46 narawntapu kernel: sonewconn: pcb
    0xfffff80086231930: Listen queue overflow: 8 already in queue
    awaiting acceptance (57 occurrences)
    Nov 28 10:32:46 narawntapu kernel: sonewconn: pcb
    0xfffff80086231930: Listen queue overflow: 8 already in queue
    awaiting acceptance (68 occurrences)

Both systems are largely idle now. I'm not in a hurry -- is anybody
interested in investigating it in situ? What is "fifoor" -- does this
point to a trouble in the ZFS, the NFS-client, or the NFS-server? Both
systems run FreeBSD/amd64 of recent 10.x-vintage.

Thanks!

    -mi