"panic: len 0" on NFS read

[Peter Jeremy]

Whilst trying to debug a RPC issue with a NFS tunneling tool, I mounted a
NFS filesystem onto the same host and got a panic when I tried to access it.

I'm running FreeBSD/amd64 10-stable r276177.

I mounted the filesystem with:
# mount -o udp,nfsv3 $(hostname):/tank/src92 /dist

(/tank/src92 and / are ZFS)

And then ran:
$ grep zzzz /dist/*

And got:
panic: len 0
cpuid = 3
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0861448f30
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0861448fe0
vpanic() at vpanic+0x126/frame 0xfffffe0861449020
kassert_panic() at kassert_panic+0x139/frame 0xfffffe0861449090
nfsm_mbufuio() at nfsm_mbufuio+0x9c/frame 0xfffffe08614490f0
nfsrpc_read() at nfsrpc_read+0x584/frame 0xfffffe08614492d0
ncl_readrpc() at ncl_readrpc+0xa5/frame 0xfffffe08614493e0
ncl_doio() at ncl_doio+0x228/frame 0xfffffe0861449480
ncl_bioread() at ncl_bioread+0xb44/frame 0xfffffe08614495f0
VOP_READ_APV() at VOP_READ_APV+0xf1/frame 0xfffffe0861449620
vn_read() at vn_read+0x211/frame 0xfffffe0861449690
vn_io_fault_doio() at vn_io_fault_doio+0x22/frame 0xfffffe08614496d0
vn_io_fault1() at vn_io_fault1+0x7c/frame 0xfffffe0861449830
vn_io_fault() at vn_io_fault+0x18b/frame 0xfffffe08614498b0
dofileread() at dofileread+0x95/frame 0xfffffe0861449900
kern_readv() at kern_readv+0x68/frame 0xfffffe0861449950
sys_read() at sys_read+0x63/frame 0xfffffe08614499a0
amd64_syscall() at amd64_syscall+0x22e/frame 0xfffffe0861449ab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0861449ab0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800fd3cba, rsp = 0x7fffffffe048, rbp = 0x7fffffffe090 ---

I have a crashdump that looks sane and relevant bits around nfsm_mbufuio() are:

#4  0xffffffff8041e63c in nfsm_mbufuio (nd=0xfffffe08614491b0, uiop=0xfffffe0861449420, siz=0x4000)
    at /usr/src/sys/fs/nfs/nfs_commonsubs.c:222
(kgdb) p mp
$1 = 0xfffff80053bab500
(kgdb) p *mp
$2 = {
  m_hdr = {
    mh_next = 0xfffff8023433dc00, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff80053bab57c "���"...,
    mh_len = 0x0, 
    mh_type = 0x1, 
    mh_flags = 0x2
  }, 
...
(kgdb) p *nd
$4 = {
  nd_md = 0xfffff8005366c500, 
  nd_dpos = 0xfffff80562d92068 "���"...,
...
(kgdb) p *nd->nd_md
$5 = {
  m_hdr = {
    mh_next = 0xfffff80486b05b00, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff80562d92000 "", 
    mh_len = 0x68, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$5.m_hdr.mh_next
$11 = {
  m_hdr = {
    mh_next = 0xfffff8005325e400, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff80234291800 "�", 
    mh_len = 0x800, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$11.m_hdr.mh_next
$12 = {
  m_hdr = {
    mh_next = 0xfffff80486b02400, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff8023453c000 "\t", 
    mh_len = 0x800, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$12.m_hdr.mh_next
$13 = {
  m_hdr = {
    mh_next = 0xfffff8023433f800, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff80562d92800 "its", 
    mh_len = 0x800, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$13.m_hdr.mh_next
$14 = {
  m_hdr = {
    mh_next = 0xfffff80020f36500, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff8058cb1b000 "sbconfig", 
    mh_len = 0x800, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$14.m_hdr.mh_next
$15 = {
  m_hdr = {
    mh_next = 0xfffff800533d5e00, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff8041b423800 "", 
    mh_len = 0x800, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$15.m_hdr.mh_next
$16 = {
  m_hdr = {
    mh_next = 0xfffff80053182600, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff8023429a800 "ilters", 
    mh_len = 0x800, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$16.m_hdr.mh_next
$17 = {
  m_hdr = {
    mh_next = 0xfffff8005379b200, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff8058cb1e000 "", 
    mh_len = 0x800, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...
(kgdb) p *$17.m_hdr.mh_next
$18 = {
  m_hdr = {
    mh_next = 0xfffff80053bab500, 
    mh_nextpkt = 0x0, 
    mh_data = 0xfffff8058cb1c800 "\002", 
    mh_len = 0x760, 
    mh_type = 0x1, 
    mh_flags = 0x1
  }, 
...

Which is points to mp.

I gather the first mbuf is NFS RPC metadata (since it's skipped).  The
remaining mbufs are the start of a 3.9MB binary file (an identifier
database).

Any suggestions as to what has gone wrong?

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 949 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20141226/3f439881/attachment.sig>