NFS client hang on 13.2-RELEASE-p2 on file locking / wrong interface selected
Date: Thu, 24 Aug 2023 03:56:34 UTC
Hello, We are seeing NFS hangs on FreeBSD 13.2-RELEASE-p2 clients talking to a Debian bookworm NFS server using NFSv3. Whenever a process attempts to lock a file on an NFS mount, for example: lockf x sleep 3 That process hangs in state "nlmrcv" and goes to 100% CPU. I found huge numbers of exchanges like this via tcpdump: 03:05:41.432581 IP 172.17.200.2.998 > 172.17.250.10.50516: UDP, length 172 0x0000: 4500 00c8 9afb 0000 4011 c4f9 ac11 c802 E.......@....... 0x0010: ac11 fa0a 03e6 c554 00b4 1af6 04ed ac8b .......T........ 0x0020: 0000 0000 0000 0002 0001 86b5 0000 0004 ................ 0x0030: 0000 0002 0000 0001 0000 001c 64e6 c8e0 ............d... 0x0040: 0000 0002 6332 0000 0001 bb87 0001 bb87 ....c2.......... 0x0050: 0000 0001 0000 61a8 0000 0000 0000 0000 ......a......... 0x0060: 0000 0004 3873 0200 0000 0000 0000 0001 ....8s.......... 0x0070: 0000 0002 6332 0000 0000 0020 0100 0601 ....c2.......... 0x0080: a0da 65c4 00d0 6316 0000 0000 0000 0000 ..e...c......... 0x0090: 0a00 0a00 0000 0000 c733 0800 0000 0009 .........3...... 0x00a0: 3130 3030 3031 4063 3200 0000 0001 86a1 100001@c2....... 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0025 .......% 03:05:41.432632 IP 172.17.250.10.50516 > 172.17.200.2.998: UDP, length 20 0x0000: 4500 0030 4675 4000 4011 da17 ac11 fa0a E..0Fu@.@....... 0x0010: ac11 c802 c554 03e6 001c 1a5e 04ed ac8b .....T.....^.... 0x0020: 0000 0001 0000 0001 0000 0001 0000 0001 ................ 03:05:41.432647 IP 172.17.200.2.998 > 172.17.250.10.50516: UDP, length 172 0x0000: 4500 00c8 9afc 0000 4011 c4f8 ac11 c802 E.......@....... 0x0010: ac11 fa0a 03e6 c554 00b4 1af6 04ed ac8c .......T........ 0x0020: 0000 0000 0000 0002 0001 86b5 0000 0004 ................ 0x0030: 0000 0002 0000 0001 0000 001c 64e6 c8e0 ............d... 0x0040: 0000 0002 6332 0000 0001 bb87 0001 bb87 ....c2.......... 0x0050: 0000 0001 0000 61a8 0000 0000 0000 0000 ......a......... 0x0060: 0000 0004 3973 0200 0000 0000 0000 0001 ....9s.......... 0x0070: 0000 0002 6332 0000 0000 0020 0100 0601 ....c2.......... 0x0080: a0da 65c4 00d0 6316 0000 0000 0000 0000 ..e...c......... 0x0090: 0a00 0a00 0000 0000 c733 0800 0000 0009 .........3...... 0x00a0: 3130 3030 3031 4063 3200 0000 0001 86a1 100001@c2....... 0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x00c0: 0000 0000 0000 0025 .......% 03:05:41.432697 IP 172.17.250.10.50516 > 172.17.200.2.998: UDP, length 20 0x0000: 4500 0030 4676 4000 4011 da16 ac11 fa0a E..0Fv@.@....... 0x0010: ac11 c802 c554 03e6 001c 1a5e 04ed ac8c .....T.....^.... 0x0020: 0000 0001 0000 0001 0000 0001 0000 0001 ................ A huge number of these are exchanged. Like, several million over the span of a couple of minutes. Then the FreeBSD client system becomes unresponsive with these console messages: [nl_neigh] rtnl_lle_event: error allocating group writer [nl_neigh] rtnl_lle_event: error allocating group writer [nl_neigh] rtnl_lle_event: error allocating group writer [zone: mbuf] kern.ipc.nmbufs limit reached [nl_neigh] rtnl_lle_event: error allocating group writer [nl_neigh] rtnl_lle_event: error allocating group writer [nl_neigh] rtnl_lle_event: error allocating group writer Now, while 172.17.200.2 is an IP address on the client and 172.17.250.10 is an IP address on the server, that's the wrong subnet. The filesystem is mounted over a dedicated VLAN for NFS which has the IPs 172.20.200.2 and 172.20.250.10. So whatever this traffic is, it's using the wrong interface. I was able to work around this by swapping the 172.17.0.0 network from the NFS server. But that's not exactly an optimal solution. The main problem may be on the Debian side. I was able to workaround the issue by swapping the order of the network interfaces on the Debian side so the NFS VLAN was the "first" interface. That suggests to me that something on the Debian side is choosing the first interface instead of the right interface. So I'll pursue that. But it'd sure be nice if the FreeBSD client didn't hang in this situation. Does anyone know what might be happening here or have any other insight that might help me track this down? Thanks for any advice!