High Send-Q values in netstat after upgrading to 12.1-RELEASE: system is unusable

[Jose G. Juanino]

Hi all,

today I have upgraded to 12.1-RELEASE from 12.0-RELEASE-p10 via
freebsd-update. The host is a VMware guest with vmx net interface. The
system runs with 12.0-RELEASE perfectly, with no issue, reliably for months.

I have boot environments enabled (thanks god) and therefore I am able to
switch between both 12.0 and 12.1 versions.

Just after boot the new 12.1 release, I noticed that my apache did not
serve pages with the previous speed, but rather the opposite: any web
operation from clients hangs.

After a single inspection in the FreeBSD 12.1 server, I see that Send-Q
in netstat is the root cause of the issue. While in 12.0 Send-Q is
almost always zero or close to zero in every socket, in 12.1 I get the
following (snipped output):


$ netstat -x -4
Active Internet connections
Proto Recv-Q Send-Q Local Address          Foreign Address        R-MBUF S-MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA R-BCNT S-BCNT R-BMAX S-BMAX   rexmt persist    keep    2msl  delack rcvtime
tcp4       0  31848 XXXXXXXXXXXXX.http     YYYYYYYYYYYY.57220          0     12      0     12  66350  33175      1   2048      0  48128 530800 265400   62.90    0.00 7069.49    0.00    0.00    1.09
tcp4       0     36 XXXXXXXXXXXXX.ssh      YYYYYYYYYYYY.57189          0      1      0      0  66350  33175      1   2048      0    256 530800 265400    0.28    0.00 6836.19    0.00    0.00    0.00


As you can see, Send-Q is huge.

But after this revelation, I have to admit that I am stuck. I have no
idea whether the cause of this high Send-Q values are related with the
vmx driver, with the upgrade itself, or with something related with the
hypervisor.

Please, help me to make a deeper troubleshooting. As I have said before,
my boot environment let me to start cleanly with both versions.

Other relevant fact may be also the difference between netstat -m and
ifconfig outputs:

In 12.0 netstat -m outputs:

268/1517/1785 mbufs in use (current/cache/total)
260/766/1026/250998 mbuf clusters in use (current/cache/total/max)
260/499 mbuf+clusters out of packet secondary zone in use (current/cache)
0/2/2/125499 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/37184 9k jumbo clusters in use (current/cache/total/max)
0/0/0/20916 16k jumbo clusters in use (current/cache/total/max)
587K/1919K/2506K bytes allocated to network (current/cache/total)


while in 12.1 outputs:

572/1213/1785 mbufs in use (current/cache/total)
515/515/1030/250998 mbuf clusters in use (current/cache/total/max)
4/502 mbuf+clusters out of packet secondary zone in use (current/cache)
24/2/26/125499 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/37184 9k jumbo clusters in use (current/cache/total/max)
0/0/0/20916 16k jumbo clusters in use (current/cache/total/max)
1269K/1341K/2610K bytes allocated to network (current/cache/total)


In 12.0 ifconfig vmx0 outputs (ip address and mac obfuscated):

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether TTTTTTTTTTTTTTTTT
        inet DD.DD.DD.DD netmask 0xffffff00 broadcast BB.BB.BB.BB
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

while in 12.1 outputs (notices JUMBO frames enabled):

vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether TTTTTTTTTTTTTTTTT
        inet DDDDDDDDDDDD netmask 0xffffff00 broadcast DDDDDDDDDDDDD
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>




Thanks in advanced, regards

-- 
Jose G. Juanino