RFC: patches to pcbhash and porthash

[Richard Russo]

I've developed the attached patches that helped reduce overhead on a tcp proxy machine with high usage (2M+ open sessions / 4M+ tcp connections, 20k + tcp open/second). Software stack is FreeBSD 12-RELEASE, with RSS enabled and HAProxy 2 with local patches for RSS support.

Please let me know if these could be considered for inclusion, and what's the best way to proceed.

The first patch fbsd12-siphash_for_pcb_hash.patch changes the pcbhash so that it's using siphash instead of bit operations, uses the local address in addition to the foreign address, and includes the full address for ipv6 addresses. Because of the high number of sessions, I've had to use multiple local addresses to connect to the backend servers; in the unpatched case, these all hash to the same value; additional, because of how my organization allocates IPv6, all of my IPv6 backend servers have the same lower 32-bits, so I ended up with very large lists in the hash tables, resulting in excessive CPU consumption during pcb lookups.

The second patch fbsd12-move_comparison_to_portlist_head.patch changes the pcbporthash so that comparisons are done with the portheads, instead of potentially iterating on all pcbs that match the port. I also limited the size of the hashtable to a maximum 65536, since the hash is on port number and port number is limited to 65536. This helps reduce CPU spent checking for conflicts during binding of outgoing connections. I believe I correctly handled all of the cases, but there's a possibly wrong optimization for sockets with SO_REUSEADDR where I group all of these sockets as if they had their local address set to inaddr_any (or ipv6 version) -- this reduces the number of head entries to one per port number for outgoing connections in my case, instead of one per local address; this may not be correct or appropriate for all uses though.

Thank you,

Richard

-- 
  Richard Russo
  freebsd at ruka.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fbsd12-siphash_for_pcb_hash.patch
Type: application/octet-stream
Size: 14568 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-transport/attachments/20190618/7568fda2/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fbsd12-move_comparison_to_portlist_head.patch
Type: application/octet-stream
Size: 8790 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-transport/attachments/20190618/7568fda2/attachment-0001.obj>