svn commit: r357051 - head/sys/dev/bge

[Gleb Smirnoff]

On Sun, Jan 26, 2020 at 09:48:42AM +0000, John Baldwin wrote:
J> > We also are concerned about that theoretically. Haven't yet seen effect
J> > in practice, but our sessions are mostly longer living. First we have the
J> > tunable to limit batching. Second, there are some ideas on how to improve
J> > the garbage collector performance if it becomes an issue.
J> 
J> There are other workloads than Netflix. ;)  Verisign has incredibly short-lived
J> connections with very high turnover.  I think though that they have already
J> abandoned the in-tree network stack for a userland stack built on netmap.  Still,
J> I think that there are probably other FreeBSD users that are probably somewhere
J> in the middle that shouldn't be ignored.

I understand that.

First, my change doesn't create extra work for the garbage collector, but it
potentially may affect its burstiness. If a machine has very high connection
turnover it already may exhibit some problems with the PCB garbage collector
on 12.0-RELEASE. Have we observed this yet?

Note that PCBs are very different to other things protected by the network
epoch: address lists, interface lists, firewall rules, etc. They are short
lived.

We got several ideas on how to deal with this potential problem:

1) The current PCB destructor does lots of things like freeing and
   unrefcounting associated multicast options, credentials, etc. This
   is because it was converted to epoch with a principle of minimal
   diff in r335015. However, since all operations with a PCB happen
   under its individual lock, we can free it differently. We can mark
   it INP_FREED and do all the destruction immediately, leaving only
   actual free to the garbage collector.
   Once this is achieved, the garbage collection can be batched at
   level of UMA. Jeff is working on that.

2) Use separate epoch for PCBs, leaving the network epoch for long
   lived structures only. Don't hold the PCB epoch long.

3) Just don't use epoch for PCBs. Decompose the hash lock into per
   slot hash lock.

-- 
Gleb Smirnoff