Silly experiments with netisr

[Adrian Chadd]

On 5 February 2015 at 11:31, Scott Long via freebsd-net
<freebsd-net at freebsd.org> wrote:
>
>> On Feb 5, 2015, at 12:03 PM, Sean Bruno <sbruno at ignoranthack.me> wrote:
>>
>>
>> Signed PGP part
>> Some questions came up around the office and we ended up doing some
>> quite silly things with lo0 and netcat.
>>
>> If one runs a continuous netcat on localhost to another netcat listener
>> on localhost that writes the output to /dev/null, netisr gets super busy
>> doing stuff/things.
>>
>> E.g.
>>   -- listener running "nc -k -l 10000 > /dev/null"
>>   -  sender running in a while loop "nc -N localhost 10000 <
>> /var/tmp/testfile"
>>
>> Interesting things start happening on the machine.  top -SH shows netisr
>> eating up about 1/2 of a cpu core.  If you drop the MTU on lo0 to 1500
>> (so that it looks like something in the real world), netisr will peg out
>> a cpu core.  This seems logical, in that smaller MTU means busier
>> netisr.  Its interesting though.
>>
>> Looking at some pmcstat things, shows that the system is busilly
>> chugging along in tcp_do_segment().  I wonder if this is meaningful in
>> anyway or just "interesting".
>
>
> Welcome to our workload.  Granted, we don’t involve pf, but the majority of our CPU processing right now is spent in TCP (with the rest being spent in the VM, but that’s a different matter).
>
> FWIW, Randall has some optimizations in this area of the stack.  They aren’t huge, IIRC they’re only a few percent, but worth looking at.

Yeah, I see that too in all the TCP concurrency testing I'm doing.

The moment TCP TSO drops in effectiveness in any way, the cost of
tcp_do_segment() jumps dramatically. :(



-adrian