Poor NFS server performance in 6.0 with SMP and mpsafenet=1

[Jeremie Le Hen]

Hi, Gavin, Martin,

> Certainly.  For heavily loaded machines, polling seems to significantly
> improve performance, but in my experience on lightly loaded machines it
> seems to increase latency on response to packets.  For the particular
> workloads it will be used for, (serving web content and databases), the
> latency can be noticeable.
> 
> IMHO, polling is great for lower-spec boxes or highly loaded machines,
> but can hinder performance when the machine is lightly loaded.  I have a
> few machines which run with polling on, and for those usage patterns it
> really helps.  I just don't believe this is one of those cases.

Just FYI, em(4) has powerful thresholds managements concerning what
Intel calls "interrupt moderation".  This is plainly described in
this document (quite short to read) [1].  This looks like software
polling except this is managed by the network adapter itself.

Basically, em(4) network adapters have an "absolute timer" which
achieves, IIUC, the same goal as our software polling : when the timer
reaches zero, the adapter generates an interrupt.  But those adapters
also have a "packet timer", and this is very interesting and just
impossible to emulate this behaviour with software polling.  A packet
timer is shorter than an absolute timer in order to decrease latency :
when a packet is received the packet timer countdown begins.  If
another packet is received before it reaches zero, it is reset but
goes on couting down... and so on, until one of the absolute timer or
the packet timer timeouts.  (I hope I'm clear enough).

This allows to have the benefits of polling (limiting the interrupt
rate) without having the latency introduced by software polling.

AFAIK, this is tunable through these sysctls :
% obiwan:gogo# sysctl dev.em.0 | grep delay
% dev.em.0.rx_int_delay: 0
% dev.em.0.tx_int_delay: 66
% dev.em.0.rx_abs_int_delay: 66
% dev.em.0.tx_abs_int_delay: 66

Note that the packet timer is useless here.

BTW, given the above feature, is there any benefits to use software
polling of em(4) chips ?

Best regards,

[1] http://www.intel.com/design/network/applnots/ap450.pdf
-- 
Jeremie Le Hen
< jeremie at le-hen dot org >< ttz at chchile dot org >