[solved/workaround?] Re: em very slow, shared irq... on 6.3p8

[Holger Kipp]

On Wed, Feb 27, 2008 at 09:50:16AM -0500, Mike Tancsa wrote:

more details below. as it currently is, polling seems to do
the trick, however handling several em-interfaces with the
same irq (mind you, it is pci) shouldn't cause delays of
up to 1.5 seconds for a simple ping... Therefore I consider
using polling for a nearly idle system more a workaround
than a solution to this problem :-(

> At 05:49 AM 2/27/2008, Holger Kipp wrote:
> 
> >I therefore assume that the problem is between receiving the
> >irq from em<x> and getting the data from the interface on the firewall
> >itself.
> 
> I would try upgrading to 6.3R (there are several em driver bug fixes) 
done. system is now 6.3-RELEASE-p1 which also gave me -c option for pciconf
and msi syscontrols (were missing in the old 6.2).


> and then try the box with
> % cat /boot/loader.conf
> hw.pci.enable_msi=1
> 
> ...if the cards support msi.
> 
> I think pciconf -lvc should tell you if the cards and slots support it or 
> not.

pciconf -lvc says for all em<x>:
cap 05[d0] = MSI supports 1 message, 64 bit
so I assume they do support MSI.

with msi disabled I get

38 packets transmitted, 38 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.833/228.022/1539.337/339.768 ms

with msi enabled (via sysctl) I get

33 packets transmitted, 33 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.865/156.421/1339.841/239.375 ms

so looks equally bad (I don't consider 30-40 packets a meaningful sample).
I don't know if it makes any differences if switched on directly in
loader.conf, though.

enabling polling (withous MSI) gives

30 packets transmitted, 30 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.366/0.790/1.339/0.290 ms

(maybe I should have used HZ=2000 to keep it below 0.6ms ;-)

> Also, if you dont need IPV6, use FAST_IPSEC. It does not need 
> mpsafe.  If you do need IPSEC and IPV6, 7.0R got rid of that restriction.

I think this enough changes in one go for a production system ;-)

many thanks for the recommendations!

Best regards,
Holger