Userland freezes during heavy packet forwarding workloads

[Scott Ullrich]

On 10/12/06, Chuck Swiger <cswiger at mac.com> wrote:
> On Oct 12, 2006, at 2:11 PM, Scott Ullrich wrote:
> > We (pfSense developers) have noticed an interesting problem where
> > userland stops functioning under high packet forwarding workloads.
> > Userland applications such as sshd and lighttpd freeze but userland
> > resumes after the network load eases.
>
> [ ...note that the links you posted included the trailing square
> bracket,
>       and thus were broken, although easily fixable by hand... ]
>
> > Is there anything we can do to prevent userland from stopping
> > completely when under heavy load?  Is this a bug?
>
> You're probably experiencing some form of "livelock"; your hardware
> isn't horribly fast, and if the NICs are generating interrupts
> rapidly enough due to the high rate of packet forwarding, you're not
> going to have a lot of spare CPU available to run userland tasks.
>
> I notice you're compiling in support for DEVICE_POLLING; does
> enabling it do anything to help the responsiveness of the userland
> tasks under high network load?  You might try adjusting HZ from 100
> to 250 or so...I found that made a decent tradeoff between packet
> delay and scheduler overhead on similar Soerkis 4801 or VIA C3/EPIA-M
> hardware to yours.

When polling is enabled, userland does indeed stay responsive, but
throughput also drops by 25-35% in testing on a Soekris 4801, 266 MHz.
 Again, this is not the case with 4.x.  These slower boxes actually
get as much as 20% faster with polling in 4.x, not slower as it does
in 6.x.  And with or without polling, userland always stays responsive
(albeit much slower, of course) on 4.x when under much higher pps
load.

At this time, we're not sure if this same problem exists in 5.x, but
if anyone thinks that information might be useful, we can test that as
well.

Thanks