User process starvation under heavy network traffic in FreeBSD 5.3

[Youlin Feng]

Hello,

 

I posted this question to freebsd-arch earlier. Stefan Bethke suggested
that I try freebsd-net instead.

 

We are building a network appliance running FreeBSD 5.3 and under very
heavy network traffic the user processes don't get scheduled for an
unacceptable period of time. Marking the user process/thread real-time
class doesn't help since the real-time user threads priorities are still
lower than the interrupt threads. BTW, in our case, the CPU spends very
long time at PI_NET (16) priority level, instead of at the (PI_SOFT + 4)
soft intr level due to the packet forwarding nature. Either way, the
user processes don't have a chance. In the following discussion I'll use
PI_NET to represent the network interrupt threads priority.

 

I am experimenting with improving the real-time threads scheduling
performance and would like to hear from you.

 

Here is what I am doing:

1.	Raise the minimum real-time user threads priority to a value
higher than PI_NET, e.g. 

#define PRI_MIN_REALTIME            0

And use the rtprio() syscall or command to set the priority to be higher
than PI_NET, e.g. "rtprio 10 <my_proc_id>"

 

This didn't turn out to be enough, since the real-time user process
still uses system services or drivers that run at a lower priority than
PI_NET, e.g. disk, tty, etc.

 

2.	Change the PI_NET value to be the highest of all interrupt
threads. Potential performance impact isn't a concern here since we have
relatively rare non-network events. So PI_NET is changed to
(PRI_MIN_ITHD + 32) from (PRI_MIN_ITHD + 16). This should give
preference to all other I/O interrupt threads. I am assuming the
software interrupt scheduling isn't the bottleneck. 

 

I haven't got a chance to reproduce the problem using the 2nd change
yet. I suspect that it isn't enough to get what I want, since lots of
kernel code do msleep(...,pri,...) with "pri" bigger than the changed
PI_NET. But, if this thinking of changing the priority ranges makes
sense, PI_NET can always be changed to the highest value of all kernel
priority values, ie the lowest kernel priority.

 

Hopefully this change would work and not be intrusive to the kernel.

 

Thank you for your comments in advance.

 

Youlin Feng