User process starvation under heavy network traffic in FreeBSD 5.3

[Robert Watson]

On Mon, 3 Jan 2005, Youlin Feng wrote:

> We are building a network appliance running FreeBSD 5.3 and under very
> heavy network traffic the user processes don't get scheduled for an
> unacceptable period of time. Marking the user process/thread real-time
> class doesn't help since the real-time user threads priorities are still
> lower than the interrupt threads. BTW, in our case, the CPU spends very
> long time at PI_NET (16) priority level, instead of at the (PI_SOFT + 4) 
> soft intr level due to the packet forwarding nature. Either way, the
> user processes don't have a chance. In the following discussion I'll use
> PI_NET to represent the network interrupt threads priority. 

I've seen some similar live lock issues with large quantities of local IPC
traffic on SMP systems.  Could you tell me if you are using an SMP system
or SMP kernel?  Are you using direct dispatch or link layer bridging
that's running in the ithread as opposed to the netisr swi?

Robert N M Watson


> 
>  
> 
> I am experimenting with improving the real-time threads scheduling
> performance and would like to hear from you.
> 
>  
> 
> Here is what I am doing:
> 
> 1.	Raise the minimum real-time user threads priority to a value
> higher than PI_NET, e.g. 
> 
> #define PRI_MIN_REALTIME            0
> 
> And use the rtprio() syscall or command to set the priority to be higher
> than PI_NET, e.g. "rtprio 10 <my_proc_id>"
> 
>  
> 
> This didn't turn out to be enough, since the real-time user process
> still uses system services or drivers that run at a lower priority than
> PI_NET, e.g. disk, tty, etc.
> 
>  
> 
> 2.	Change the PI_NET value to be the highest of all interrupt
> threads. Potential performance impact isn't a concern here since we have
> relatively rare non-network events. So PI_NET is changed to
> (PRI_MIN_ITHD + 32) from (PRI_MIN_ITHD + 16). This should give
> preference to all other I/O interrupt threads. I am assuming the
> software interrupt scheduling isn't the bottleneck. 
> 
>  
> 
> I haven't got a chance to reproduce the problem using the 2nd change
> yet. I suspect that it isn't enough to get what I want, since lots of
> kernel code do msleep(...,pri,...) with "pri" bigger than the changed
> PI_NET. But, if this thinking of changing the priority ranges makes
> sense, PI_NET can always be changed to the highest value of all kernel
> priority values, ie the lowest kernel priority.
> 
>  
> 
> Hopefully this change would work and not be intrusive to the kernel.
> 
>  
> 
> Thank you for your comments in advance.
> 
>  
> 
> Youlin Feng
> 
>  
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>