Panic in the udp_input() under heavy load

[Bjoern A. Zeeb]

On Mon, 7 Nov 2011, Maxim Sobolev wrote:

> Hi Gang,
>
> We are seeing repeatable panics under high PPS load on our production 
> systems. It happens when the traffic gets into the range or 200MBps and 
> 150-200K PPS. We have been managed to track it down to the following piece of 
> code:
>
> (gdb) l *udp_input+0x5d2
> 0xffffffff806f6202 is in udp_input (/usr/src/sys/netinet/udp_usrreq.c:628).
> 623             if (inp->inp_ip_minttl && inp->inp_ip_minttl > ip->ip_ttl) {
> 624                     INP_RUNLOCK(inp);
> 625                     goto badunlocked;
> 626             }
> 627             up = intoudpcb(inp);
> 628             if (up->u_tun_func == NULL) {
> 629                     udp_append(inp, ip, m, iphlen + sizeof(struct 
> udphdr), &udp_in);
> 630             } else {
> 631                     /*
> 632                      * Engage the tunneling protocol.
>
> The faulty line appears to be 628, with up value is being NULL, attempt to 
> deference it causes NULL pointer exception. I believe this particular piece 
> of code has been introduced here:

Unlikely; the inp is properly locked there and the udp info attach
better still be valid there; your problem is most likely elsewhere;
try to see if you have other threads and see what they do at the same
time, etc.  You would need to race with udp_detach();  you also want
to make sure that the inp still looks sane from either ddb or a dump
and we are not talking about random memory corruption here.

/bz

-- 
Bjoern A. Zeeb                                 You have to have visions!
          Stop bit received. Insert coin for new address family.