kern/172963: Kernel panic in udp_input()
Miller, Vincent (Rick)
vmiller at verisign.com
Wed Oct 24 12:20:02 UTC 2012
The following reply was made to PR kern/172963; it has been noted by GNATS.
From: "Miller, Vincent (Rick)" <vmiller at verisign.com>
To: "bug-followup at FreeBSD.org" <bug-followup at FreeBSD.org>,
"Miller, Vincent
(Rick)" <vmiller at verisign.com>
Cc:
Subject: kern/172963: Kernel panic in udp_input()
Date: Wed, 24 Oct 2012 12:10:16 +0000
--_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Some of our developers spent a significant amount of time troubleshooting t=
his. The race condition has been identified. The relevant information is =
below:
It is a race condition between:
- udp_input()
- udp_detach()
- udp_pcblist()
How to reproduce:
- Call sysctl(" net.inet.udp.pcblist") continuously with:
$ while true; do sysctl -x net.inet.udp.pcblist > /dev/null; done;
- Open a bunch of UDP sockets (to slow down udp_pcblist() call):
$ for port in $(jot - 20000 24000 1); do socat -u -T 1 UDP4-LISTEN:$port,re=
useaddr GOPEN:/dev/null & done
- Launch a UDP server that close() after each request it received:
$ while true; do socat -u -T 0.0001 UDP4-LISTEN:12345,reuseaddr GOPEN:/dev/=
null; done
- Bombard this server with UDP request from another machine (could be also=
on the same machine):
$ while true; do socat -u EXEC:'/bin/echo' UDP4:10.51.33.40:12345 & sleep 0=
.0001; done
- Check these messages in /var/log/messages after 5/10 minutes:
Oct 24 09:05:21 flumpe4-qa2 kernel: udp_input(): Using freed inp 0xffffff0d=
f43d7930 inp->inp_refcount 1 inp->inp_ppcb 0
Oct 24 09:05:21 flumpe4-qa2 kernel: udp_pcblist(): Using freed inp 0xffffff=
0df43d7930 inp->inp_refcount 1 inp->inp_ppcb 0
(inp pointer being the same for udp_input() _and_ udp_pcblist(), _and_ inp=
->inp_ppcb being NULL)
Which is the proof that: udp_input() retrieves an inp pointer 0xffffff0df4=
3d7930 _and_ this inp has been released in_pcbrele() but not deleted becaus=
e someone had still a reference on it (flag INP_FREED introduced by debug p=
atch) _and_ udp_detach() has been called on this inp (inp->inp_ppcb being N=
ULL) _and_ this is udp_pcblist() that holds this reference on this inp.
Without the patch the crash will occur in udp_input():
/* Check inp state */
if ((inp->inp_flags2 & INP_FREED) && (inp->inp_socket =3D=3D NULL))=
{
log(LOG_INFO, "udp_input(): Using freed inp %p inp->inp_ref=
count %d inp->inp_ppcb %p\n",
inp, inp->inp_refcount, inp->inp_ppcb);
INP_RUNLOCK(inp);
goto badunlocked;
}
up =3D intoudpcb(inp); // intoudpcb(ip) being ((struct udpcb *)(ip)=
->inp_ppcb) and inp_ppcb being NULL...
if (up->u_tun_func =3D=3D NULL) { // Panic here in default kernel
--_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_
Content-Type: text/html; charset="us-ascii"
Content-ID: <6AB7A0C7A250C648A166B4B1E0BEA58E at verisign.com>
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
</head>
<body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin=
e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami=
ly: Calibri, sans-serif; ">
<div>
<div>
<div>Some of our developers spent a significant amount of time troubleshoot=
ing this. The race condition has been identified. The relevant =
information is below:</div>
<div><br>
</div>
<div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
It is a race condition between:</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- udp_input()</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- udp_detach()</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- udp_pcblist()</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
How to reproduce:</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- Call sysctl(" net.inet.udp.pcblist") continuou=
sly with:</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<font face=3D"Courier" style=3D"font-size: 12px; ">$ while true; do sysctl =
-x net.inet.udp.pcblist > /dev/null; done;</font></div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- Open a bunch of UDP sockets (to slow down udp_pcblist() call):</div=
>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<font face=3D"Courier" style=3D"font-size: 12px; ">$ for port in $(jot - 20=
000 24000 1); do socat -u -T 1 UDP4-LISTEN:$port,reuseaddr GOPEN:/dev/null =
& done</font></div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- Launch a UDP server that close() after each request it received:</d=
iv>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<font face=3D"Courier" style=3D"font-size: 12px; ">$ while true; do socat -=
u -T 0.0001 UDP4-LISTEN:12345,reuseaddr GOPEN:/dev/null; done</font></div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- Bombard this server with UDP request from another machine (could be=
also on the same machine):</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<font face=3D"Courier" style=3D"font-size: 12px; ">$ while true; do socat -=
u EXEC:'/bin/echo' UDP4:10.51.33.40:12345 & sleep 0.0001; done</font></=
div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
- Check these messages in /var/log/messages after 5/10 minutes:</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<div><font face=3D"Courier" style=3D"font-size: 12px; ">Oct 24 09:05:21 flu=
mpe4-qa2 kernel: udp_input(): Using freed inp 0xffffff0df43d7930 inp->in=
p_refcount 1 inp->inp_ppcb 0</font></div>
<div><font face=3D"Courier" style=3D"font-size: 12px; ">Oct 24 09:05:21 flu=
mpe4-qa2 kernel: udp_pcblist(): Using freed inp 0xffffff0df43d7930 inp->=
inp_refcount 1 inp->inp_ppcb 0</font></div>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<div><br>
</div>
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
(inp pointer being the same for udp_input() _and_ udp_pcblist(), _and=
_ inp->inp_ppcb being NULL)</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"font-family: Calibri; font-size: medium; "><font face=3D"Cali=
bri,sans-serif"> Which is the proof that: </font><font face=3D"Ca=
libri,sans-serif">udp_input() retrieves an inp pointer </font><fo=
nt face=3D"Calibri,sans-serif" style=3D"color: rgb(0, 0, 0); font-family: C=
alibri, sans-serif; font-size: 14px; ">0xffffff0df43d7930</font><font face=
=3D"Calibri,sans-serif" style=3D"color: rgb(0, 0, 0); font-family: Calibri,=
sans-serif; font-size: 14px; "> _and_ </font><font face=3D"Calib=
ri,sans-serif" style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-ser=
if; font-size: 14px; ">this
inp has been released in_pcbrele() but not deleted because someone had sti=
ll a reference on it (flag INP_FREED introduced by debug patch</f=
ont><font face=3D"Calibri,sans-serif" style=3D"color: rgb(0, 0, 0); font-fa=
mily: Calibri, sans-serif; font-size: 14px; ">)
_and_ udp</font><font face=3D"Calibri,sans-serif" style=3D"color: rgb(0, 0=
, 0); font-family: Calibri, sans-serif; font-size: 14px; ">_detach() has be=
en called on this inp (inp->inp_ppcb being NULL) _and_ this is udp_pcbli=
st() that holds this reference on this </font><font face=3D"Calibri,sa=
ns-serif" style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; f=
ont-size: 14px; ">inp</font><font face=3D"Calibri,sans-serif">.</font></div=
>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
Without the patch the crash will occur in udp_input():</div>
<div style=3D"color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-s=
ize: 14px; ">
<br>
</div>
<div style=3D"font-family: Calibri; font-size: medium; ">
<div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
ize: 12px; "> /* Check inp state */</font></div>
<div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
ize: 12px; "> if ((inp->inp_flags2 & INP_=
FREED) && (inp->inp_socket =3D=3D NULL)) {</font></div>
<div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
ize: 12px; "> log(LO=
G_INFO, "udp_input(): Using freed inp %p inp->inp_refcount %d inp-&=
gt;inp_ppcb %p\n",</font></div>
<div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
ize: 12px; "> =
inp, inp->inp_refcount, inp->inp_ppcb);</font><=
/div>
<div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
ize: 12px; "> INP_RU=
NLOCK(inp);</font></div>
<div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
ize: 12px; "> goto b=
adunlocked;</font></div>
<div style=3D"color: rgb(0, 0, 0); "><font face=3D"Courier" style=3D"font-s=
ize: 12px; "> }</font></div>
<div><font face=3D"Courier" style=3D"color: rgb(0, 0, 0); font-size: 12px; =
"> up =3D intoudpcb(inp); // </font><font f=
ace=3D"Courier"><span style=3D"font-size: 12px; ">intoudpcb(ip) being ((str=
uct udpcb *)(ip)->inp_ppcb) and </span></font><span style=3D"font-f=
amily: Courier; font-size: 12px; ">inp_ppcb
being NULL...</span></div>
<div>
<div><font face=3D"Courier"><span style=3D"font-size: 12px; "> =
if (up->u_tun_func =3D=3D NULL) { // Panic here in d=
efault kernel</span></font></div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
--_000_9E5449BF91F9ED478E519A7166D3A82A0D682CA0BRN1WNEXMBX01vc_--
More information about the freebsd-bugs
mailing list