help w/panic under heavy load - 5.4

[Max Laier]

On Saturday 23 July 2005 20:41, Edwin wrote:
> Kernel name: D1-0722 (for reference)
>
> mbsd05#	kgdb kernel.debug /usr/local/STORAGE/crash/vmcore.5
> #13 0xc06933c1 in ip_fastforward (m=0xc12e6c00) at
> /usr/src/sys/netinet/ip_fastfwd.c:572 warning: Source file is more recent
> than executable.

Let's hope that's still correct ...

> 572				if (ip_fragment(ip, &m, mtu, ifp->if_hwassist,
> (kgdb) l
> 567				m->m_pkthdr.csum_flags |= CSUM_IP;
> 568				/*
> 569				 * ip_fragment expects ip_len and ip_off in host byte
> 570				 * order but returns all packets in network byte order
> 571				 */
> 572				if (ip_fragment(ip, &m, mtu, ifp->if_hwassist,
> 573						(~ifp->if_hwassist & CSUM_DELAY_IP))) {
> 574					goto drop;
> 575				}
> 576				KASSERT(m != NULL, ("null mbuf and no error"));
> (kgdb) i loc
> ip = (struct ip *) 0xc12f700e
> m0 = (struct mbuf *) 0xc12f700e
> ro = {ro_rt = 0xc11f8420, ro_dst = {sa_len = 16 '\020', sa_family = 2
> '\002', sa_data = "\000\000ˬ\002\005\000\000\000\000\000\000\000"}}
> dst = (struct sockaddr_in *) 0xc76bfc3c
> ia = (struct in_ifaddr *) 0x0
> ifa = (struct ifaddr *) 0x0
> ifp = (struct ifnet *) 0xc0f91800
> odest = {s_addr = 84060352}
> dest = {s_addr = 84060352}
> sum = 0
> ip_len = 0

This should not happen. ip_len is initialize from ntohs(ip->ip_len) and never 
touched again.  Anyway, let's look some more ...

> error = 84060352
> hlen = -1057417216
> mtu = 0
> __func__ = "ip_fastforward"
> (kgdb) p *ip
> $1 = {ip_hl = 5, ip_v = 4, ip_tos = 0 '\0', ip_len = 10240, ip_id = 61249,

ip_len should be 40 as ip_len is supposed to be in HOST BYTE ORDER at this 
point.  Feeding 10240 to ntohs() give the correct value, so something 
obviously went wrong.

Let's see how we got here:
355 does the byteorder flip to host byte order
366 pfil OUT
451 pfil IN
527 first check ip_len < if_mtu etc ...

Obviously, the only thing that might mess with the byte order (unless I missed 
something along the way) is one of the pfil consumers.

***
*** What firewall(s) are you running with?
***

> ip_off = 0, ip_ttl = 63 '?', ip_p = 17 '\021', ip_sum = 31921, ip_src =
> {s_addr = 67479744}, ip_dst = {s_addr = 84060352}} (kgdb) p *m
> $2 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc12f700e "E",
> mh_len = 40, mh_flags = 3, mh_type = 1}, M_dat = {MH = {MH_pkthdr = {rcvif
> = 0xc0f90000, len = 40, header = 0x0, csum_flags = 769, csum_data = 0, tags

40, there you have it - no need to fragment at all!

> /usr/src/sys/netinet/ip_output.c:967
> 967			m->m_next = m_copy(m0, off, len);
> (kgdb) l
> 962				len = ip->ip_len - off;
> 963				m->m_flags |= M_LASTFRAG;
> 964			} else
> 965				mhip->ip_off |= IP_MF;
> 966			mhip->ip_len = htons((u_short)(len + mhlen));
> 967			m->m_next = m_copy(m0, off, len);
> 968			if (m->m_next == NULL) {	/* copy failed */
> 969				m_free(m);
> 970				error = ENOBUFS;	/* ??? */
> 971				ipstat.ips_odropped++;

Just to make sure, we didn't touch the original packet at this point so the 
above values are still the ones we based the (wrong) decision on.

-- 
/"\  Best regards,                      | mlaier at freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier at EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20050724/71a40944/attachment.bin