7.2-STABLE page fault with kernel from 12.01.2010 / crashinfo available

[John Baldwin]

On Thursday 21 January 2010 2:09:34 pm Florian Smeets wrote:
> On 1/21/10 8:05 PM, John Baldwin wrote:
> > On Thursday 21 January 2010 1:33:35 pm Florian Smeets wrote:
> >> On 1/21/10 6:58 PM, John Baldwin wrote:
> >>> On Thursday 21 January 2010 8:25:22 am Florian Smeets wrote:
> >>>> (kgdb) frame 8
> >>>> #8  0xc05f8b28 in ip_forward (m=0xc23dc900, srcrt=0) at
> >>>> /usr/src/sys/netinet/ip_input.c:1307
> >>>> 1307			m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t));
> >>>> (kgdb) l
> >>>> 1302			mcopy = NULL;
> >>>> 1303		}
> >>>> 1304		if (mcopy != NULL) {
> >>>> 1305			mcopy->m_len = min(ip->ip_len, M_TRAILINGSPACE(mcopy));
> >>>> 1306			mcopy->m_pkthdr.len = mcopy->m_len;
> >>>> 1307			m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t));
> >>>> 1308		}
> >>>> 1309	
> >>>> 1310	#ifdef IPSTEALTH
> >>>> 1311		if (!ipstealth) {
> >>>> (kgdb) p *m
> >>>> $1 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc271e80e
> >>>> "E\020", mh_len = 164, mh_flags = 3, mh_type = 1, pad = "\000"}, M_dat 
=
> >>>> {MH = {MH_pkthdr = {rcvif = 0xc20a4800, header = 0x0, len = 164,
> >>>> csum_flags = 3072,
> >>>>            csum_data = 65535, tso_segsz = 0, ether_vtag = 0, tags =
> >>>> {slh_first = 0xc35bc380}}, MH_dat = {MH_ext = {ext_buf = 0xc271e800 "",
> >>>> ext_free = 0, ext_args = 0x0, ext_size = 2048, ref_cnt = 0xc2703ab4,
> >>>> ext_type = 6},
> >>>>            MH_databuf =
> >>>> "\000?q?\000\000\000\000\000\000\000\000\000\b\000\000?:p?
> >>> \006\000\000\000dL?\t<+?\202\200\020
> >>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l?
\000\000\001%r???
> >>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b߼&?
> >>>
> > 
\237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Ljava/l\000\220\032Ae\207\000\002?
> >>> 36@\210d\021\000\001?
> > \001B\000!E\000\001 at bV\000\000 at 2\032$W\213\n\034"...}},
> >>>>
> >>>>        M_databuf =
> >>>> "\000H\n?\000\000\000\000?\000\000\000\000\f\000\000??
> >>> \000\000\000\000\000\000\200?[?\000?q?
> >>> \000\000\000\000\000\000\000\000\000\b\000\000?:p?\006\000\000\000dL?
\t<+?
> >>> \202\200\020
> >>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l?
\000\000\001%r???
> >>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b߼&?
> >>>
> > 
\237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Ljava/l\000\220\032Ae\207\000\002?
> >>> 3"...}}
> >>>
> >>> Ok, can you do 'p *m_copy'?
> >>>
> >>
> >> What ever you want :-)
> >>
> >> (kgdb) p *m_copy
> >> No symbol "m_copy" in current context.
> >> (kgdb) p *m_copydata
> >> $2 = {void (const struct mbuf *, int, int, caddr_t)} 
0xc0572e10<m_copydata>
> >> (kgdb) p *mcopy
> >> $1 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc23cce34
> >> "E\020", mh_len = 204, mh_flags = 2, mh_type = 1, pad = "\000"}, M_dat =
> >> {MH = {MH_pkthdr = {rcvif = 0xc20a4800, header = 0x0,
> >>           len = 204, csum_flags = 3072, csum_data = 65535, tso_segsz = 0,
> >> ether_vtag = 0, tags = {slh_first = 0xc23c3e00}}, MH_dat = {MH_ext =
> >> {ext_buf = 0x84001045<Address 0x84001045 out of bounds>,
> >
> > Hmm, ok.  Can you do 'p *ip'?  mcopy->m_len (204) is larger than m->m_len
> > (164).  That shouldn't be the case unless ip->ip_len is somehow larger 
than m-
> >> m_len.
> >
> 
> (kgdb) p *ip
> $3 = {ip_hl = 5, ip_v = 4, ip_tos = 16 '\020', ip_len = 33792, ip_id = 
> 61492, ip_off = 64, ip_ttl = 64 '@', ip_p = 6 '\006', ip_sum = 34849, 
> ip_src = {s_addr = 355576000}, ip_dst = {
>      s_addr = 2251401408}}

Looks like ip_len is in network byte order instead of host byte order and that 
is causing the problem.  33792 == 0x8400.  Swapping that gives 0x84 == 132 
which would be a reasonable length.  Are you using any firewall rules that 
would rewrite packets?  I wonder if you are having a packet rewritten and the 
new IP header is written in network byte order, but we swap the IP header len 
field to host byte order earlier in ip_input().  Luigi Rizzo may have some 
insight into this.

-- 
John Baldwin