7.2-STABLE page fault with kernel from 12.01.2010 / crashinfo available

Fri Jan 22 14:32:06 UTC 2010

On Friday 22 January 2010 3:08:45 am Florian Smeets wrote:
> On 1/21/10 9:15 PM, John Baldwin wrote:
> > On Thursday 21 January 2010 2:09:34 pm Florian Smeets wrote:
> >> On 1/21/10 8:05 PM, John Baldwin wrote:
> >>> On Thursday 21 January 2010 1:33:35 pm Florian Smeets wrote:
> >>>> On 1/21/10 6:58 PM, John Baldwin wrote:
> >>>>> On Thursday 21 January 2010 8:25:22 am Florian Smeets wrote:
> >>>>>> (kgdb) frame 8
> >>>>>> #8  0xc05f8b28 in ip_forward (m=0xc23dc900, srcrt=0) at
> >>>>>> /usr/src/sys/netinet/ip_input.c:1307
> >>>>>> 1307			m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t));
> >>>>>> (kgdb) l
> >>>>>> 1302			mcopy = NULL;
> >>>>>> 1303		}
> >>>>>> 1304		if (mcopy != NULL) {
> >>>>>> 1305			mcopy->m_len = min(ip->ip_len, M_TRAILINGSPACE(mcopy));
> >>>>>> 1306			mcopy->m_pkthdr.len = mcopy->m_len;
> >>>>>> 1307			m_copydata(m, 0, mcopy->m_len, mtod(mcopy, caddr_t));
> >>>>>> 1308		}
> >>>>>> 1309	
> >>>>>> 1310	#ifdef IPSTEALTH
> >>>>>> 1311		if (!ipstealth) {
> >>>>>> (kgdb) p *m
> >>>>>> $1 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc271e80e
> >>>>>> "E\020", mh_len = 164, mh_flags = 3, mh_type = 1, pad = "\000"}, M_dat
> > =
> >>>>>> {MH = {MH_pkthdr = {rcvif = 0xc20a4800, header = 0x0, len = 164,
> >>>>>> csum_flags = 3072,
> >>>>>>             csum_data = 65535, tso_segsz = 0, ether_vtag = 0, tags =
> >>>>>> {slh_first = 0xc35bc380}}, MH_dat = {MH_ext = {ext_buf = 0xc271e800 "",
> >>>>>> ext_free = 0, ext_args = 0x0, ext_size = 2048, ref_cnt = 0xc2703ab4,
> >>>>>> ext_type = 6},
> >>>>>>             MH_databuf =
> >>>>>> "\000?q?\000\000\000\000\000\000\000\000\000\b\000\000?:p?
> >>>>> \006\000\000\000dL?\t<+?\202\200\020
> >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l?
> > \000\000\001%r???
> >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b߼&?
> >>>>>
> >>>
> > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Ljava/l\000\220\032Ae\207\000\002?
> >>>>> 36@\210d\021\000\001?
> >>> \001B\000!E\000\001 at bV\000\000 at 2\032$W\213\n\034"...}},
> >>>>>>
> >>>>>>         M_databuf =
> >>>>>> "\000H\n?\000\000\000\000?\000\000\000\000\f\000\000??
> >>>>> \000\000\000\000\000\000\200?[?\000?q?
> >>>>> \000\000\000\000\000\000\000\000\000\b\000\000?:p?\006\000\000\000dL?
> > \t<+?
> >>>>> \202\200\020
> >>>>>> O/\207\000\000\001\001\b\n-?b\230qms?\000\000\004\001?l?
> > \000\000\001%r???
> >>>>> \200\000????\034?Ot?\b?{sr\000\034org.jboss.mq.ConnectionToken?\b߼&?
> >>>>>
> >>>
> > \237N\002\000\005I\000\004hashZ\000\asameJVML\000\bclientIDt\000\022Ljava/l\000\220\032Ae\207\000\002?
> >>>>> 3"...}}
> >>>>>
> >>>>> Ok, can you do 'p *m_copy'?
> >>>>>
> >>>>
> >>>> What ever you want :-)
> >>>>
> >>>> (kgdb) p *m_copy
> >>>> No symbol "m_copy" in current context.
> >>>> (kgdb) p *m_copydata
> >>>> $2 = {void (const struct mbuf *, int, int, caddr_t)}
> > 0xc0572e10<m_copydata>
> >>>> (kgdb) p *mcopy
> >>>> $1 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc23cce34
> >>>> "E\020", mh_len = 204, mh_flags = 2, mh_type = 1, pad = "\000"}, M_dat =
> >>>> {MH = {MH_pkthdr = {rcvif = 0xc20a4800, header = 0x0,
> >>>>            len = 204, csum_flags = 3072, csum_data = 65535, tso_segsz = 0,
> >>>> ether_vtag = 0, tags = {slh_first = 0xc23c3e00}}, MH_dat = {MH_ext =
> >>>> {ext_buf = 0x84001045<Address 0x84001045 out of bounds>,
> >>>
> >>> Hmm, ok.  Can you do 'p *ip'?  mcopy->m_len (204) is larger than m->m_len
> >>> (164).  That shouldn't be the case unless ip->ip_len is somehow larger
> > than m-
> >>>> m_len.
> >>>
> >>
> >> (kgdb) p *ip
> >> $3 = {ip_hl = 5, ip_v = 4, ip_tos = 16 '\020', ip_len = 33792, ip_id =
> >> 61492, ip_off = 64, ip_ttl = 64 '@', ip_p = 6 '\006', ip_sum = 34849,
> >> ip_src = {s_addr = 355576000}, ip_dst = {
> >>       s_addr = 2251401408}}
> >
> > Looks like ip_len is in network byte order instead of host byte order and that
> > is causing the problem.  33792 == 0x8400.  Swapping that gives 0x84 == 132
> > which would be a reasonable length.  Are you using any firewall rules that
> > would rewrite packets?  I wonder if you are having a packet rewritten and the
> > new IP header is written in network byte order, but we swap the IP header len
> > field to host byte order earlier in ip_input().  Luigi Rizzo may have some
> > insight into this.
> >
> 
> Well, when looking at MH_databuf i see Jboss MQ traffic that would mean 
> that this traffic was coming from or going to an IPsec tunnel, i could 
> say for sure when i would have a clue how to get an IP address from 
> something like ip_src = {s_addr = 355576000}.

Something like this should show you the IP:

(gdb) set $i = 355576000
(gdb) printf "%d.%d.%d.%d\n", $i >> 24, $i >> 16 & 0xff, $i >> 8 & 0xff, $i & 0xff
21.49.168.192

In this case I probably printed it backwards, so it is probably
192.168.49.21.

> If it really is IPsec traffic then there are no rewrite rules only 10 pf 
> pass rules on the enc0 interface and a "scrub in all" rule.
> 
> Perhaps it matters that i have these set:
> 
> net.enc.out.ipsec_bpf_mask=0x00000001
> net.enc.out.ipsec_filter_mask=0x00000001
> net.enc.in.ipsec_bpf_mask=0x00000002
> net.enc.in.ipsec_filter_mask=0x00000002
> 
> so that i can filter the "encapsulated" traffic.

I have no idea, I've cc'd mlaier@ (pf) and bz@ (ipsec) to see if they have
any ideas.

-- 
John Baldwin