[jcigar at ulb.ac.be: Listen queue overflow: 8 already in queue awaiting acceptance]

Julien Cigar jcigar at ulb.ac.be
Tue Oct 7 11:22:07 UTC 2014


On Fri, Oct 03, 2014 at 10:31:56AM -0700, hiren panchasara wrote:
> On Fri, Oct 3, 2014 at 1:01 AM, Julien Cigar <jcigar at ulb.ac.be> wrote:
> > On Thu, Oct 02, 2014 at 04:36:49PM -0700, hiren panchasara wrote:
> >> On Thu, Oct 2, 2014 at 11:19 AM, Julien Cigar <jcigar at ulb.ac.be> wrote:
> >> > On Thu, Oct 02, 2014 at 10:24:13AM -0700, hiren panchasara wrote:
> >> >> On Thu, Oct 2, 2014 at 9:42 AM, Julien Cigar <jcigar at ulb.ac.be> wrote:
> >> >> > sorry for cross-posting, I'm forwarding this as it seems that part of
> >> >> > the problem is also related to:
> >> >> > https://lists.freebsd.org/pipermail/freebsd-net/2014-September/039664.html
> >> >>
> >> >> Umm, this looks like a different problem than the subject of this email.
> >> >
> >> > yes and no, seems the same hardware (HP and igb) and I have also some
> >> > "requests for mbufs denied" (https://dpaste.de/t8kJ/raw) without any
> >> > reasons. I should add that the box hanged a week ago and I had to do a
> >> > hard reboot, I have the feeling that it's somewhat related to this
> >> > problem ..
> >> >
> >> I suggest you try to debug these 2 problems separately.  Did you get a
> >> chance to look at kgdb to find the culprit process as I suggested
> >> below?
> >
> > I tried what you suggested, but I get a "No struct type named inpcb"
> > Any idea ? :)
> 
> Is your kernel not build with debug symbols? Can you share your kernconf?

indeed.. I always disable debug symbols on the production machines
this is my kernconf: https://dpaste.de/8658/raw
maybe I should give 10.1-RELEASE a try now that it's out soon ..

> 
> I have following in my kernconf:
> 
> makeoptions     DEBUG=-g
> options         KDB
> options         KDB_TRACE
> options         DDB
> 
> cheers,
> Hiren
> 
> >>
> >> cheers,
> >> Hiren
> >> >> >
> >> >> > I also wonder if something has been fixed in -STABLE in this area ..
> >> >> >
> >> >> > (please keep me in CC as I'm not subscribed on freebsd-net@ an
> >> >> > freebsd-stable@)
> >> >> >
> >> >> > --
> >> >> > Julien Cigar
> >> >> > Belgian Biodiversity Platform (http://www.biodiversity.be)
> >> >> > PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> >> >> > No trees were killed in the creation of this message.
> >> >> > However, many electrons were terribly inconvenienced.
> >> >> >
> >> >> >
> >> >> > ---------- Forwarded message ----------
> >> >> > From: Julien Cigar <jcigar at ulb.ac.be>
> >> >> > To: freebsd-questions at freebsd.org
> >> >> > Cc:
> >> >> > Date: Thu, 2 Oct 2014 11:52:06 +0200
> >> >> > Subject: Listen queue overflow: 8 already in queue awaiting acceptance
> >> >> > Hello,
> >> >> >
> >> >> > I'm running 10-RELEASE on a HP Proliant DL160 Gen8 and I'm seeing the
> >> >> > following in my kernel logs:
> >> >> > sonewconn: pcb 0xfffff8010e561310: Listen queue overflow: 8 already in
> >> >> > queue awaiting acceptance
> >> >>
> >> >> This usually means the application is not keeping up with the incoming
> >> >> connections.
> >> >> >
> >> >> > I already raised kern.ipc.soacceptqueue to 1024 and  netstat -naA | grep
> >> >> > "fffff8010e561310" returns nothing
> >> >>
> >> >> This is the usual way of finding the culprit process. If this doesn't
> >> >> return anything, it probably means that it is a short-lived process.
> >> >>
> >> >> Here is an example of what you could do:
> >> >>
> >> >> sonewconn: pcb 0xfffff8008f40cb10: Listen queue overflow: 1 already in queue
> >> >> awaiting acceptance
> >> >>
> >> >> From kgdb,
> >> >> (kgdb) p ((struct inpcb *)0xfffff8008f40cb10)->inp_inc
> >> >> $3 = {inc_flags = 0 '\0', inc_len = 0 '\0', inc_fibnum = 0, inc_ie = {ie_fport
> >> >> = 0, ie_lport = 10295, ie_dependfaddr = {
> >> >>       ie46_foreign = {ia46_pad32 = {0, 0, 0}, ia46_addr4 = {s_addr = 0}},
> >> >> ie6_foreign = {__u6_addr = {
> >> >>           __u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0,
> >> >> 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}},
> >> >>     ie_dependladdr = {ie46_local = {ia46_pad32 = {0, 0, 0}, ia46_addr4 =
> >> >> {s_addr = 0}}, ie6_local = {__u6_addr = {
> >> >>           __u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0,
> >> >> 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}}}}
> >> >>
> >> >> Here, ie_lport = 10295 which is in n/w byte order and converting it to host
> >> >> byte order, 10295 -> 0x2837 and swapping them gives us 0x3728 which is 14120.
> >> >>
> >> >> Now, use sockstat to find out what process is on that port:
> >> >>
> >> >> $ sockstat -l | grep 14120
> >> >>
> >> >> cheers,
> >> >> Hiren
> >> >
> >> > --
> >> > Julien Cigar
> >> > Belgian Biodiversity Platform (http://www.biodiversity.be)
> >> > PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> >> > No trees were killed in the creation of this message.
> >> > However, many electrons were terribly inconvenienced.
> >
> > --
> > Julien Cigar
> > Belgian Biodiversity Platform (http://www.biodiversity.be)
> > PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> > No trees were killed in the creation of this message.
> > However, many electrons were terribly inconvenienced.

-- 
Julien Cigar
Belgian Biodiversity Platform (http://www.biodiversity.be)
PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20141007/0fdf1c81/attachment.sig>


More information about the freebsd-stable mailing list