serious packet routing issue causing ntpd high load?

[Li, Qing]

Hi,

>
> RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks:  inits:
> sockaddrs: <DST>
>  ::A.B.C.D
>

Would it be possible for you to email me what exactly does "::A.B.C.D" 
map into WRT your system or infrastructure ?

And are you able to share your "ifconfig -a" and "netstat -rn" output
with me privately ?

--Qing


> -----Original Message-----
> From: owner-freebsd-stable at freebsd.org [mailto:owner-freebsd-
> stable at freebsd.org] On Behalf Of Steven Hartland
> Sent: Tuesday, October 04, 2011 5:00 PM
> To: freebsd-stable at freebsd.org
> Cc: liv3d at multiplay.co.uk
> Subject: serious packet routing issue causing ntpd high load?
> 
> We just updated a machine to 8-STABLE and I've noticed
> that ntpd is using notible amounts of CPU 5-7% which
> is very high for such a trivial daemon.
> 
> 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct  4 09:53:17 UTC 2011
> 
> truss indicates its constantly checking and reading
> from a socket
> 
> 0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1
> (0x1)
> 0.047513160 clock_gettime(0,{1317770389.969538247 }) = 0 (0x0)
> 0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 })
> = 1 (0x1)
> 0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184
> (0xb8)
> 0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1
> (0x1)
> 0.049503689 clock_gettime(0,{1317770389.971526820 }) = 0 (0x0)
> 0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 })
> = 1 (0x1)
> 0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184
> (0xb8)
> 0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1
> (0x1)
> ...
> 
> running with debug enabled it sits looping outputting:-
> routing message op = 7: ignored
> routing message op = 7: ignored
> routing message op = 7: ignored
> routing message op = 7: ignored
> routing message op = 7: ignored
> routing message op = 7: ignored
> routing message op = 7: ignored
> ...
> 
> It seems socket 28 is a duplicate of an internal routing socket
> as seen here in the trace:-
> 0.044544269 socket(PF_ROUTE,SOCK_RAW,0)      = 4 (0x4)
> 0.044595394 fcntl(4,F_DUPFD,0x14)        = 28 (0x1c)
> 0.044645960 close(4)                 = 0 (0x0)
> 0.044695968 fcntl(28,F_SETFL,O_NONBLOCK)     = 0 (0x0)
> 
> Now this looks like its RTM_MISS as defined:-
> sys/net/route.h:#define RTM_MISS        0x7     /* Lookup failed on
> this address */
> 
> So the question was why is PF_ROUTE socket constantly
> spamming RTM_MISS?
> 
> route -n monitor on this machines shows:-
> got message of size 184 on Tue Oct  4 23:46:36 2011
> RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks:  inits:
> sockaddrs: <DST>
>  ::A.B.C.D
> 
> got message of size 184 on Tue Oct  4 23:46:36 2011
> RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks:  inits:
> sockaddrs: <DST>
>  ::A.B.C.D
> 
> This seems very much like the following pr which was fixed:-
> "Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029,
> generating an RTM_MISS for every IP packet forwarded making user space
> routing daemons unhappy":-
> http://www.freebsd.org/cgi/query-pr.cgi?pr=124540
> 
> The box is doing no routing, its fairly basic install with
> 1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1.
> 
> Its running mysql and thats about it.
> 
> Any ideas?
> 
>     Regards
>     Steve
> 
> 
> 
> ================================================
> This e.mail is private and confidential between Multiplay (UK) Ltd. and
> the person or entity to whom it is addressed. In the event of
> misdirection, the recipient is prohibited from using, copying, printing
> or otherwise disseminating it or any information contained in it.
> 
> In the event of misdirection, illegible or incomplete transmission
> please telephone +44 845 868 1337
> or return the E.mail to postmaster at multiplay.co.uk.
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-
> unsubscribe at freebsd.org"