netstat fails with memory allocation error and error in kvm_read

[Doug White]

On Mon, 13 Dec 2004, J. Martin Petersen wrote:

> > > We are trying to gather some debug information for a problem that is
> > > difficult to diagnose, as the machine always ends up hard frozen
> > > (does not do anything, can not break to debugger, does not respond
> > > to keyboard, etc.), so we are dumping output from netstat, vmstat,
> > > iostat etc. quite often.
> > >
> > > The cron jobs fail ever so often with error messages I do not quite
> > > understand, and I can not find anything relevant in the archives.
> > > Can anyone shed some light on this?
> > >
> > > Command:	netstat -r
> > > Error message: netstat: kvm_read: Bad address Debug before:
> > > http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.34.41.gz
> > > Debug after: http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.35.06.gz
> > > # errors:	7
> > >
> > > Command:	netstat -an
> > > Error message:netstat: sysctl: net.inet.udp.pcblist: Cannot allocate
> > > memory Debug before:
> > > http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.38.48.gz
> > > Debug after: http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.39.04.gz
> > > # errors:	3
> >
> > You appear to be running out of kernel memory. Since you're capturing
> > the output of vmstat -m, you should check that for any bins that are
> > growing at a high rate of speed.
> >
> > Seems possible that its in pf :)
>
> I've checked the numbers from just before the freeze (it's within 15 secs)
> with two sets of data: From a fresh boot and five minutes minutes before the
> freeze.

You might also log 'sysctl vm.kvm_free' and 'sysctl vm.zone'.


> Here are the stuff that changed significantly between the fresh boot and
> just before the freeze:
>
> Just before the freeze
> (http://www.aub.dk/~jmp/fw/tmp/debug-2004.12.11-22.59.01.gz):
>    AR driver     2     1K    268K  2922822  64,256,512,2048
>        kqueue     0     0K     38K 13304405  128,1024
>   UFS dirhash   444    88K    107K     2559
> 16,32,64,128,256,512,1024,2048,4096
>      freeblks    13     4K     29K   103030  256
>      freefrag     0     0K      1K   164217  32
>    allocindir   287    18K    162K  1966413  64
>      indirdep     1     1K    209K     9925  32
>   allocdirect    27     4K     18K   301048  128
>      inodedep    18   131K    150K   164032  128,256
>      routetbl   566    47K     67K   800649  16,32,64,128,256
>       subproc    99   301K    849K  1873146  32,4096
>
> Five minutes before the freeze
> (http://www.aub.dk/~jmp/fw/tmp/debug-2004.12.11-22.55.42.gz):
>      AR driver     2     1K    268K  2921793  64,256,512,2048
>        kqueue     0     0K     38K 13296556  128,1024
>   UFS dirhash   444    88K    107K     2559
> 16,32,64,128,256,512,1024,2048,4096
>      freeblks     1     1K     29K   102978  256
>      freefrag     0     0K      1K   164153  32
>    allocindir     0     0K    162K  1965284  64
>      indirdep     0     0K    209K     9921  32
>   allocdirect     1     1K     18K   300886  128
>      inodedep    14   130K    150K   163954  128,256
>      routetbl   562    46K     67K   800255  16,32,64,128,256
>       subproc    99   301K    849K  1872250  32,4096
>
> >From a fresh boot
> (http://www.aub.dk/~jmp/fw/tmp/debug-2004.12.11-23.31.31.gz):
>     AR driver     2     1K    190K    23450  64,256,512,2048
>        kqueue     0     0K      3K     1062  128,1024
>   UFS dirhash    36    13K     13K       42  16,32,512,2048,4096
>      freeblks   115    29K     29K      253  256
>      freefrag     0     0K      1K       51  32
>    allocindir     2     1K    135K     3332  64
>      indirdep    10     1K    173K      630  32
>   allocdirect     2     1K     40K      456  128
>      inodedep   137   145K    168K      506  128,256
>      routetbl   306    26K     27K      495  16,32,64,128,256
>       subproc   107   317K    466K     1554  32,4096
>
> The numbers for pflog and pf_if does not change at all. I checked vmstat -z,
> and the highest pf-related entries we're actually decreasing at the time of
> the deadlock, but I noticed the following:
> VM OBJECT:       132,        0,  31508,   2132, 14364021
> 128 Bucket:      524,        0,    727,      1,        0
> 64 Bucket:       268,        0,     23,     19,        0
> 32 Bucket:       140,        0,     34,     22,        0
> 16 Bucket:        76,        0,     15,     35,        0
>
> Can you or anyone else deduce anything from the numbers? If not, I'll whip
> something together that runs vmstat -m ever so often, parses the output and
> remove the non-increasing entries so it'll be easier to spot the trends.
>
> Thanks, Martin
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite at gumbysoft.com          |  www.FreeBSD.org