panic: kmem_map too small at heavy packet traffic

[Tugrul Erdogan]

Specifically, I am taking this panic when doing ip spoof attack while
syn-proxy activated. The output of system arguments below:

kern.malloc_count: 315
vm.md_malloc_wait: 0
vfs.bufmallocspace: 0
vfs.maxmallocbufspace: 86269952
vm.kmem_size: 16686845952
vm.kmem_size_min: 0
vm.kmem_size_max: 329853485875
vm.kmem_size_scale: 1
vm.kmem_map_size: 543973376
vm.kmem_map_free: 15974895616
kern.maxvnodes: 350097
kern.minvnodes: 87524
vfs.numvnodes: 112329
vfs.wantfreevnodes: 87524
vfs.freevnodes: 87502

[root@ ~]# pfctl -si
No ALTQ support in kernel
ALTQ related functions disabled
Status: Enabled for 0 days 00:17:39           Debug: Urgent

State Table                          Total             Rate
  current entries                  5142886
  searches                        26982141        25478.9/s
  inserts                         29055053        27436.3/s
  removals                        24218654        22869.4/s
Counters
  match                           24901305        23514.0/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                             18            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                        29378439        27741.7/s


[root@~]# panic: kmem_malloc(-1 814 425 600): kmem_map too small: 543 956 992 to
tal allocated
cpuid = 8
Uptime: 1d18h2m14s
(ada0:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada0:ahcich1:0:0:0): CAM status: CCB request is in progress
(ada0:ahcich1:0:0:0): Error 5, Retries exhausted
(ada0:ahcich1:0:0:0): Synchronize cache failed
(ada1:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada1:ahcich2:0:0:0): CAM status: CCB request is in progress
(ada1:ahcich2:0:0:0): Error 5, Retries exhausted
(ada1:ahcich2:0:0:0): Synchronize cache failed
Dumping 8243 out of 16357 MB:..1%..11%


On Thu, Jul 25, 2013 at 5:57 PM, Tugrul Erdogan
<h.tugrul.erdogan at gmail.com>wrote:

> howdy all,
>
> At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB
> ram. I am taking
>
> "panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated"
>
> message with configuration below:
>
> [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale
>
> vm.kmem_size_min: 0
> vm.kmem_size_max: 329853485875
> vm.kmem_size: 16686845952
> vm.kmem_size_scale: 1
> [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem
> hw.physmem: 17151787008
> hw.usermem: 8282652672
>
> hw.realmem: 18253611008
> [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages
> hw.pagesize: 4096
> hw.pagesizes: 4096 2097152 0
> hw.availpages: 4187448
>
>
> When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at:
>
> pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128
>
> and after the panic at the core dump file the major vmstat difference is:
>
> temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2
>
> When I explore the source code of kernel (at vm_kern.c and vm_map.c), I
> see that the panic can occur with the cases at below:
>
> * negative malloc size parameter
>
> * longer than free buffer respect to kmem_map min_offset and max_offset
> values
>
> * try to allocate when the root entry of map is the rightmost entry of map
>
> * try to allocate bigger than map's max_free value
>
> I think the panic occurs at mbuf creation process when calling malloc() as
> a result of couldn't be able to allocate memory; but I don't understand why
> one of this panic case activating? The memory is almost empty but the
> device is saying kmem_map small when using about 0.5GB memory purely. How
> can i solve this panic problem?
>
> Thank you all for your time.
>
> -- Best Wishes,
>
> Tugrul Erdogan
>
>