kern/180844: [panic] [re] Intermittent panic (re driver?)

Alex alex323 at gmail.com
Thu Jul 25 15:20:00 UTC 2013


>Number:         180844
>Category:       kern
>Synopsis:       [panic] [re] Intermittent panic (re driver?)
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jul 25 15:20:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Alex
>Release:        10.0-CURRENT
>Organization:
>Environment:
FreeBSD poseidon 10.0-CURRENT FreeBSD 10.0-CURRENT #18 r253581: Tue Jul 23 17:41:53 EDT 2013     alex at poseidon:/usr/obj/usr/src/sys/POSEIDON  amd64
>Description:
I just recently switched from -STABLE to -CURRENT and have been
experiencing intermittent crashes almost daily. Based on the stack
trace, I suspect that the problem lies in the re driver:

#10 0xffffffff803ddb00 in re_txeof (sc=0xffffff8000c53000)
    at /usr/src/sys/dev/re/if_re.c:2388

I have attached the crash report to this PR.
>How-To-Repeat:

>Fix:


Patch attached with submission follows:

poseidon dumped core - see /archive/crash/vmcore.0

Thu Jul 25 05:49:06 EDT 2013

FreeBSD poseidon 10.0-CURRENT FreeBSD 10.0-CURRENT #18 r253581: Tue Jul 23 17:41:53 EDT 2013     alex at poseidon:/usr/obj/usr/src/sys/POSEIDON  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address	= 0x28
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff807ac4e7
stack pointer	        = 0x28:0xffffff8000346840
frame pointer	        = 0x28:0xffffff8000346850
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi5: fast taskq)
trap number		= 12
panic: page fault
cpuid = 4
Uptime: 1d9h18m37s
Dumping 4132 out of 16335 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:236
236	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:236
#1  0xffffffff805950f0 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff80595475 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff8088d96a in trap_fatal (frame=<value optimized out>, 
    eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:873
#4  0xffffffff8088dcff in trap_pfault (frame=0x0, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:731
#5  0xffffffff8088d368 in trap (frame=0xffffff8000346790)
    at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff80877dd3 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff807ac4e7 in pf_mtag_free (t=0xfffffe034e9b63c0)
    at /usr/src/sys/netpfil/pf/pf.c:830
#8  0xffffffff80851750 in uma_zfree_arg (zone=0xfffffe043f7b5000, 
    item=0xfffffe015a42ac00, udata=0x0) at /usr/src/sys/vm/uma_core.c:2561
#9  0xffffffff805f6705 in m_freem (mb=<value optimized out>) at uma.h:364
#10 0xffffffff803ddb00 in re_txeof (sc=0xffffff8000c53000)
    at /usr/src/sys/dev/re/if_re.c:2388
#11 0xffffffff803df622 in re_int_task (arg=0xffffff8000c53000, npending=0)
    at /usr/src/sys/dev/re/if_re.c:2566
#12 0xffffffff805d7b66 in taskqueue_run_locked (queue=0xfffffe0028e1a000)
    at /usr/src/sys/kern/subr_taskqueue.c:333
#13 0xffffffff805d7a21 in taskqueue_run (queue=0xfffffe0028e1a000)
    at /usr/src/sys/kern/subr_taskqueue.c:347
#14 0xffffffff8056af6b in intr_event_execute_handlers (
    p=<value optimized out>, ie=0xfffffe0028e19e00)
    at /usr/src/sys/kern/kern_intr.c:1263
#15 0xffffffff8056b3b6 in ithread_loop (arg=0xfffffe0028e10ac0)
    at /usr/src/sys/kern/kern_intr.c:1276
#16 0xffffffff80568bf5 in fork_exit (
    callout=0xffffffff8056b320 <ithread_loop>, arg=0xfffffe0028e10ac0, 
    frame=0xffffff8000346ac0) at /usr/src/sys/kern/kern_fork.c:991
#17 0xffffffff8087830e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:606
#18 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) 

------------------------------------------------------------------------
ps -axl

UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT  TIME COMMAND

------------------------------------------------------------------------
vmstat -s

534062163 cpu context switches
 50297213 device interrupts
 33805915 software interrupts
207077764 traps
4113747409 system calls
       22 kernel threads created
    22831  fork() calls
    22189 vfork() calls
        0 rfork() calls
   194640 swap pager pageins
  1363806 swap pager pages paged in
   232424 swap pager pageouts
  2625877 swap pager pages paged out
   149218 vnode pager pageins
  2624383 vnode pager pages paged in
   140734 vnode pager pageouts
  2125039 vnode pager pages paged out
      634 page daemon wakeups
        0 pages examined by the page daemon
   177225 pages reactivated
  1838832 copy-on-write faults
     2476 copy-on-write optimized faults
 51545593 zero fill pages zeroed
     6741 zero fill pages prezeroed
   242183 intransit blocking page faults
 62035031 total VM faults taken
  1470132 page faults requiring I/O
        0 pages affected by kernel thread creation
  1004633 pages affected by  fork()
   941121 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
 86909444 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
   262673 pages active
   741498 pages inactive
    31838 pages in VM cache
  2187852 pages wired down
   835578 pages free
     4096 bytes per page
 75753374 total name lookups
          cache hits (87% pos + 6% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
         cdev     8     2K       -        8  256
     filedesc 23146  1833K       -    91799  16,32,64,128,256,512,1024,2048,4096
        sigio     2     1K       -        3  64
     filecaps     0     0K       -       44  64
         kenv    90    12K       -      107  16,32,64,128
       kqueue    39   119K       -    36417  256,512,2048,4096
    proc-args    87     8K       -    59352  16,32,64,128,256
        hhook     2     1K       -        2  256
      ithread   124    20K       -      124  32,128,256
       KTRACE   100    13K       -      100  128
       linker   132    12K       -      146  16,32,64,128,512
        lockf   193    24K       -   640985  64,128
   loginclass     2     1K       -     1334  64
       devbuf 36741 71111K       -    51746  16,32,64,128,256,512,1024,2048,4096
         temp    81    15K       -  1133781  16,32,64,128,256,512,1024,2048,4096
       ip6opt     1     1K       -       10  256
       ip6ndp    13     1K       -       24  64,128
       module   271    34K       -      271  128
     mtx_pool     2    16K       -        2  
          osd    28     1K       -     1022  16,32,64,128
     pmchooks     1     1K       -        1  128
         pgrp    51     7K       -     2369  128
      session    46     6K       -     1477  128
         proc     2    64K       -        2  
      subproc   291   538K       -    45223  512,4096
         cred   116    19K       -  1503268  64,256
       plimit    16     4K       -    18153  256
      uidinfo    10    10K       -      697  128
       sysctl     0     0K       -    24168  16,32,64
    sysctloid  8135   405K       -     8490  16,32,64,128
    sysctltmp     0     0K       -    27005  16,32,64,128,4096
      tidhash     1    64K       -        1  
      callout     9  3208K       -        9  
         umtx  1800   225K       -     1800  128
     p1003.1b     1     1K       -        1  16
         SWAP     2  1097K       -        2  64
          bus   957    91K       -     8708  16,32,64,128,256,1024
       bus-sc   139  2868K       -     1657  16,32,64,128,256,512,1024,2048,4096
      devstat    14    29K       -       14  32,4096
 eventhandler    96     8K       -       96  64,128
         kobj   163   652K       -      476  4096
      Per-cpu     1     1K       -        1  32
  nullfs_hash     1  2048K       -        1  
    pfs_nodes    73    19K       -       73  256
         rman   266    30K       -      677  16,32,128
         sbuf     1     4K       -     6386  16,32,64,128,256,512,1024,2048,4096
       sglist    36    57K       -     1974  32,64,128,512,1024,2048,4096
        stack     0     0K       -       12  256
    taskqueue   123    20K       -      183  16,32,64,256,1024
       Unitno    29     2K       -   158427  32,64
         vmem     1     2K       -        7  256,512,1024,2048,4096
     ioctlops     0     0K       - 2607132327  16,32,64,128,256,512,1024,2048,4096
       select   341    43K       -      341  128
          iov     0     0K       - 41783668  16,32,64,128,256,512
          msg     4    30K       -        4  2048,4096
          sem     4   106K       -        4  2048,4096
          shm    18   376K       -      413  
          tty    26    26K       -       56  1024,2048,4096
          pts     8     2K       -       24  256
     mbuf_tag     0     0K       -   637525  32,128
        shmfd     1     8K       -        1  
       soname   161    19K       - 182301116  16,32,64,128
          pcb   139  2201K       -   154166  16,32,64,128,1024,2048
         UART     3     3K       -        3  16,1024
          acl     0     0K       -      365  4096
     vfscache     1  4096K       -        1  
   cl_savebuf     0     0K       -       38  64
     vfs_hash     1  2048K       -        1  
       vnodes     2     1K       -        3  32,256
        mount   211     7K       -      510  16,32,64,128,256
  vnodemarker     0     0K       -   153206  512
      fadvise     0     0K       -       59  64
          BPF    26  1052K       -      423  16,32,128,256,512,2048,4096
        ifnet    11    21K       -       14  128,2048
       ifaddr   478    40K       -      507  16,32,64,128,256,512,2048,4096
  ether_multi    62     4K       -      106  16,32,64
      CAM SIM    16     4K       -       16  256
         GEOM   138    25K       -     1832  16,32,64,128,256,512,1024,2048
      entropy  1024    64K       -     1024  64
     acpitask     1     8K       -        1  
        clone    14     2K       -       14  128
       arpcom     5     1K       -        5  16
          gif     1     1K       -        1  256
      lltable    28    12K       -       74  256,512
          tun     0     0K       -        3  256
          tap     1     1K       -        1  256
      acpisem    21     3K       -       21  128
      CAM XPT    61     6K       -      309  16,32,64,128,256,1024
         hdaa    22    54K       -       22  128,256,512,1024,2048,4096
         hdac     2     2K       -        2  512,1024
        hdacc     5     1K       -        5  32
      acpidev    40     3K       -       40  64
     routetbl    55     8K       -    50578  32,64,128,256,512
         vnet     1     1K       -        1  64
    vnet_data     1    36K       -        1  
vnet_data_free     1     1K       -        1  32
     80211vap     1     4K       -        1  4096
  80211crypto     2     1K       -        2  512
     80211com     1     8K       -        1  
    80211node     3    42K       -       12  16,1024
  80211nodeie     2     1K       -        2  32,512
   80211power     1     1K       -        1  16
    80211scan     1     4K       -        1  4096
         igmp    10     3K       -       13  256
         ipid     2    24K       -        2  
     in_multi     4     1K       -       12  256
      CAM DEV    22    44K       -       41  2048
      CAM CCB    46    92K       -      118  2048
          USB    82   109K       -      105  16,32,128,256,512,1024,2048,4096
encap_export_host     3     3K       -        4  1024
    sctp_a_it     0     0K       -       22  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa     8     1K       -       18  128
     sctp_ifn     4     1K       -       11  128
    sctp_iter     0     0K       -       22  256
    hostcache     1    28K       -        1  
     syncache     1    64K       -        1  
    in6_multi    41     5K       -       58  32,256
       USBdev   120    41K       -    29495  32,64,128,512,4096
    raid_data     0     0K       -      180  32,128,256
          mld    10     2K       -       13  128
       athdev    16   428K       -       16  64,128,1024,2048
      ath_hal     2    64K       -        2  
  inpcbpolicy   533    17K       -   382549  32
  ipsecpolicy  1066   267K       -   939916  256
    ipsec-reg     0     0K       -        2  32
       pfsync     1     1K       -        1  1024
      pf_temp     0     0K       -        1  128
      pf_hash     3  2880K       -        3  
     pf_ifnet    22     8K       -      270  256,2048
      pf_rule    21    21K       -      287  128,1024
      pf_osfp  1130   115K       -    23730  64,128
     pf_table     1     2K       -       24  2048
      NFS FHA     1     2K       -        1  2048
       crypto     1     1K       -        1  512
        xform     0     0K       -     4262  16,32
          rpc     2     1K       -        2  256
audit_evclass   188     6K       -      229  32
      pagedep     1   512K       -        1  
     inodedep     1  2048K       -        1  
    bmsafemap     1     8K       -        1  
       newblk     1   512K       -        1  
     freework     1     1K       -        1  16
    ufs_mount     3    17K       -        3  512,4096
    vm_pgdata     2  2049K       -        2  128
      UMAHash     5    17K       -       16  512,1024,2048,4096
       kbdmux     8    18K       -        8  16,512,1024,2048
          LED    24     2K       -       24  16,128
       feeder    50     6K       -    14386  32,128
     CAM path    26     1K       -       86  32
   CAM periph    12     3K       -       33  16,32,64,256
        mixer     9    36K       -        9  4096
      memdesc     1     4K       -        2  4096
    CAM queue    45     4K       -      147  16,32,64,512
     atkbddev     2     1K       -        2  64
md_nvidia_data     0     0K       -       30  512
  md_sii_data     0     0K       -       30  512
CAM dev queue    16     1K       -       16  32
     acpiintr     1     1K       -        1  64
       acpica  3773   384K       -    75356  16,32,64,128,256,512,1024,2048
        linux    17     2K       -       17  64
     pci_link    16     2K       -       16  64,128
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
       DEVFS3   209    53K       -      259  256
       DEVFS1   186    93K       -      234  512
   DEVFS_RULE    59    28K       -       59  64,512
        DEVFS    32     1K       -       35  16,128
      io_apic     1     2K       -        1  2048
       DEVFSP    25     2K       -    30080  64
          MCA    18     3K       -       18  64,128
          msi    14     2K       -       14  128
     nexusdev     4     1K       -        4  16
       isadev     8     1K       -        8  128
   kstat_data     5     1K       -        5  64
      solaris 1703874 4696895K       - 129974965  16,32,64,128,256,512,1024,2048,4096
       nvidia  4039 11225K       -  5746247  16,32,64,128,256,512,1024,2048,4096
     iprtheap    54    56K       -    10439  32,64,128,256,512,1024,2048,4096

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     122,       8,     122,   0,   0
UMA Zones:             1664,      0,     122,       0,     122,   0,   0
UMA Slabs:               80,      0,  243624,    1326, 4257206,   0,   0
UMA RCntSlabs:           88,      0,    1266,     174,    2244,   0,   0
UMA Hash:               256,      0,       4,      11,       9,   0,   0
4 Bucket:                32,      0,      67,    1808,367110954,   0,   0
8 Bucket:                64,      0,      66,    1608,   85546,   0,   0
16 Bucket:              128,      0,     125,    1642,   12240,   3,   0
32 Bucket:              256,      0,    4590,    1275,  724111,78724,   0
64 Bucket:              512,      0,    1342,    1114,  248923,1013420,   0
128 Bucket:            1024,      0,    1792,     876, 5890708,  25,   0
vmem btag:               56,      0,     241,     611,     358,  12,   0
VM OBJECT:              240,      0,   48385,   29039, 2407085,   0,   0
RADIX NODE:             144, 3930201,  184598, 3745657,11794456,  49,   0
MAP:                    240,      0,       5,      59,       5,   0,   0
KMAP ENTRY:             128, 1074708,    5518,    2511,10871183,   0,   0
MAP ENTRY:              128,      0,   10722,    3352,10199635,   0,   0
fakepg:                 104,      0,    1700,    1302,  213208,   0,   0
mt_zone:               4112,      0,     346,       0,     346,   0,   0
16:                      16,      0,  139163,     644,190710842,   0,   0
32:                      32,      0,   42888,   17612,29159638,   0,   0
64:                      64,      0,  487243,   88799,2652298344,   0,   0
128:                    128,      0,   91691,   10795,61092940,   0,   0
256:                    256,      0,   43806,  104949,16360080,   0,   0
512:                    512,      0,  815848,   16800,15321575,   0,   0
1024:                  1024,      0,   18260,     920, 1643903,   0,   0
2048:                  2048,      0,    5723,     735,  493608,   0,   0
4096:                  4096,      0,   48020,     814, 1817437,   0,   0
uint64 pcpu:              8,      0,    2672,     528,    2834,   0,   0
Files:                   80,      0,    8147,    1753, 2940171,   0,   0
rl_entry:                40,      0,     362,     638,     362,   0,   0
TURNSTILE:              136,      0,     901,     319,     901,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,     111,      66,   45042,   0,   0
THREAD:                1168,      0,     825,      72,   58776,   0,   0
SLEEPQUEUE:              80,      0,     901,     494,     901,   0,   0
VMSPACE:                400,      0,      90,     290,   44895,   0,   0
cpuset:                  72,      0,     420,     735,     723,   0,   0
audit_record:          1240,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 6690900,     536,    1653,22379129,   0,   0
mbuf:                   256, 6690900,     149,    1686,406372258,   0,   0
mbuf_cluster:          2048, 1045452,    2125,      31,    7375,2586,   0
mbuf_jumbo_page:       4096, 522726,     146,      42,367726674, 795,   0
mbuf_jumbo_9k:         9216, 464643,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 348484,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       0,    2240,16942104,   0,   0
ttyinq:                 160,      0,     570,     705,    2235,   0,   0
ttyoutq:                256,      0,     289,     536,    1144,   0,   0
ata_request:            336,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
nv_stack_t:           12288,      0,       7,      15,  241825,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1577,   71795,   0,   0
VNODE:                  472,      0,  102177,   66951, 3352332,   0,   0
VNODEPOLL:              112,      0,    6616,    1434,    8487,   0,   0
BUF TRIE:               144,      0,    6149,   99070,  122749,   0,   0
NAMEI:                 1024,      0,       0,     156,21682204,   0,   0
S VFS Cache:            108,      0,   88422,   79718, 3512321,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,    4132,   46796,  590789,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,       0,       0,   0,   0
pipe:                   744,      0,     176,     139,   32067,   0,   0
space_seg_cache:         64,      0,   39258,    2034, 3858700,3858370,   0
zio_cache:              944,      0,       2,    9074,58597040,58593610,   0
zio_link_cache:          48,      0,       0,    9711,43090001,43088152,   0
sa_cache:                80,      0,  102069,   66881, 3351934,3351926,   0
dnode_t:                856,      0,  807637,   10919, 1559335,1559328,   0
dmu_buf_impl_t:         224,      0,  925335,   18845, 2913405,2913361,   0
arc_buf_hdr_t:          216,      0,  265906,   73070, 1046452,1046410,   0
arc_buf_t:               72,      0,  157312,    5763, 1275892,1275848,   0
zil_lwb_cache:          192,      0,      10,     970,   60629,60629,   0
zfs_znode_cache:        368,      0,  102069,   66841, 3351934,3351926,   0
Mountpoints:            816,      0,      21,      34,      21,   0,   0
AIO:                    208,      0,       0,       0,       0,   0,   0
AIOP:                    32,      0,       0,       0,       0,   0,   0
AIOCB:                  480,      0,       0,       0,       0,   0,   0
AIOL:                   128,      0,       0,       0,       0,   0,   0
AIOLIO:                 272,      0,       0,       0,       0,   0,   0
ksiginfo:               112,      0,     164,    1691, 1059868,   0,   0
itimer:                 352,      0,       1,      65,       2,   0,   0
pf states:              296,  10010,    1113,     746,  575325,   0,   0
pf state keys:           88,      0,    1113,    1902,  575736,   0,   0
pf source nodes:        136,  10005,       0,       0,       0,   0,   0
pf mtags:                40,      0,       1,     199,       1,   0,   0
pf table entries:       160, 200000,     142,     108,     142,   0,   0
pf table counters:       64,      0,       0,       0,       0,   0,   0
pf frags:                80,      0,       0,     250,      18,   0,   0
pf frag entries:         32,   5000,       0,     250,      74,   0,   0
pf state scrubs:         40,      0,       0,       0,       0,   0,   0
bridge_rtnode:           64,      0,       0,     310,       3,   0,   0
KNOTE:                  128,      0,    7118,    1872,4570631536,   0,   0
socket:                 680, 522726,     753,     183,  397602,   0,   0
unpcb:                  240, 522736,     314,     486,   14119,   0,   0
ipq:                     56,  32731,       0,       0,       0,   0,   0
udp_inpcb:              392, 522730,      22,     328,  188550,   0,   0
udpcb:                   16, 522833,      22,    1484,  188550,   0,   0
tcp_inpcb:              392, 522730,     509,     391,  193981,   0,   0
tcpcb:                 1024, 522728,     411,     177,  193981,   0,   0
tcptw:                   72,  27775,      98,    1552,   93100,   0,   0
syncache:               152,  15366,       1,     675,   87672,   0,   0
hostcache:              136,  15370,    1849,     906,   21622,   0,   0
tcpreass:                40,  65400,       0,    1200,   47005,   0,   0
sackhole:                32,      0,       0,    1625,   29163,   0,   0
sctp_ep:               1408, 522726,       0,       0,       0,   0,   0
sctp_asoc:             2344,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     830,      27,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
ripcb:                  392, 522730,       2,     178,      18,   0,   0
rtentry:                200,      0,      28,     412,      53,   0,   0
selfd:                   56,      0,     855,    1275,2144983378,   0,   0
SWAPMETA:               288, 2034188,    5259,   38421,  104472, 156,   0
FFS inode:              168,      0,      22,     415,     159,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,      22,     413,     159,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                         168          0
irq16: vgapci0 re0+             22556998      79147
irq17: hdac0                          31          0
irq19: ath0                      8665617      30405
irq23: ehci1                      240470        843
cpu0:timer                      10559361      37050
irq264: hdac1                   11132985      39063
irq265: ahci0                    1014110       3558
irq266: xhci0                        141          0
irq268: re1                       612749       2149
irq269: ahci1                    6073944      21312
cpu1:timer                      10974093      38505
cpu6:timer                       9463473      33205
cpu4:timer                       7065873      24792
cpu3:timer                      11054444      38787
cpu7:timer                      11060358      38808
cpu5:timer                      10144606      35595
cpu2:timer                      10887243      38200
Total                          131506664     461426

------------------------------------------------------------------------
pstat -T

8147/522726 files
234M/8191M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/gpt/swap     16776960   480792 16296168     3%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
            ada0             ada1             ada2             cpu
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  7.50 4192 30.69  16.43 9395 150.76  17.13 9015 150.80   2  0  1  0 97

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   
m       131072            0 --rw------- alex     alex     alex     alex                2       393216         2763         2540 18:40:18 15:53:35 18:40:18
m        65537            0 --rw------- alex     alex     alex     alex                2       393216         2768         2540 18:40:19 15:53:35 18:40:19
m        65538            0 --rw------- alex     alex     alex     alex                2       393216         2763         2540 18:40:19 15:53:35 18:40:19
m        65539            0 --rw------- alex     alex     alex     alex                2       393216         2768         2540 18:40:19 15:53:35 18:40:19
m        65540            0 --rw------- alex     alex     alex     alex                2       393216         2794         2540 18:40:19 15:53:35 18:40:19
m        65541            0 --rw------- alex     alex     alex     alex                2       393216         2766         2540 18:40:24  0:42:21 18:40:24
m       131078            0 --rw------- alex     alex     alex     alex                2       393216         3120         2540 18:53:18 21:32:06 18:53:18
m       131079            0 --rw------- alex     alex     alex     alex                2       393216         3120         2540 18:53:20 21:32:06 18:53:20
m        65544            0 --rwarwarwa alex     alex     alex     alex                2         7680         2852         2540 18:44:00 15:53:35 18:44:00
m        65545            0 --rwarwarwa alex     alex     alex     alex                2      1036800         2852         2540 18:44:00 15:53:35 18:44:00
m        65546            0 --rwarwarwa alex     alex     alex     alex                2         4096         2852         2540 18:44:00 15:53:35 18:44:00
m        65547            0 --rwarwarwa alex     alex     alex     alex                2         8192         2852         2540 18:44:00 15:53:35 18:44:00
m        65548            0 --rwarwarwa alex     alex     alex     alex                2        12288         2852         2540 18:44:00 15:53:35 18:44:00
m        65549            0 --rwarwarwa alex     alex     alex     alex                2        16384         2852         2540 18:44:00 15:53:35 18:44:00
m        65550            0 --rwarwarwa alex     alex     alex     alex                2        20480         2852         2540 18:44:00 15:53:35 18:44:00
m        65551            0 --rwarwarwa alex     alex     alex     alex                2        24576         2852         2540 18:44:00 15:53:35 18:44:00
m        65552            0 --rwarwarwa alex     alex     alex     alex                2        28672         2852         2540 18:44:00 15:53:35 18:44:00
m        65553            0 --rwarwarwa alex     alex     alex     alex                2        32768         2852         2540 18:44:00 15:53:35 18:44:00
m        65554            0 --rwarwarwa alex     alex     alex     alex                2        36864         2852         2540 18:44:00 15:53:35 18:44:00
m        65555            0 --rwarwarwa alex     alex     alex     alex                2        40960         2852         2540 18:44:00 15:53:35 18:44:00
m        65556            0 --rwarwarwa alex     alex     alex     alex                2        45056         2852         2540 18:44:00 15:53:35 18:44:00
m        65557            0 --rwarwarwa alex     alex     alex     alex                2        49152         2852         2540 18:44:00 15:53:35 18:44:00
m        65558            0 --rwarwarwa alex     alex     alex     alex                2        53248         2852         2540 18:44:00 15:53:35 18:44:00
m        65559            0 --rwarwarwa alex     alex     alex     alex                2        57344         2852         2540 18:44:00 15:53:35 18:44:00
m        65560            0 --rwarwarwa alex     alex     alex     alex                2        61440         2852         2540 18:44:00 15:53:35 18:44:00
m        65561            0 --rwarwarwa alex     alex     alex     alex                2        65536         2852         2540 18:44:00 15:53:35 18:44:00
m        65562            0 --rwarwarwa alex     alex     alex     alex                2        69632         2852         2540 18:44:00 15:53:35 18:44:00
m        65563            0 --rwarwarwa alex     alex     alex     alex                2        73728         2852         2540 18:44:00 15:53:35 18:44:00
m        65564            0 --rwarwarwa alex     alex     alex     alex                2        77824         2852         2540 18:44:00 15:53:35 18:44:00
m        65565            0 --rwarwarwa alex     alex     alex     alex                2        81920         2852         2540 18:44:00 15:53:35 18:44:00
m        65566            0 --rwarwarwa alex     alex     alex     alex                2        86016         2852         2540 18:44:00 15:53:35 18:44:00
m        65567            0 --rwarwarwa alex     alex     alex     alex                2        90112         2852         2540 18:44:00 15:53:35 18:44:00
m        65568            0 --rwarwarwa alex     alex     alex     alex                2        94208         2852         2540 18:44:00 15:53:35 18:44:00
m        65569            0 --rwarwarwa alex     alex     alex     alex                2        98304         2852         2540 18:44:00 15:53:35 18:44:00
m        65570            0 --rwarwarwa alex     alex     alex     alex                2       102400         2852         2540 18:44:00 15:53:35 18:44:00
m        65571            0 --rwarwarwa alex     alex     alex     alex                2       106496         2852         2540 18:44:00 15:53:35 18:44:00
m        65572            0 --rwarwarwa alex     alex     alex     alex                2       110592         2852         2540 18:44:00 15:53:35 18:44:00
m        65573            0 --rwarwarwa alex     alex     alex     alex                2       114688         2852         2540 18:44:00 15:53:35 18:44:00
m        65574            0 --rwarwarwa alex     alex     alex     alex                2       118784         2852         2540 18:44:00 15:53:35 18:44:00
m        65575            0 --rwarwarwa alex     alex     alex     alex                2       122880         2852         2540 18:44:00 15:53:35 18:44:00
m        65576            0 --rwarwarwa alex     alex     alex     alex                2       126976         2852         2540 18:44:00 15:53:35 18:44:00
m        65577            0 --rwarwarwa alex     alex     alex     alex                2       131072         2852         2540 18:44:00 15:53:35 18:44:00
m        65578            0 --rwarwarwa alex     alex     alex     alex                2       135168         2852         2540 18:44:00 15:53:35 18:44:00
m        65579            0 --rwarwarwa alex     alex     alex     alex                2       139264         2852         2540 18:44:00 15:53:35 18:44:00
m        65580            0 --rwarwarwa alex     alex     alex     alex                2       143360         2852         2540 18:44:00 15:53:35 18:44:00
m        65581            0 --rwarwarwa alex     alex     alex     alex                2       147456         2852         2540 18:44:00 15:53:35 18:44:00
m        65582            0 --rwarwarwa alex     alex     alex     alex                2       151552         2852         2540 18:44:00 15:53:35 18:44:00
m        65583            0 --rwarwarwa alex     alex     alex     alex                2       155648         2852         2540 18:44:00 15:53:35 18:44:00
m        65584            0 --rwarwarwa alex     alex     alex     alex                2       159744         2852         2540 18:44:00 15:53:35 18:44:00
m        65585            0 --rwarwarwa alex     alex     alex     alex                2       163840         2852         2540 18:44:00 15:53:35 18:44:00
m        65586            0 --rwarwarwa alex     alex     alex     alex                2       167936         2852         2540 18:44:00 15:53:35 18:44:00
m        65587            0 --rwarwarwa alex     alex     alex     alex                2       172032         2852         2540 18:44:00 15:53:35 18:44:00
m        65588            0 --rwarwarwa alex     alex     alex     alex                2       176128         2852         2540 18:44:00 15:53:35 18:44:00
m        65589            0 --rwarwarwa alex     alex     alex     alex                2       180224         2852         2540 18:44:00 15:53:35 18:44:00
m        65590            0 --rwarwarwa alex     alex     alex     alex                2       184320         2852         2540 18:44:00 15:53:35 18:44:00
m        65591            0 --rwarwarwa alex     alex     alex     alex                2       188416         2852         2540 18:44:00 15:53:35 18:44:00
m        65592            0 --rwarwarwa alex     alex     alex     alex                2       192512         2852         2540 18:44:00 15:53:35 18:44:00
m        65593            0 --rwarwarwa alex     alex     alex     alex                2       196608         2852         2540 18:44:00 15:53:35 18:44:00
m        65594            0 --rwarwarwa alex     alex     alex     alex                2       200704         2852         2540 18:44:00 15:53:35 18:44:00
m        65595            0 --rwarwarwa alex     alex     alex     alex                2       204800         2852         2540 18:44:00 15:53:35 18:44:00
m        65596            0 --rwarwarwa alex     alex     alex     alex                2       208896         2852         2540 18:44:00 15:53:35 18:44:00
m        65597            0 --rwarwarwa alex     alex     alex     alex                2       212992         2852         2540 18:44:00 15:53:35 18:44:00
m        65598            0 --rwarwarwa alex     alex     alex     alex                2       217088         2852         2540 18:44:00 15:53:35 18:44:00
m        65599            0 --rwarwarwa alex     alex     alex     alex                2       221184         2852         2540 18:44:00 15:53:35 18:44:00
m        65600            0 --rwarwarwa alex     alex     alex     alex                2       225280         2852         2540 18:44:00 15:53:35 18:44:00
m        65601            0 --rwarwarwa alex     alex     alex     alex                2       229376         2852         2540 18:44:00 15:53:35 18:44:00
m        65602            0 --rwarwarwa alex     alex     alex     alex                2       233472         2852         2540 18:44:00 15:53:35 18:44:00
m        65603            0 --rwarwarwa alex     alex     alex     alex                2       237568         2852         2540 18:44:00 15:53:35 18:44:00
m        65604            0 --rwarwarwa alex     alex     alex     alex                2       241664         2852         2540 18:44:00 15:53:35 18:44:00
m        65605            0 --rwarwarwa alex     alex     alex     alex                2       245760         2852         2540 18:44:00 15:53:35 18:44:00
m        65606            0 --rw------- alex     alex     alex     alex                2        12288         2763         2540 18:48:43 15:53:35 18:48:43
m        65607            0 --rw------- alex     alex     alex     alex                2       393216         2766         2540 18:48:43  0:42:21 18:48:43
m        65608            0 --rw------- alex     alex     alex     alex                2        12288         2766         2540 18:48:43  0:42:21 18:48:43
m        65609            0 --rw------- alex     alex     alex     alex                2       393216         2753         2540 18:48:43 15:53:35 18:48:43
m        65610            0 --rw------- alex     alex     alex     alex                2        12288         2753         2540 18:48:43 15:53:35 18:48:43
m       196683            0 --rw------- alex     alex     alex     alex                2       393216        39581         2540 19:39:56 no-entry 19:39:56
m        65612            0 --rw------- alex     alex     alex     alex                2       393216         3128         2540 18:53:42 15:53:35 18:53:42
m        65613            0 --rw------- alex     alex     alex     alex                2       393216         3128         2540 18:53:42 15:53:35 18:53:42
m        65614            0 --rw------- alex     alex     alex     alex                2       393216         3138         2540 18:54:20  3:44:55 18:54:20
m       458831            0 --rwarwarwa alex     alex     alex     alex                2      7764480        41470         2540 20:23:45 no-entry 20:23:45
m        65616            0 --rw------- alex     alex     alex     alex                2       393216         3138         2540 18:54:22  3:44:55 18:54:22
m        65617            0 --rwarwarwa alex     alex     alex     alex                2       614400         3152         2540 18:54:27 15:53:35 18:54:27
m        65618            0 --rwarwarwa alex     alex     alex     alex                2      1768680         3156         2540 18:54:29 15:53:35 18:54:29
m       131155            0 --rwarwarwa alex     alex     alex     alex                2      1870000         3152         2540 18:54:58 15:53:35 18:54:58
m       524372            0 --rwarwarwa alex     alex     alex     alex                2       100040        41470         2540 20:23:46 no-entry 20:23:46
m       393301            0 --rw------- alex     alex     alex     alex                2       393216         8517         2540 21:46:47 15:53:35 21:46:47
m      1704022            0 --rwarwarwa alex     alex     alex     alex                2        57552        41470         2540 20:23:46 no-entry 20:23:46
m      3145815            0 --rwarwarwa alex     alex     alex     alex                2       118728        41470         2540 20:34:37 no-entry 20:34:37
m      2555992            0 --rwarwarwa alex     alex     alex     alex                2       147784        41470         2540 20:36:50 no-entry 20:36:50
m       852057            0 --rwarwarwa alex     alex     alex     alex                2        44676        41470         2540 20:36:55 no-entry 20:36:55
m      1769562            0 --rwarwarwa alex     alex     alex     alex                2       341224        41598         2540 20:45:38 no-entry 20:45:38
m      1507419            0 --rwarwarwa alex     alex     alex     alex                2       395520        43461         2540  0:42:22 no-entry  0:42:22
m       589916            0 --rwarwarwa alex     alex     alex     alex                2       327684        41598         2540 20:45:42 no-entry 20:45:42
m      2490461            0 --rwarwarwa alex     alex     alex     alex                2      5281344        41598         2540 21:52:55 no-entry 21:52:55
m       327802            0 --rw------- alex     alex     alex     alex                2       393216        39581         2540 15:23:14 15:53:35 15:23:14
m       524421            0 --rwarwarwa alex     alex     alex     alex                2      8294400        43461         2540  0:42:42 no-entry  0:42:42
m       327814            0 --rwarwarwa alex     alex     alex     alex                2       230400        43461         2540  0:42:42 no-entry  0:42:42

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:         1024	(max number of shared memory identifiers)
	shmseg:         1024	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0         0
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
        0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         0         0

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
	5224274 packets sent
		2555947 data packets (2098914935 bytes)
		86842 data packets (99152287 bytes) retransmitted
		1472 data packets unnecessarily retransmitted
		6074 resends initiated by MTU discovery
		2295561 ack-only packets (952849 delayed)
		0 URG only packets
		7 window probe packets
		1906 window update packets
		285651 control packets
	5804659 packets received
		2598511 acks (for 2100483620 bytes)
		245742 duplicate acks
		0 acks for unsent data
		3737393 packets (3440615878 bytes) received in-sequence
		25253 completely duplicate packets (12728742 bytes)
		245 old duplicate packets
		1858 packets with some dup. data (1253681 bytes duped)
		47001 out-of-order packets (58600045 bytes)
		3 packets (3 bytes) of data after window
		3 window probes
		9513 window update packets
		4170 packets received after close
		26 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	105974 connection requests
	87405 connection accepts
	0 bad connection attempts
	252 listen queue overflows
	193 ignored RSTs in the windows
	180007 connections established (including accepts)
	193472 connections closed (including 2832 drops)
		140987 connections updated cached RTT on close
		141829 connections updated cached RTT variance on close
		32725 connections updated cached ssthresh on close
	1633 embryonic connections dropped
	2500421 segments updated rtt (of 2309275 attempts)
	94164 retransmit timeouts
		1367 connections dropped by rexmit timeout
	9 persist timeouts
		1 connection dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	92 keepalive timeouts
		2 keepalive probes sent
		90 connections dropped by keepalive
	857321 correct ACK header predictions
	2754454 correct data packet header predictions
	87668 syncache entries added
		802 retransmitted
		1065 dupsyn
		4 dropped
		87405 completed
		0 bucket overflow
		0 cache overflow
		51 reset
		96 stale
		252 aborted
		0 badack
		1 unreach
		0 zone failures
	87672 cookies sent
	139 cookies received
	21622 hostcache entries added
		0 bucket overflow
	10851 SACK recovery episodes
	20498 segment rexmits in SACK recovery episodes
	28997940 byte rexmits in SACK recovery episodes
	102742 SACK options (SACK blocks) received
	40996 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
udp:
	5011390 datagrams received
	0 with incomplete header
	0 with bad data length field
	471 with bad checksum
	66294 with no checksum
	11663 dropped due to no socket
	1619 broadcast/multicast datagrams undelivered
	10733 dropped due to full socket buffers
	0 not for hashed pcb
	4986904 delivered
	4617407 datagrams output
	0 times multicast source filter matched
ip:
	11307986 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	10845066 packets for this host
	0 packets for unknown/unsupported protocol
	448468 packets forwarded (0 packets fast forwarded)
	358 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	9851219 packets sent from this host
	890 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	191 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	17 calls to icmp_error
	0 errors not generated in response to an icmp message
	Output histogram:
		echo reply: 4
		destination unreachable: 17
	0 messages with bad code fields
	0 messages less than the minimum length
	52 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	Input histogram:
		echo reply: 5
		destination unreachable: 22917
		routing redirect: 80
		echo: 4
		time exceeded: 150
	4 message responses generated
	0 invalid return addresses
	2 no return routes
igmp:
	0 messages received
	0 messages received with too few bytes
	0 messages received with wrong TTL
	0 messages received with bad checksum
	0 V1/V2 membership queries received
	0 V3 membership queries received
	0 membership queries received with invalid field(s)
	0 general queries received
	0 group queries received
	0 group-source queries received
	0 group-source queries dropped
	0 membership reports received
	0 membership reports received with invalid field(s)
	0 membership reports received for groups to which we belong
	0 V3 reports received without Router Alert
	0 membership reports sent
ipsec:
	0 inbound packets processed successfully
	0 inbound packets violated process security policy
	0 inbound packets with no SA available
	0 invalid inbound packets
	0 inbound packets failed due to insufficient memory
	0 inbound packets failed getting SPI
	0 inbound packets failed on AH replay check
	0 inbound packets failed on ESP replay check
	0 inbound packets considered authentic
	0 inbound packets failed on authentication
	AH input histogram:
		#81: 4
		#82: 368
		#90: 2
		#92: 1
		#102: 1
	ESP input histogram:
		#22: 5388428899424924254
		#70: 4
		#71: 64
		#79: 2
		#81: 1
		#91: 1
	IPComp input histogram:
		#85: 2
	0 outbound packets processed successfully
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 invalid outbound packets
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route
	AH output histogram:
		#25: 18446744069452039826
		#29: 18446739684043457005
		#30: 18446739684043457005
		#32: 18446744069435070473
		#34: 18446744067297799018
		#36: 18446744069435073259
		#38: 18446741881917108930
		#39: 18446741872550314692
		#40: 18446744069452039714
		#42: 18446744069435070473
		#47: 18446741886747282432
		#50: 18446741886747282432
		#52: 18446744069435070669
		#58: 5208144556594611265
		#60: 2676602441897494845
		#61: 2102737147077349346
		#143: 15922656940817424955
		#145: 1588162959370749784
		#146: 4248973215496401704
		#147: 184520657423066741
		#152: 18446741875371866480
	ESP output histogram:
		#184: 13713805283037902050
		#185: 7791341353509176522
		#229: 2760502319720234799
	IPComp output histogram:
		#16: 5230142452003304741
		#18: 18446744071572311547
		#19: 18446744067297799018
		#20: 18446741872980067692
		#22: 18446741875371196416
		#24: 18446741875371866480
		#27: 18446735329741374784
		#178: 18446741886747282432
		#187: 16687363430680922001
		#190: 18446741877255011328
		#191: 18446737619112244640
		#198: 5000446388804846320
		#199: 18305527321790384210
		#202: 18446741879555006464
		#203: 18446737535058806896
		#239: 18446744069435073259
		#240: 18446739675234344433
	0 SPD cache lookups
	0 SPD cache misses
	0 inbound packets violated process security policy
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	0 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
ah:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets dropped; bad authentication detected
	0 packets dropped; bad authentication length
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
esp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 packets dropped; bad ilen
	0 replay counter wraps
	0 packets dropped; bad encryption detected
	0 packets dropped; bad authentication detected
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
ipcomp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 packets sent uncompressed; size < compr. algo. threshold
	0 packets sent uncompressed; compression was useless
pfsync:
	0 packets received (IPv4)
	0 packets received (IPv6)
	    0 clear all requests received
	    0 state inserts received
	    0 state inserted acks received
	    0 state updates received
	    0 compressed state updates received
	    0 uncompressed state requests received
	    0 state deletes received
	    0 compressed state deletes received
	    0 fragment inserts received
	    0 fragment deletes received
	    0 bulk update marks received
	    0 TDB replay counter updates received
	    0 end of frame marks received
		0 packets discarded for bad interface
		0 packets discarded for bad ttl
		0 packets shorter than header
		0 packets discarded for bad version
		0 packets discarded for bad HMAC
		0 packets discarded for bad action
		0 packets discarded for short packet
		0 states discarded for bad values
		0 stale states
		0 failed state lookup/inserts
	0 packets sent (IPv4)
	0 packets sent (IPv6)
	    1 clear all request sent
	    575325 state inserts sent
	    0 state inserted acks sent
	    0 state updates sent
	    10137482 compressed state updates sent
	    0 uncompressed state requests sent
	    0 state deletes sent
	    574212 compressed state deletes sent
	    0 fragment inserts sent
	    0 fragment deletes sent
	    0 bulk update marks sent
	    0 TDB replay counter updates sent
	    759281 end of frame marks sent
		0 failures due to mbuf memory error
		0 send errors
arp:
	116 ARP requests sent
	185 ARP replies sent
	324 ARP requests received
	112 ARP replies received
	436 ARP packets received
	6 total packets dropped due to no ARP entry
	2 ARP entrys timed out
	0 Duplicate IPs seen
ip6:
	250700 total packets received
	0 with size smaller than minimum
	0 with data size < data length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 fragments that exceeded limit
	0 packets reassembled ok
	244952 packets for this host
	0 packets forwarded
	0 packets not forwardable
	0 redirects sent
	140430 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	2620 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 packets that violated scope rules
	1 multicast packet which we don't join
	Input histogram:
		hop by hop: 2
		TCP: 244750
		UDP: 102
		fragment: 28
		ICMP6: 5818
	Mbuf statistics:
		41 one mbuf
		250659 one ext mbuf
		0 two or more ext mbuf
	0 packets whose headers are not contiguous
	0 tunneling packets that can't find gif
	0 packets discarded because of too many headers
	1195 failures of source address selection
	source addresses on an outgoing I/F
		4519 globals
	source addresses on a non-outgoing I/F
		1195 addresses scope=f
	source addresses of same scope
		4519 globals
	Source addresses selection rule applied:
		4519 first candidate
		1195 same address
		4120 appropriate scope
		356 longest match
icmp6:
	0 calls to icmp6_error
	0 errors not generated in response to an icmp6 message
	0 errors not generated because of rate limitation
	Output histogram:
		echo: 342
		echo reply: 59
		neighbor solicitation: 144
		neighbor advertisement: 66
		MLDv2 listener report: 16
	0 messages with bad code fields
	0 messages < minimum length
	0 bad checksums
	0 messages with bad length
	Input histogram:
		parameter problem: 6
		echo: 59
		echo reply: 102
		neighbor solicitation: 66
		neighbor advertisement: 71
	Histogram of error messages to be generated:
		0 no route
		0 administratively prohibited
		0 beyond scope
		0 address unreachable
		0 port unreachable
		0 packet too big
		0 time exceed transit
		0 time exceed reassembly
		0 erroneous header field
		0 unrecognized next header
		0 unrecognized option
		0 redirect
		0 unknown
	59 message responses generated
	0 messages with too many ND options
	0 messages with bad ND options
	0 bad neighbor solicitation messages
	0 bad neighbor advertisement messages
	0 bad router solicitation messages
	0 bad router advertisement messages
	0 bad redirect messages
	0 path MTU changes
ipsec6:
	0 inbound packets processed successfully
	0 inbound packets violated process security policy
	0 inbound packets with no SA available
	0 invalid inbound packets
	0 inbound packets failed due to insufficient memory
	0 inbound packets failed getting SPI
	0 inbound packets failed on AH replay check
	0 inbound packets failed on ESP replay check
	0 inbound packets considered authentic
	0 inbound packets failed on authentication
	AH input histogram:
		#65: 4
		#66: 368
		#74: 2
		#76: 1
		#86: 1
	ESP input histogram:
		cast128-cbc: 5388428899424924254
		#54: 4
		#55: 64
		#63: 2
		#65: 1
		#75: 1
	IPComp input histogram:
		#69: 2
	0 outbound packets processed successfully
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 invalid outbound packets
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route
	AH output histogram:
		aes-xcbc-mac: 18446744069452039826
		#13: 18446739684043457005
		#14: 18446739684043457005
		#16: 18446744069435070473
		#18: 18446744067297799018
		#20: 18446744069435073259
		#22: 18446741881917108930
		#23: 18446741872550314692
		#24: 18446744069452039714
		#26: 18446744069435070473
		#31: 18446741886747282432
		#34: 18446741886747282432
		#36: 18446744069435070669
		#42: 5208144556594611265
		#44: 2676602441897494845
		#45: 2102737147077349346
		#127: 15922656940817424955
		#129: 1588162959370749784
		#130: 4248973215496401704
		#131: 184520657423066741
		#136: 18446741875371866480
	ESP output histogram:
		#168: 13713805283037902050
		#169: 7791341353509176522
		#213: 2760502319720234799
	IPComp output histogram:
		none: 5230142452003304741
		deflate: 18446744071572311547
		lzs: 18446744067297799018
		#4: 18446741872980067692
		#6: 18446741875371196416
		#8: 18446741875371866480
		#11: 18446735329741374784
		#162: 18446741886747282432
		#171: 16687363430680922001
		#174: 18446741877255011328
		#175: 18446737619112244640
		#182: 5000446388804846320
		#183: 18305527321790384210
		#186: 18446741879555006464
		#187: 18446737535058806896
		#223: 18446744069435073259
		#224: 18446739675234344433
	0 SPD cache lookups
	0 SPD cache misses
	0 inbound packets violated process security policy
	0 outbound packets violated process security policy
	18446744071570230624 outbound packets with no SA available
	18446741323484043320 outbound packets failed due to insufficient memory
	7521981600624922732 outbound packets with no route available
	18446739675069395832 invalid outbound packets
	18012200171794434 outbound packets with bundled SAs
	18012200171129204 mbufs coalesced during clone
	18446741351724389120 clusters coalesced during clone
	9005001085775874 clusters copied during clone
	18446741875371870576 mbufs inserted during makespace
rip6:
	0 messages received
	0 checksum calculations on inbound
	0 messages with bad checksum
	0 messages dropped due to no socket
	0 multicast messages dropped due to no socket
	0 messages dropped due to full socket buffers
	0 delivered
	0 datagrams output
pfkey:
	4 requests sent from userland
	64 bytes sent from userland
	histogram by message type:
		register: 2
		flush: 1
		x_spdflush: 1
	0 messages with invalid length field
	0 messages with invalid version field
	0 messages with invalid message type field
	0 messages too short
	0 messages with memory allocation failure
	0 messages with duplicate extension
	0 messages with invalid extension type
	0 messages with invalid sa type
	0 messages with invalid address extension
	4 requests sent to userland
	368 bytes sent to userland
	histogram by message type:
		register: 2
		flush: 1
		x_spdflush: 1
	0 messages toward single socket
	0 messages toward all sockets
	2 messages toward registered sockets
	0 messages with memory allocation failure

------------------------------------------------------------------------
netstat -m

685/3339/4024 mbufs in use (current/cache/total)
472/1684/2156/1045452 mbuf clusters in use (current/cache/total/max)
536/1653 mbuf+clusters out of packet secondary zone in use (current/cache)
146/42/188/522726 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/464643 9k jumbo clusters in use (current/cache/total/max)
0/0/0/348484 16k jumbo clusters in use (current/cache/total/max)
1699K/4370K/6070K bytes allocated to network (current/cache/total)
0/2586/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
795/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

------------------------------------------------------------------------
fstat

fstat: procstat_getprocs()

------------------------------------------------------------------------
dmesg

Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address	= 0x28
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff807ac4e7
stack pointer	        = 0x28:0xffffff8000346840
frame pointer	        = 0x28:0xffffff8000346850
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi5: fast taskq)
trap number		= 12
panic: page fault
cpuid = 4
Uptime: 1d9h18m37s
Dumping 4132 out of 16335 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	POSEIDON
machine	amd64
cpu	HAMMER
makeoptions	MODULES_OVERRIDE=opensolaris zfs netgraph
makeoptions	COPTFLAGS=-O -pipe
makeoptions	DEBUG=-g
options	USB_DEBUG
options	ATH_ENABLE_11N
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_DEBUG
options	SC_PIXEL_MODE
options	ATA_STATIC_ID
options	SMP
options	NULLFS
options	VIMAGE
options	LINPROCFS
options	COMPAT_LINUX32
options	IPSEC_NAT_T
options	IPSEC
options	ALTQ_PRIQ
options	ALTQ_CBQ
options	ALTQ
options	GEOM_JOURNAL
options	GEOM_ELI
options	GEOM_AES
options	VFS_AIO
options	INCLUDE_CONFIG_FILE
options	MAC
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD32
options	GEOM_RAID
options	GEOM_LABEL
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	MD_ROOT
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	crypto
device	cryptodev
device	tap
device	if_bridge
device	epair
device	enc
device	coretemp
device	pf
device	pflog
device	pfsync
device	cpufreq
device	acpi
device	pci
device	ahci
device	ata
device	scbus
device	da
device	pass
device	mfi
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	uart
device	miibus
device	re
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	wlan_xauth
device	ath
device	ath_pci
device	ath_hal
device	ath_rate_sample
device	loop
device	random
device	ether
device	vlan
device	tun
device	pty
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	ukbd
device	ulpt
device	umass
device	ums
device	sound
device	snd_hda
device	snd_uaudio

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist


>Release-Note:
>Audit-Trail:
>Unformatted:


More information about the freebsd-bugs mailing list