CURRENT [r308668] autofs kernel panic

Alex Deiter alex.deiter at gmail.com
Tue Nov 15 17:21:39 UTC 2016


Hello Edward,

Thank you for the quick reply!
I was able to reproduce this panic on default GENERIC kernel (my test
system and laptop):

# kgdb /boot/kernel/kernel /var/crash/vmcore.2
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address = 0xe0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff809fe487
stack pointer        = 0x28:0xfffffe085f5d4560
frame pointer        = 0x28:0xfffffe085f5d45b0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 (pkg)

kgdb: kvm_read: invalid address (0xffff)
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols
from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from
/usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/autofs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/autofs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/autofs.ko
#0  0x0000000000000000 in ?? ()
(kgdb)


(kgdb) bt
#0  0x0000000000000000 in ?? ()
Cannot access memory at address 0x0
(kgdb)

Please take a look attached file. Let me know how I can assist further.

Thank you!

2016-11-15 19:31 GMT+03:00 Edward Tomasz Napierała <trasz at freebsd.org>:
> On 1115T1016, Alex Deiter wrote:
>> Hello,
>>
>> FreeBSD 12.0-CURRENT-r308668 amd64 crashed on "pkg install” command:
>>
>> # cat /usr/local/etc/pkg/repos/local.conf
>> local: {
>> url: "file:///net/nostromo/var/devel/pkg",
>> enabled: yes
>> }
>>
>> Path /net/nostromo/var/devel/pkg accessible via autofs/automountd:
>>
>> # ls /net/nostromo/var/devel/pkg
>> All Latest digests.txz meta.txz packagesite.txz
>>
>> # mount -t nfs
>> nostromo:/var/devel/pkg on /net/nostromo/var/devel/pkg (nfs, nosuid,
>> automounted)
>>
>> # kgdb /usr/lib/debug/boot/kernel/kernel.debug /var/crash/vmcore.0
>> ...
>> Unread portion of the kernel message buffer:
>> kernel trap 12 with interrupts disabled
>>
>>
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 0; apic id = 00
>> fault virtual address = 0x100df
>> fault code = supervisor read data, page not present
>> instruction pointer = 0x20:0xffffffff8063c88d
>> stack pointer         = 0x28:0xfffffe085f966520
>> frame pointer         = 0x28:0xfffffe085f966570
>> code segment = base 0x0, limit 0xfffff, type 0x1b
>> = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags = resume, IOPL = 0
>> current process = -134214453 (pkg)
>>
>> kgdb: kvm_read: invalid address (0x6c6576656c6572)
>> kgdb: kvm_read: invalid address (0xf8000ce021d8ffff)
>> #0  0x0000000000000000 in ?? ()
>> (kgdb) list *0xffffffff8063c88d
>> 0xffffffff8063c88d is in hardclock_cnt (/usr/src/sys/kern/kern_clock.c:568).
>
> [..]
>
> It looks a bit weird - the "kvm_read" errors, and also the one-function
> traceback that contains only one function, that seems unrelated to autofs.
> Are you sure you're using the right kernel?  Could you try the usual
>
> kgdb /boot/kernel/kernel /var/crash/vmcore.0
>
> instead?  Thanks!
>



-- 
Alex Deiter
-------------- next part --------------
serenity.deiter.local dumped core - see /var/crash/vmcore.2

Tue Nov 15 20:11:24 MSK 2016

FreeBSD serenity.deiter.local 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r308668M: Tue Nov 15 19:46:51 MSK 2016     root at nostromo.deiter.local:/usr/obj/usr/src/sys/GENERIC  amd64

panic: 

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...kgdb: kvm_read: invalid address (0xffff)


Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address	= 0xe0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff809fe487
stack pointer	        = 0x28:0xfffffe085f5d4560
frame pointer	        = 0x28:0xfffffe085f5d45b0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= resume, IOPL = 0
current process		= 0 (pkg)

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/autofs.ko...Reading symbols from /usr/lib/debug//boot/kernel/autofs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/autofs.ko
#0  0x0000000000000000 in ?? ()
(kgdb) #0  0x0000000000000000 in ?? ()
Cannot access memory at address 0x0
(kgdb) 

------------------------------------------------------------------------
ps -axlww

UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT  TIME COMMAND

------------------------------------------------------------------------
vmstat -s

   180506 cpu context switches
     6139 device interrupts
      447 software interrupts
    83397 traps
   491548 system calls
       22 kernel threads created
      708  fork() calls
      235 vfork() calls
        0 rfork() calls
        0 swap pager pageins
        0 swap pager pages paged in
        0 swap pager pageouts
        0 swap pager pages paged out
     1835 vnode pager pageins
     2151 vnode pager pages paged in
        0 vnode pager pageouts
        0 vnode pager pages paged out
        0 page daemon wakeups
        0 pages examined by the page daemon
        0 clean page reclamation shortfalls
        0 pages reactivated by the page daemon
    26632 copy-on-write faults
      105 copy-on-write optimized faults
    44748 zero fill pages zeroed
        0 zero fill pages prezeroed
       13 intransit blocking page faults
    80741 total VM faults taken
     1737 page faults requiring I/O
        0 pages affected by kernel thread creation
    25733 pages affected by  fork()
     8225 pages affected by vfork()
        0 pages affected by rfork()
        0 pages cached
   127896 pages freed
        0 pages freed by daemon
        0 pages freed by exiting processes
     4843 pages active
      721 pages inactive
        0 pages in the laundry queue
   117944 pages wired down
  8025210 pages free
     4096 bytes per page
        0 total name lookups
          cache hits (0% pos + 0% neg) system 0% per-directory
          deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
       feeder     7     1K       -        7  32
  ddb_capture     1    64K       -        1  65536
         UART     6     5K       -        6  16,1024
          USB    57   116K       -       70  16,32,128,256,512,1024,2048,4096,16384,32768
       USBdev    39     5K       -       39  32,64,128,256,512
     acpiintr     1     1K       -        1  64
       acpica  2206   224K       -    56529  16,32,64,128,256,512,1024,2048,4096
     acpitask     1    64K       -        1  65536
      acpisem    40     5K       -       40  128
      acpidev    36     3K       -       36  64
      CAM SIM    13     4K       -       13  256
      CAM XPT   133    12K       -      533  16,32,64,128,256,512,1024,2048,65536
      CAM DEV    25    50K       -       60  2048
      CAM CCB     0     0K       -     4613  2048
     CAM path    38     2K       -      217  32
        vtbuf    24  2400K       -       46  4096
           vt    11     6K       -       11  512
   CAM periph    28     7K       -       61  16,32,64,128,256
       DEVFS3   199    50K       -      208  256
       DEVFS1   158    79K       -      166  512
   DEVFS_RULE    55    26K       -       55  64,512
        DEVFS    17     1K       -       18  16,128
NFSD V4client     1     1K       -        1  256
 NFSD lckfile     1     1K       -        1  256
  NFSD string     1     1K       -        1  16
NFSD usrgroup    61    40K       -       61  128,8192
       NFS fh     3     1K       -       10  32
 NFSD session     1     1K       -        1  1024
    newnfsmnt     1     1K       -        1  1024
    pfs_nodes    21     6K       -       21  256
         GEOM   409    71K       -     2394  16,32,64,128,256,512,1024,2048,8192,16384
    raid_data     0     0K       -      306  32,128,256
       isadev     6     1K       -        6  128
     pci_link    16     2K       -       16  32,128
    acpi_perf     8     4K       -        8  512
         cdev     3     1K       -        3  256
        sigio     1     1K       -        1  64
     filecaps     3     1K       -       27  16
      kdtrace   516   121K       -     1713  64,256
         kenv    97    12K       -      100  16,32,64,128,8192
       kqueue    45     4K       -      982  64,256,512,2048
    proc-args    21     1K       -      397  16,32,64,128,256
        hhook    13     4K       -       13  256
      ithread   194    29K       -      194  32,128,256
       KTRACE   100    13K       -      100  128
       linker   202    74K       -      217  16,32,64,128,256,512,1024,2048,4096,8192,16384
CAM I/O Scheduler    13     2K       -       13  128
        lockf    14     2K       -       26  64,128
   loginclass     3     1K       -        3  64
       devbuf 19803 42342K       -    20325  16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
         temp    60    18K       -     1283  16,32,64,128,256,512,1024,2048,16384,65536
    CAM queue    38   153K       -      155  16,32,64,128,256,512,16384
       module   494    62K       -      495  128
     mtx_pool     2    16K       -        2  8192
          osd     7     1K       -       62  16,32,64,128,256
     pmchooks     1     1K       -        1  128
         pgrp    14     2K       -       17  128
      session    13     2K       -       13  128
         proc     2   128K       -        2  65536
      subproc   152   227K       -     1075  512,4096
         cred    28     7K       -       74  256
       kbdmux     6    22K       -        6  16,512,1024,2048,16384
       plimit    11     3K       -      154  256
      uidinfo     2    17K       -        3  128,16384
       dumper     1     1K       -        1  512
          LED     8     1K       -        8  16,128
       sysctl     0     0K       -      839  16,32
    sysctloid  7782   400K       -     7880  16,32,64,128
    sysctltmp     0     0K       -       58  16,32,64,256,1024
      tidhash     1   128K       -        1  
      callout     9  3208K       -        9  
         umtx  1026   129K       -     1026  128
     p1003.1b     1     1K       -        1  16
         SWAP     2  8741K       -        2  64
          bus  1316   113K       -     6175  16,32,64,128,256,512,1024
       bus-sc    88   345K       -     3622  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
      devstat    22    45K       -       22  32,4096
 eventhandler   108     9K       -      108  64,128
    taskqueue    29    19K       -       29  16,32,256,8192
         kobj   336  1344K       -      846  4096
      Per-cpu     1     1K       -        1  32
         rman   304    36K       -      723  32,128
         sbuf     0     0K       -     3885  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
    toponodes    29     4K       -       29  128
    taskqueue   267    29K       -      384  16,32,64,128,256
     terminal    11     3K       -       11  256
       Unitno    22     2K       -      240  32,64
         vmem     2   144K       -        3  8192,16384
      Witness  1539  3282K       -     1539  2048,16384
     ioctlops     1     8K       -       67  256,1024,8192
       select    20     3K       -       20  128
          iov     0     0K       -    16685  64,128,256,512
          msg     4    30K       -        4  2048,4096,8192,16384
          sem     4   106K       -        4  2048,4096
          shm     1    32K       -        1  32768
          tty    15    15K       -       15  1024
          pts     1     1K       -        1  256
     mbuf_tag     0     0K       -        2  32
        shmfd     1     8K       -        1  8192
       soname     6     1K       -    17303  16,32,128
          pcb    27  4247K       -       61  16,32,128,1024,2048,8192
     vfscache     4  8353K       -        4  512,32768
     vfs_hash     1  4096K       -        1  
       vnodes     1     1K       -        1  256
        mount   198     8K       -      435  16,32,64,128,256
  vnodemarker     0     0K       -       16  512
          BPF     5     1K       -        5  128
        ifnet     6    11K       -        6  128,2048
       ifaddr    65    16K       -       65  32,64,256,512,2048,4096
  ether_multi    40     3K       -       40  16,32,64
        clone     6     1K       -        6  128
      lltable    24     7K       -       24  256,512
     routetbl    18     3K       -       21  32,64,128,512
         igmp     5     1K       -        5  128
     in_multi     2     1K       -        2  256
encap_export_host     2     2K       -        2  1024
    sctp_a_it     0     0K       -        3  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa     5     1K       -        5  128
     sctp_ifn     2     1K       -        2  128
    sctp_iter     0     0K       -        3  256
    hostcache     1    32K       -        1  32768
      tcpfunc     1     1K       -        1  32
     syncache     1    64K       -        1  65536
    in6_multi    25     3K       -       25  32,256
          mld     5     1K       -        5  128
       ip6ndp     7     1K       -        8  64,256
  inpcbpolicy    14     1K       -       77  32
  ipsecpolicy    28     7K       -      156  256
       crypto     1     1K       -        1  512
          rpc    21    22K       -       31  32,64,128,256,512,1024,8192
audit_evclass   189     6K       -      233  32
    ufs_quota     1  4096K       -        1  
    vm_pgdata     2  4097K       -        2  128
      UMAHash    10     7K       -       14  512,1024
      memdesc     1     4K       -        1  4096
          mps    21   479K       -       84  16,32,64,128,256,512,1024,4096,8192,65536
CAM dev queue    13     1K       -       13  64
      scsi_da     0     0K       -      129  32,64
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        1  4096
      entropy     1     1K       -        4  32,4096
      io_apic     1     2K       -        1  2048
          MCA     8     1K       -        8  128
          msi    40     5K       -       40  128
     nexusdev     4     1K       -        4  16
      solaris 10988 101722K       -   701325  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
   kstat_data     6     1K       -        6  64
       autofs    19     9K       -      530  16,32,128,8192

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     329,       1,     329,   0,   0
UMA Zones:             1664,      0,     346,       0,     346,   0,   0
UMA Slabs:              112,      0,    8722,      28,    9052,   0,   0
UMA Hash:               256,      0,      92,      13,     102,   0,   0
4 Bucket:                32,      0,     558,    1550,    3145,   0,   0
6 Bucket:                48,      0,      74,    2001,     255,   0,   0
8 Bucket:                64,      0,      95,    1703,    3339,  11,   0
12 Bucket:               96,      0,      77,     873,     483,   0,   0
16 Bucket:              128,      0,     184,     839,     844,   0,   0
32 Bucket:              256,      0,     257,     433,    1233, 139,   0
64 Bucket:              512,      0,     173,     219,    1246, 247,   0
128 Bucket:            1024,      0,     183,      65,     293,   0,   0
256 Bucket:            2048,      0,     427,      17,    1057,   4,   0
vmem btag:               56,      0,   11977,     874,   12184,  91,   0
VM OBJECT:              264,      0,    1189,     236,   12686,   0,   0
RADIX NODE:             144,      0,    6672,     672,   33222,   0,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,       7,     396,       7,   0,   0
MAP ENTRY:              128,      0,     503,    1049,   28520,   0,   0
VMSPACE:               2512,      0,      21,      30,     945,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
mt_zone:              16400,      0,     404,       0,     404,   0,   0
16:                      16,      0,      32,    1960,   23801,   0,   0
16:                      16,      0,       1,     497,       1,   0,   0
16:                      16,      0,     882,    1857,   41329,   0,   0
16:                      16,      0,     257,    1984,    3639,   0,   0
16:                      16,      0,    2535,    1698,    3272,   0,   0
16:                      16,      0,     130,    2111,    1551,   0,   0
16:                      16,      0,     233,    2008,     452,   0,   0
16:                      16,      0,      60,    1434,     200,   0,   0
32:                      32,      0,     119,    1989,    6544,   0,   0
32:                      32,      0,     265,    2091,     580,   0,   0
32:                      32,      0,    1376,    1972,   91754,   0,   0
32:                      32,      0,     108,    2248,    1985,   0,   0
32:                      32,      0,    3201,    1387,    3436,   0,   0
32:                      32,      0,     177,    1683,     730,   0,   0
32:                      32,      0,     187,    1673,     291,   0,   0
32:                      32,      0,      30,    1334,     189,   0,   0
64:                      64,      0,     897,     901,    1920,   0,   0
64:                      64,      0,      74,    1972,     997,   0,   0
64:                      64,      0,    5594,    1288,  248688,   0,   0
64:                      64,      0,     617,    1925,    1276,   0,   0
64:                      64,      0,     385,    1661,     491,   0,   0
64:                      64,      0,     145,    1901,    2891,   0,   0
64:                      64,      0,    9338,    1326,   25410,   0,   0
64:                      64,      0,      81,     973,     119,   0,   0
128:                    128,      0,    1321,     570,   21862,   0,   0
128:                    128,      0,     359,     912,     829,   0,   0
128:                    128,      0,    1018,     873,   33029,   0,   0
128:                    128,      0,     378,    1017,     580,   0,   0
128:                    128,      0,    3527,     627,    3530,   0,   0
128:                    128,      0,     120,     900,     900,   0,   0
128:                    128,      0,     705,     814,   18136,   0,   0
128:                    128,      0,     132,     643,     422,   0,   0
256:                    256,      0,      48,     447,    1118,   0,   0
256:                    256,      0,     515,     400,    1136,   0,   0
256:                    256,      0,     604,     566,   58705,   0,   0
256:                    256,      0,     221,     454,     621,   0,   0
256:                    256,      0,       6,     249,       9,   0,   0
256:                    256,      0,     209,     406,     345,   0,   0
256:                    256,      0,     514,     221,    1218,   0,   0
256:                    256,      0,     149,     406,     601,   0,   0
512:                    512,      0,       4,      59,      30,   0,   0
512:                    512,      0,      17,      74,      21,   0,   0
512:                    512,      0,     171,     298,  189813,   0,   0
512:                    512,      0,      58,     180,     946,   0,   0
512:                    512,      0,       1,      34,       1,   0,   0
512:                    512,      0,      18,     157,      56,   0,   0
512:                    512,      0,     224,     126,     234,   0,   0
512:                    512,      0,       0,     175,     611,   0,   0
1024:                  1024,      0,       2,     102,    1967,   0,   0
1024:                  1024,      0,      19,      21,      19,   0,   0
1024:                  1024,      0,    1469,     171,   33875,   0,   0
1024:                  1024,      0,      16,     156,     658,   0,   0
1024:                  1024,      0,       2,      26,       2,   0,   0
1024:                  1024,      0,       0,     104,     424,   0,   0
1024:                  1024,      0,      55,      37,      68,   0,   0
1024:                  1024,      0,      17,      51,     168,   0,   0
2048:                  2048,      0,       3,      17,      33,   0,   0
2048:                  2048,      0,       1,       5,       1,   0,   0
2048:                  2048,      0,      56,      94,    3139,   0,   0
2048:                  2048,      0,      12,      68,    4730,   0,   0
2048:                  2048,      0,      32,      18,      67,   0,   0
2048:                  2048,      0,    1538,      48,    1654,   0,   0
2048:                  2048,      0,      20,      16,      20,   0,   0
2048:                  2048,      0,       1,      15,      52,   0,   0
4096:                  4096,      0,       0,       1,       8,   0,   0
4096:                  4096,      0,       1,       0,       1,   0,   0
4096:                  4096,      0,     556,      12,    2463,   0,   0
4096:                  4096,      0,       1,       1,     213,   0,   0
4096:                  4096,      0,      11,       0,      11,   0,   0
4096:                  4096,      0,       3,       1,       7,   0,   0
4096:                  4096,      0,      33,       1,      44,   0,   0
4096:                  4096,      0,       1,       3,       5,   0,   0
8192:                  8192,      0,       1,       0,       7,   0,   0
8192:                  8192,      0,       0,       0,       0,   0,   0
8192:                  8192,      0,      17,      12,     601,   0,   0
8192:                  8192,      0,      13,       0,     286,   0,   0
8192:                  8192,      0,       2,       0,       2,   0,   0
8192:                  8192,      0,       2,       2,       7,   0,   0
8192:                  8192,      0,       8,       0,       8,   0,   0
8192:                  8192,      0,       7,       3,     209,   0,   0
16384:                16384,      0,       0,       0,       0,   0,   0
16384:                16384,      0,       0,       0,       0,   0,   0
16384:                16384,      0,       0,       3,       5,   0,   0
16384:                16384,      0,       6,       3,     555,   0,   0
16384:                16384,      0,       1,       0,       1,   0,   0
16384:                16384,      0,      12,       7,      29,   0,   0
16384:                16384,      0,       2,       0,       2,   0,   0
16384:                16384,      0,       1,       9,      37,   0,   0
32768:                32768,      0,       1,       0,       1,   0,   0
32768:                32768,      0,       0,       0,       0,   0,   0
32768:                32768,      0,      14,       0,      14,   0,   0
32768:                32768,      0,       0,       1,     273,   0,   0
32768:                32768,      0,       0,       0,       0,   0,   0
32768:                32768,      0,       2,       2,      18,   0,   0
32768:                32768,      0,      64,       0,      64,   0,   0
32768:                32768,      0,       0,       0,       0,   0,   0
65536:                65536,      0,       1,       1,     105,   0,   0
65536:                65536,      0,       0,       0,       0,   0,   0
65536:                65536,      0,       1,       0,       1,   0,   0
65536:                65536,      0,       5,       1,     213,   0,   0
65536:                65536,      0,       0,       0,       0,   0,   0
65536:                65536,      0,       0,       9,      66,   0,   0
65536:                65536,      0,      41,       0,      41,   0,   0
65536:                65536,      0,       2,       0,       2,   0,   0
SLEEPQUEUE:              88,      0,     514,     881,     514,   0,   0
64 pcpu:                  8,      0,    2849,     991,    2849,   0,   0
Files:                   80,      0,      52,     856,    7746,   0,   0
filedesc0:             1104,      0,      43,      83,     966,   0,   0
TURNSTILE:              136,      0,     514,     346,     514,   0,   0
rl_entry:                40,      0,      52,    2027,      52,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
umtx_shm:                88,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1320,      0,      42,      66,     965,   0,   0
THREAD:                1280,      0,     472,      41,     746,   0,   0
cpuset:                  96,      0,     448,     700,     762,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 13037955,    8184,    1177,    8341,   0,   0
mbuf:                   256, 13037955,       1,    2023,     333,   0,   0
mbuf_cluster:          2048, 2037180,    9361,       9,    9361,   0,   0
mbuf_jumbo_page:       4096, 1018589,       0,       3,       7,   0,   0
mbuf_jumbo_9k:         9216, 905412,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 679056,       0,       0,       0,   0,   0
g_bio:                  384,      0,       0,     350,   15965,   0,   0
ttyinq:                 160,      0,      30,     162,      60,   0,   0
ttyoutq:                256,      0,      16,     179,      32,   0,   0
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0
nvme_request:           128,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
FPU_save_area:          512,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    2075,     264,   0,   0
VNODE:                  472,      0,     515,     101,     538,   0,   0
VNODEPOLL:              120,      0,       0,       0,       0,   0,   0
BUF TRIE:               144,      0,       0,  105948,       0,   0,   0
NAMEI:                 1024,      0,       2,     102,   19887,   0,   0
rentr:                   24,      0,       0,       0,       0,   0,   0
S VFS Cache:            108,      0,     445,     570,    1364,   0,   0
STS VFS Cache:          148,      0,       2,      76,       2,   0,   0
L VFS Cache:            328,      0,       0,       0,       0,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       3,      25,       3,   0,   0
Mountpoints:            872,      0,      17,      63,      17,   0,   0
AIO:                    224,      0,       0,       0,       0,   0,   0
AIOP:                    32,      0,       0,       0,       0,   0,   0
AIOCB:                  752,      0,       0,       0,       0,   0,   0
AIOL:                   128,      0,       0,       0,       0,   0,   0
AIOLIO:                 280,      0,       0,       0,       0,   0,   0
pipe:                   760,      0,       2,     103,     469,   0,   0
reference_cache:         40,      0,       6,    2073,    2138,   0,   0
reference_history_cache:      8,      0,       6,    2235,    2138,   0,   0
range_seg_cache:         64,      0,    2838,    1998,    4829,   0,   0
zio_cache:              984,      0,       8,    1060,  151221,   0,   0
zio_link_cache:          48,      0,       0,    3071,   78506,   0,   0
zio_buf_512:            512,      0,    1079,     139,  114537,   0,   0
zio_data_buf_512:       512,      0,      70,     133,     976,   0,   0
zio_buf_1024:          1024,      0,      99,     917,    2497,   0,   0
zio_data_buf_1024:     1024,      0,      67,      37,      78,   0,   0
zio_buf_1536:          1536,      0,      32,      50,     107,   0,   0
zio_data_buf_1536:     1536,      0,      49,      33,      53,   0,   0
zio_buf_2048:          2048,      0,      58,      48,    1267,   0,   0
zio_data_buf_2048:     2048,      0,      29,      11,      34,   0,   0
zio_buf_2560:          2560,      0,      79,      14,     237,   0,   0
zio_data_buf_2560:     2560,      0,      25,       5,      32,   0,   0
zio_buf_3072:          3072,      0,      65,      19,     219,   0,   0
zio_data_buf_3072:     3072,      0,      14,      10,      16,   0,   0
zio_buf_3584:          3584,      0,      43,       7,     129,   0,   0
zio_data_buf_3584:     3584,      0,      13,       2,      16,   0,   0
zio_buf_4096:          4096,      0,      27,      22,     440,   0,   0
zio_data_buf_4096:     4096,      0,      12,       1,      15,   0,   0
zio_buf_5120:          5120,      0,       2,      13,      94,   0,   0
zio_data_buf_5120:     5120,      0,      12,       1,      15,   0,   0
zio_buf_6144:          6144,      0,       1,       9,      47,   0,   0
zio_data_buf_6144:     6144,      0,      17,       0,      17,   0,   0
zio_buf_7168:          7168,      0,       3,       9,      94,   0,   0
zio_data_buf_7168:     7168,      0,      15,       0,      15,   0,   0
zio_buf_8192:          8192,      0,       4,      11,      57,   0,   0
zio_data_buf_8192:     8192,      0,      14,       1,      22,   0,   0
zio_buf_10240:        10240,      0,       3,      10,      73,   0,   0
zio_data_buf_10240:   10240,      0,      15,       1,      17,   0,   0
zio_buf_12288:        12288,      0,       1,      10,      35,   0,   0
zio_data_buf_12288:   12288,      0,      16,       1,      17,   0,   0
zio_buf_14336:        14336,      0,       1,       5,      17,   0,   0
zio_data_buf_14336:   14336,      0,      17,       0,      17,   0,   0
zio_buf_16384:        16384,      0,     308,       9,     783,   0,   0
zio_data_buf_16384:   16384,      0,      12,       0,      12,   0,   0
zio_buf_20480:        20480,      0,       0,       9,      27,   0,   0
zio_data_buf_20480:   20480,      0,      18,       0,      18,   0,   0
zio_buf_24576:        24576,      0,       0,       9,      15,   0,   0
zio_data_buf_24576:   24576,      0,      16,       1,      18,   0,   0
zio_buf_28672:        28672,      0,       1,       4,      15,   0,   0
zio_data_buf_28672:   28672,      0,      13,       0,      13,   0,   0
zio_buf_32768:        32768,      0,       2,       8,      39,   0,   0
zio_data_buf_32768:   32768,      0,      12,       0,      12,   0,   0
zio_buf_40960:        40960,      0,       0,       2,       2,   0,   0
zio_data_buf_40960:   40960,      0,      16,       0,      16,   0,   0
zio_buf_49152:        49152,      0,       0,       5,       9,   0,   0
zio_data_buf_49152:   49152,      0,      18,       0,      18,   0,   0
zio_buf_57344:        57344,      0,       1,       5,       7,   0,   0
zio_data_buf_57344:   57344,      0,      10,       0,      10,   0,   0
zio_buf_65536:        65536,      0,       0,       5,       9,   0,   0
zio_data_buf_65536:   65536,      0,      14,       0,      14,   0,   0
zio_buf_81920:        81920,      0,       0,       1,       1,   0,   0
zio_data_buf_81920:   81920,      0,      30,       0,      30,   0,   0
zio_buf_98304:        98304,      0,       0,       4,       4,   0,   0
zio_data_buf_98304:   98304,      0,      46,       0,      46,   0,   0
zio_buf_114688:      114688,      0,       0,      11,      24,   0,   0
zio_data_buf_114688: 114688,      0,       4,       0,       4,   0,   0
zio_buf_131072:      131072,      0,      35,       5,     258,   0,   0
zio_data_buf_131072: 131072,      0,     102,       0,     102,   0,   0
zio_buf_163840:      163840,      0,       0,       0,       0,   0,   0
zio_data_buf_163840: 163840,      0,       0,       0,       0,   0,   0
zio_buf_196608:      196608,      0,       0,       0,       0,   0,   0
zio_data_buf_196608: 196608,      0,       0,       0,       0,   0,   0
zio_buf_229376:      229376,      0,       0,       0,       0,   0,   0
zio_data_buf_229376: 229376,      0,       0,       0,       0,   0,   0
zio_buf_262144:      262144,      0,       0,       0,       0,   0,   0
zio_data_buf_262144: 262144,      0,       0,       0,       0,   0,   0
zio_buf_327680:      327680,      0,       0,       0,       0,   0,   0
zio_data_buf_327680: 327680,      0,       0,       0,       0,   0,   0
zio_buf_393216:      393216,      0,       0,       0,       0,   0,   0
zio_data_buf_393216: 393216,      0,       0,       0,       0,   0,   0
zio_buf_458752:      458752,      0,       0,       0,       0,   0,   0
zio_data_buf_458752: 458752,      0,       0,       0,       0,   0,   0
zio_buf_524288:      524288,      0,       0,       0,       0,   0,   0
zio_data_buf_524288: 524288,      0,       0,       0,       0,   0,   0
zio_buf_655360:      655360,      0,       0,       0,       0,   0,   0
zio_data_buf_655360: 655360,      0,       0,       0,       0,   0,   0
zio_buf_786432:      786432,      0,       0,       0,       0,   0,   0
zio_data_buf_786432: 786432,      0,       0,       0,       0,   0,   0
zio_buf_917504:      917504,      0,       0,       0,       0,   0,   0
zio_data_buf_917504: 917504,      0,       0,       0,       0,   0,   0
zio_buf_1048576:     1048576,      0,       0,       0,       0,   0,   0
zio_data_buf_1048576: 1048576,      0,       0,       0,       0,   0,   0
zio_buf_1310720:     1310720,      0,       0,       0,       0,   0,   0
zio_data_buf_1310720: 1310720,      0,       0,       0,       0,   0,   0
zio_buf_1572864:     1572864,      0,       0,       0,       0,   0,   0
zio_data_buf_1572864: 1572864,      0,       0,       0,       0,   0,   0
zio_buf_1835008:     1835008,      0,       0,       0,       0,   0,   0
zio_data_buf_1835008: 1835008,      0,       0,       0,       0,   0,   0
zio_buf_2097152:     2097152,      0,       0,       0,       0,   0,   0
zio_data_buf_2097152: 2097152,      0,       0,       0,       0,   0,   0
zio_buf_2621440:     2621440,      0,       0,       0,       0,   0,   0
zio_data_buf_2621440: 2621440,      0,       0,       0,       0,   0,   0
zio_buf_3145728:     3145728,      0,       0,       0,       0,   0,   0
zio_data_buf_3145728: 3145728,      0,       0,       0,       0,   0,   0
zio_buf_3670016:     3670016,      0,       0,       0,       0,   0,   0
zio_data_buf_3670016: 3670016,      0,       0,       0,       0,   0,   0
zio_buf_4194304:     4194304,      0,       0,       0,       0,   0,   0
zio_data_buf_4194304: 4194304,      0,       0,       0,       0,   0,   0
zio_buf_5242880:     5242880,      0,       0,       0,       0,   0,   0
zio_data_buf_5242880: 5242880,      0,       0,       0,       0,   0,   0
zio_buf_6291456:     6291456,      0,       0,       0,       0,   0,   0
zio_data_buf_6291456: 6291456,      0,       0,       0,       0,   0,   0
zio_buf_7340032:     7340032,      0,       0,       0,       0,   0,   0
zio_data_buf_7340032: 7340032,      0,       0,       0,       0,   0,   0
zio_buf_8388608:     8388608,      0,       0,       0,       0,   0,   0
zio_data_buf_8388608: 8388608,      0,       0,       0,       0,   0,   0
zio_buf_10485760:    10485760,      0,       0,       0,       0,   0,   0
zio_data_buf_10485760: 10485760,      0,       0,       0,       0,   0,   0
zio_buf_12582912:    12582912,      0,       0,       0,       0,   0,   0
zio_data_buf_12582912: 12582912,      0,       0,       0,       0,   0,   0
zio_buf_14680064:    14680064,      0,       0,       0,       0,   0,   0
zio_data_buf_14680064: 14680064,      0,       0,       0,       0,   0,   0
zio_buf_16777216:    16777216,      0,       0,       0,       0,   0,   0
zio_data_buf_16777216: 16777216,      0,       0,       0,       0,   0,   0
lz4_ctx:              16384,      0,       0,       8,     660,   0,   0
sa_cache:               144,      0,     445,     311,     467,   0,   0
dnode_t:                960,      0,    1173,      67,    4240,   0,   0
arc_buf_hdr_t_full:     344,      0,     969,     142,   58035,   0,   0
arc_buf_hdr_t_l2only:     88,      0,       0,       0,       0,   0,   0
arc_buf_t:               56,      0,     860,    1696,   59032,   0,   0
dmu_buf_impl_t:         344,      0,    1649,     122,   58888,   0,   0
zil_lwb_cache:          192,      0,       2,     138,       5,   0,   0
zfs_znode_cache:        264,      0,     445,     110,     467,   0,   0
procdesc:               136,      0,       0,       0,       0,   0,   0
ksiginfo:               112,      0,      69,     981,      71,   0,   0
itimer:                 352,      0,       0,       0,       0,   0,   0
KNOTE:                  128,      0,       2,     524,      18,   0,   0
socket:                 864, 1046992,      14,      90,    2964,   0,   0
unpcb:                  240, 1046992,       7,     489,    2880,   0,   0
ipq:                     56,  63687,       0,       0,       0,   0,   0
udp_inpcb:              464, 1046992,       3,     229,      65,   0,   0
udpcb:                   32, 1047056,       3,    2105,      65,   0,   0
tcp_inpcb:              464, 1046992,      11,     197,      12,   0,   0
tcpcb:                 1016, 1046992,       4,      88,      12,   0,   0
tcptw:                   88,  27810,       7,     499,       7,   0,   0
syncache:               168,  15364,       0,      69,       1,   0,   0
hostcache:               96,  15375,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
tcpreass:                40, 127413,       0,       0,       0,   0,   0
sctp_ep:               1488, 1046992,       0,       0,       0,   0,   0
sctp_asoc:             2408,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     830,       4,   0,   0
sctp_raddr:             736,  80000,       0,       0,       0,   0,   0
sctp_chunk:             152, 400010,       0,       0,       0,   0,   0
sctp_readq:             160, 400008,       0,       0,       0,   0,   0
sctp_stream_msg_out:    112, 400015,       0,       0,       0,   0,   0
sctp_asconf:             40, 400059,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
udplite_inpcb:          464, 1046992,       0,       0,       0,   0,   0
ripcb:                  464, 1046992,       0,       0,       0,   0,   0
rtentry:                208,      0,       9,     314,      10,   0,   0
selfd:                   64,      0,      34,    2012,    1124,   0,   0
SWAPMETA:               288, 4074369,       0,       0,       0,   0,   0
autofs_request:        5280,      0,       0,       1,       5,   0,   0
autofs_node:            208,      0,      17,      59,      17,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq23: ehci0                        1488          2
cpu0:timer                          5428          8
irq257: mps0                         571          1
irq258: igb0:que 0                    45          0
irq259: igb0:que 1                    40          0
irq260: igb0:que 2                    81          0
irq261: igb0:que 3                    29          0
irq262: igb0:que 4                    45          0
irq263: igb0:que 5                    51          0
irq264: igb0:que 6                   159          0
irq265: igb0:que 7                    32          0
irq266: igb0:link                      2          0
irq294: ahci1                       3498          5
irq295: ahci2                         98          0
cpu7:timer                          4393          6
cpu1:timer                          5635          8
cpu3:timer                          6277          9
cpu6:timer                          5276          7
cpu4:timer                          6312          9
cpu2:timer                          4815          7
cpu5:timer                          5980          8
Total                              50255         71

------------------------------------------------------------------------
pstat -T

 52/1046990 files
0M/65535M swap space

------------------------------------------------------------------------
pstat -s

Device          1K-blocks     Used    Avail Capacity
/dev/zvol/zroot/swap  67108736        0 67108736     0%

------------------------------------------------------------------------
iostat

       tty            ada0             ada1             ada2             cpu
 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   0    34 14.21  39  0.54  13.62  37  0.49  22.71   1  0.02   0  0  3  0 97

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME   

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME   

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME   


------------------------------------------------------------------------
ipcs -T

msginfo:
	msgmax:        16384	(max characters in a message)
	msgmni:           40	(# of message queues)
	msgmnb:         2048	(max characters in a message queue)
	msgtql:           40	(max # of messages in system)
	msgssz:            8	(size of a message segment)
	msgseg:         2048	(# of message segments in system)

shminfo:
	shmmax:    536870912	(max shared memory segment size)
	shmmin:            1	(min shared memory segment size)
	shmmni:          192	(max number of shared memory identifiers)
	shmseg:          128	(max shared memory segments per process)
	shmall:       131072	(max amount of shared memory in pages)

seminfo:
	semmni:           50	(# of semaphore identifiers)
	semmns:          340	(# of semaphores in system)
	semmnu:          150	(# of undo structures in system)
	semmsl:          340	(max # of semaphores per id)
	semopm:          100	(max # of operations per semop call)
	semume:           50	(max # of undo entries per process)
	semusz:          632	(size in bytes of undo structure)
	semvmx:        32767	(semaphore maximum value)
	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        3         0         3         0         1         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         4
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         8         2         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0        21
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
       14         1         6         3         1         1         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
        0         0         0         0         0         0         6         4

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
        0         0         0         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         0
    Mknod    Fsstat    Fsinfo  PathConf    Commit
        0         0         0         0         0
Server Ret-Failed
                0
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
        0         0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
        0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
	93 packets sent
		59 data packets (5802 bytes)
		0 data packets (0 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		19 ack-only packets (3 delayed)
		0 URG only packets
		0 window probe packets
		0 window update packets
		15 control packets
	117 packets received
		75 acks (for 5810 bytes)
		0 duplicate acks
		0 acks for unsent data
		56 packets (7173 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		0 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	8 connection requests
	1 connection accept
	0 bad connection attempts
	0 listen queue overflows
	0 ignored RSTs in the windows
	9 connections established (including accepts)
	1 connection closed (including 0 drops)
		0 connections updated cached RTT on close
		0 connections updated cached RTT variance on close
		0 connections updated cached ssthresh on close
	0 embryonic connections dropped
	75 segments updated rtt (of 68 attempts)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	0 correct ACK header predictions
	5 correct data packet header predictions
	1 syncache entry added
		0 retransmitted
		0 dupsyn
		0 dropped
		1 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	1 cookie sent
	1 cookie received
	0 hostcache entries added
		0 bucket overflow
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	0 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
	0 packets with ECN CE bit set
	0 packets with ECN ECT(0) bit set
	0 packets with ECN ECT(1) bit set
	0 successful ECN handshakes
	0 times ECN reduced the congestion window
TCP connection count by state:
	0 connections in CLOSED state
	2 connections in LISTEN state
	0 connections in SYN_SENT state
	0 connections in SYN_RCVD state
	2 connections in ESTABLISHED state
	0 connections in CLOSE_WAIT state
	0 connections in FIN_WAIT_1 state
	0 connections in CLOSING state
	0 connections in LAST_ACK state
	0 connections in FIN_WAIT_2 state
	7 connections in TIME_WAIT state
udp:
	27 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	0 dropped due to no socket
	0 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	27 delivered
	28 datagrams output
	0 times multicast source filter matched
ip:
	144 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	0 fragments received
	0 fragments dropped (dup or out of space)
	0 fragments dropped after timeout
	0 packets reassembled ok
	144 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	0 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	121 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	0 output datagrams fragmented
	0 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
icmp:
	0 calls to icmp_error
	0 errors not generated in response to an icmp message
	0 messages with bad code fields
	0 messages less than the minimum length
	0 messages with bad checksum
	0 messages with bad length
	0 multicast echo requests ignored
	0 multicast timestamp requests ignored
	0 message responses generated
	0 invalid return addresses
	0 no return routes
ipsec:
	0 inbound packets violated process security policy
	0 inbound packets failed due to insufficient memory
	0 invalid inbound packets
	0 outbound packets violated process security policy
	0 outbound packets with no SA available
	0 outbound packets failed due to insufficient memory
	0 outbound packets with no route available
	0 invalid outbound packets
	0 outbound packets with bundled SAs
	0 mbufs coalesced during clone
	0 clusters coalesced during clone
	0 clusters copied during clone
	0 mbufs inserted during makespace
ah:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets dropped; bad authentication detected
	0 packets dropped; bad authentication length
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
esp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 packets dropped; bad ilen
	0 replay counter wraps
	0 packets dropped; bad encryption detected
	0 packets dropped; bad authentication detected
	0 possible replay packets detected
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 tunnel sanity check failures
ipcomp:
	0 packets shorter than header shows
	0 packets dropped; protocol family not supported
	0 packets dropped; no TDB
	0 packets dropped; bad KCR
	0 packets dropped; queue full
	0 packets dropped; no transform
	0 replay counter wraps
	0 packets in
	0 packets out
	0 packets dropped; invalid TDB
	0 bytes in
	0 bytes out
	0 packets dropped; larger than IP_MAXPACKET
	0 packets blocked due to policy
	0 crypto processing failures
	0 packets sent uncompressed; size < compr. algo. threshold
	0 packets sent uncompressed; compression was useless
arp:
	4 ARP requests sent
	0 ARP replies sent
	0 ARP requests received
	2 ARP replies received
	2 ARP packets received
	1 total packet dropped due to no ARP entry
	0 ARP entrys timed out
	0 Duplicate IPs seen
pfkey:
	0 requests sent from userland
	0 bytes sent from userland
	0 messages with invalid length field
	0 messages with invalid version field
	0 messages with invalid message type field
	0 messages too short
	0 messages with memory allocation failure
	0 messages with duplicate extension
	0 messages with invalid extension type
	0 messages with invalid sa type
	0 messages with invalid address extension
	0 requests sent to userland
	0 bytes sent to userland
	0 messages toward single socket
	0 messages toward all sockets
	0 messages toward registered sockets
	0 messages with memory allocation failure

------------------------------------------------------------------------
netstat -m

8185/3200/11385 mbufs in use (current/cache/total)
8184/1186/9370/2037180 mbuf clusters in use (current/cache/total/max)
8184/1177 mbuf+clusters out of packet secondary zone in use (current/cache)
0/3/3/1018589 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/905412 9k jumbo clusters in use (current/cache/total/max)
0/0/0/679056 16k jumbo clusters in use (current/cache/total/max)
18414K/3184K/21598K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffff8014d2ccbe8 tcp4       0      0 172.27.10.3.781    172.27.10.2.2049   ESTABLISHED
fffff8014d2fdd10 tcp4       0      0 172.27.10.3.780    172.27.10.2.1020   TIME_WAIT
fffff8014d2b3b58 tcp4       0      0 172.27.10.3.779    172.27.10.2.2049   TIME_WAIT
fffff8014d2fdd68 tcp4       0      0 172.27.10.3.778    172.27.10.2.1020   TIME_WAIT
fffff8014d2dd160 tcp4       0      0 172.27.10.3.777    172.27.10.2.1020   TIME_WAIT
fffff8014d2bd4d0 tcp4       0      0 172.27.10.3.942    172.27.10.2.1020   TIME_WAIT
fffff8014d2c4840 tcp4       0      0 172.27.10.3.824    172.27.10.2.1020   TIME_WAIT
fffff8014d2b3bb0 tcp4       0      0 172.27.10.3.603    172.27.10.2.1020   TIME_WAIT
fffff8001427c7f0 tcp4       0      0 172.27.10.3.6666   172.27.10.112.5279 ESTABLISHED
fffff80035fb53f8 tcp4       0      0 *.7745             *.*                LISTEN
fffff80035f54000 tcp4       0      0 *.6666             *.*                LISTEN
fffff800147a0570 udp4       0      0 *.7745             *.*                
fffff80035ca4740 udp4       0      0 *.612              *.*                
fffff80035ca4910 udp4       0      0 *.514              *.*                
Active UNIX domain sockets
Address          Type   Recv-Q Send-Q            Inode             Conn             Refs          Nextref Addr
fffff80035c70b40 stream      0      0 fffff80035c583b0                0                0                0 /var/run/devd.pipe
fffff80035e115a0 dgram       0      0                0 fffff80035cc64b0                0 fffff80035e11690
fffff80035e11690 dgram       0      0                0 fffff80035cc64b0                0 fffff80035cc63c0
fffff80035cc63c0 dgram       0      0                0 fffff80035cc64b0                0                0
fffff80035cc64b0 dgram       0      0 fffff80014292ce8                0 fffff80035e115a0                0 /var/run/logpriv
fffff80035cc65a0 dgram       0      0 fffff80014293000                0                0                0 /var/run/log
fffff80035c70a50 seqpac      0      0 fffff80035c581d8                0                0                0 /var/run/devd.seqpacket.pipe

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address         
tcp4  0/0/5                            *.7745                 
tcp4  0/0/128                          *.6666                 
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe

------------------------------------------------------------------------
fstat

fstat: procstat_getprocs()

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #0 r308668M: Tue Nov 15 19:46:51 MSK 2016
    root at nostromo.deiter.local:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
WARNING: WITNESS option enabled, expect reduced performance.
VT(efifb): resolution 800x600
CPU: Intel(R) Atom(TM) CPU  C2758  @ 2.40GHz (2400.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33207201792 (31668 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
random: entropy device external interface
kbd0 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f543c0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> mem 0xdf3e0000-0xdf3fffff irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
vgapci0: <VGA-compatible display> port 0xd000-0xd07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 16 at device 0.0 on pci2
vgapci0: Boot video device
pcib3: <ACPI PCI-PCI bridge> mem 0xdf3c0000-0xdf3dffff irq 16 at device 2.0 on pci0
pci3: <ACPI PCI bus> on pcib3
ahci0: <Marvell 88SE9215 AHCI SATA controller> port 0xc050-0xc057,0xc040-0xc043,0xc030-0xc037,0xc020-0xc023,0xc000-0xc01f mem 0xdf210000-0xdf2107ff irq 17 at device 0.0 on pci3
ahci0: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
pcib4: <ACPI PCI-PCI bridge> mem 0xdf3a0000-0xdf3bffff irq 20 at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib4
mps0: <Avago Technologies (LSI) SAS2008> port 0xb000-0xb0ff mem 0xdf1c0000-0xdf1c3fff,0xdf180000-0xdf1bffff irq 22 at device 0.0 on pci4
mps0: Firmware: 20.00.04.00, Driver: 21.01.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pci0: <processor> at device 11.0 (no driver attached)
pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached)
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0c0-0xe0df mem 0xdf360000-0xdf37ffff,0xdf40c000-0xdf40ffff irq 20 at device 20.0 on pci0
igb0: Using MSIX interrupts with 9 vectors
igb0: Ethernet address: 0c:c4:7a:4e:36:b2
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: Bound queue 4 to cpu 4
igb0: Bound queue 5 to cpu 5
igb0: Bound queue 6 to cpu 6
igb0: Bound queue 7 to cpu 7
igb0: netmap queues/slots: TX 8/1024, RX 8/1024
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe0a0-0xe0bf mem 0xdf340000-0xdf35ffff,0xdf408000-0xdf40bfff irq 21 at device 20.1 on pci0
igb1: Using MSIX interrupts with 9 vectors
igb1: Ethernet address: 0c:c4:7a:4e:36:b3
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
igb1: Bound queue 4 to cpu 4
igb1: Bound queue 5 to cpu 5
igb1: Bound queue 6 to cpu 6
igb1: Bound queue 7 to cpu 7
igb1: netmap queues/slots: TX 8/1024, RX 8/1024
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe080-0xe09f mem 0xdf320000-0xdf33ffff,0xdf404000-0xdf407fff irq 22 at device 20.2 on pci0
igb2: Using MSIX interrupts with 9 vectors
igb2: Ethernet address: 0c:c4:7a:4e:36:b4
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
igb2: Bound queue 4 to cpu 4
igb2: Bound queue 5 to cpu 5
igb2: Bound queue 6 to cpu 6
igb2: Bound queue 7 to cpu 7
igb2: netmap queues/slots: TX 8/1024, RX 8/1024
igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe060-0xe07f mem 0xdf300000-0xdf31ffff,0xdf400000-0xdf403fff irq 23 at device 20.3 on pci0
igb3: Using MSIX interrupts with 9 vectors
igb3: Ethernet address: 0c:c4:7a:4e:36:b5
igb3: Bound queue 0 to cpu 0
igb3: Bound queue 1 to cpu 1
igb3: Bound queue 2 to cpu 2
igb3: Bound queue 3 to cpu 3
igb3: Bound queue 4 to cpu 4
igb3: Bound queue 5 to cpu 5
igb3: Bound queue 6 to cpu 6
igb3: Bound queue 7 to cpu 7
igb3: netmap queues/slots: TX 8/1024, RX 8/1024
ehci0: <Intel Avoton USB 2.0 controller> mem 0xdf417000-0xdf4173ff irq 23 at device 22.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
ahci1: <Intel Avoton AHCI SATA controller> port 0xe150-0xe157,0xe140-0xe143,0xe130-0xe137,0xe120-0xe123,0xe040-0xe05f mem 0xdf416000-0xdf4167ff irq 19 at device 23.0 on pci0
ahci1: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
ahcich6: <AHCI channel> at channel 2 on ahci1
ahcich7: <AHCI channel> at channel 3 on ahci1
ahci2: <Intel Avoton AHCI SATA controller> port 0xe110-0xe117,0xe100-0xe103,0xe0f0-0xe0f7,0xe0e0-0xe0e3,0xe020-0xe03f mem 0xdf415000-0xdf4157ff irq 19 at device 24.0 on pci0
ahci2: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich8: <AHCI channel> at channel 0 on ahci2
ahcich9: <AHCI channel> at channel 1 on ahci2
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
ugen0.1: <Intel EHCI root HUB> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ada0 at ahcich4 bus 0 scbus5 target 0 lun 0
ada0: <ST9500420AS 0003LVM1> ATA8-ACS SATA 2.x device
ada0: Serial Number 5VJEV436
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 476940MB (976773168 512 byte sectors)
ada1 at ahcich5 bus 0 scbus6 target 0 lun 0
ada1: <ST9500420AS 0003LVM1> ATA8-ACS SATA 2.x device
ada1: Serial Number 5VJEX1PN
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 476940MB (976773168 512 byte sectors)
ada2 at ahcich8 bus 0 scbus9 target 0 lun 0
ada2: <SanDisk SDSSDHII120G X31200RL> ACS-2 ATA SATA 3.x device
ada2: Serial Number 154301401735
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada2: Command Queueing enabled
ada2: 114473MB (234441648 512 byte sectors)
ada2: quirks=0x1<4K>
ada3 at ahcich9 bus 0 scbus10 target 0 lun 0
ada3: <SanDisk SDSSDHII120G X31200RL> ACS-2 ATA SATA 3.x device
ada3: Serial Number 154301400949
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada3: Command Queueing enabled
ada3: 114473MB (234441648 512 byte sectors)
ada3: quirks=0x1<4K>
(da0:mps0:0:0:0): UNMAPPED
(da1:mps0:0:da0 at mps0 bus 0 scbus4 target 0 lun 0
da0: <SEAGATE ST31000424SS 0005> Fixed Direct Access SPC-3 SCSI device
da0: Serial Number 9WK455XV0000C1361N2R
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 953869MB (1953525168 512 byte sectors)
1:0): UNMAPPED
(da2:da1 at mps0 bus 0 scbus4 target 1 lun 0
da1: <SEAGATE ST31000424SS 0005> Fixed Direct Access SPC-3 SCSI device
da1: Serial Number 9WK4783H00009136JAPT
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 953869MB (1953525168 512 byte sectors)
mps0:0:2:0): UNMAPPED
(da3:mps0:0:3:da2 at mps0 bus 0 scbus4 target 2 lun 0
da2: <SEAGATE ST31000424SS 0005> Fixed Direct Access SPC-3 SCSI device
da2: Serial Number 9WK4782A00009136JAXT
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 953869MB (1953525168 512 byte sectors)
0): UNMAPPED
(da4:mps0:0:da3 at mps0 bus 0 scbus4 target 3 lun 0
da3: <SEAGATE ST31000424SS 0005> Fixed Direct Access SPC-3 SCSI device
da3: Serial Number 9WK477HA00009136JA7G
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 953869MB (1953525168 512 byte sectors)
4:0): UNMAPPED
(da5:mps0:0:5:0): da4 at mps0 bus 0 scbus4 target 4 lun 0
da4: <SEAGATE ST31000424SS 0005> Fixed Direct Access SPC-3 SCSI device
da4: Serial Number 9WK3V95Q0000C135EK0P
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 953869MB (1953525168 512 byte sectors)
UNMAPPED
(da6:mps0:0:6:0): UNMAPPED
da5 at mps0 bus 0 scbus4 target 5 lun 0
da5: <SEAGATE ST31000424SS 0005> Fixed Direct Access SPC-3 SCSI device
da5: Serial Number 9WK46DRF00009136HD0U
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 953869MB (1953525168 512 byte sectors)
da6 at mps0 bus 0 scbus4 target 6 lun 0
da6: <SEAGATE ST31000424SS 0005> Fixed Direct Access SPC-3 SCSI device
da6: Serial Number 9WK43MHH0000C1337584
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 953869MB (1953525168 512 byte sectors)
(da7:mps0:0:7:0): UNMAPPED
SMP: AP CPU #7 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #5 Launched!
da7 at mps0 bus 0 scbus4 target 7 lun 0
da7: <SEAGATE ST31000424SS 0006> Fixed Direct Access SPC-3 SCSI device
da7: Serial Number 9WK2YRCM00009124RTAP
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 953869MB (1953525168 512 byte sectors)
Timecounter "TSC-low" frequency 1200029004 Hz quality 1000
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from zfs:zroot/r308668-p31-debug []...
Root mount waiting for: usbus0
Root mount waiting for: usbus0
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <vendor 0x8087 product 0x07db> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.3: <vendor 0x0409 product 0x005a> at usbus0
uhub2 on uhub1
uhub2: <vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 3> on usbus0
uhub2: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.4: <vendor 0x0409 product 0x005a> at usbus0
uhub3 on uhub1
uhub3: <vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 4> on usbus0
uhub3: 3 ports with 3 removable, self powered
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
ugen0.5: <Kingston DataTraveler 3.0> at usbus0
umass0 on uhub3
umass0: <Kingston DataTraveler 3.0, class 0/0, rev 2.10/1.00, addr 5> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0xc100
umass0:11:0: Attached to scbus11
da8 at umass-sim0 bus 0 scbus11 target 0 lun 0
da8: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
da8: Serial Number 6C626D7C22EABF3039350093
da8: 40.000MB/s transfers
da8: 15004MB (30728832 512 byte sectors)
da8: quirks=0x2<NO_6_BYTE>
ugen0.6: <vendor 0x0557 product 0x7000> at usbus0
uhub4 on uhub1
uhub4: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 6> on usbus0
uhub4: 4 ports with 3 removable, self powered
Root mount waiting for: usbus0
ugen0.7: <vendor 0x0557 product 0x2419> at usbus0
ukbd0 on uhub4
ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 7> on usbus0
kbd1 at ukbd0
eval: /usr/sbin/ip6addrctl: not found
ifconfig: inet6: bad value
igb0: link state changed to UP
ifconfig: inet6: bad value
ifconfig: inet6: bad value
ifconfig: inet6: bad value
Starting Network: lo0 igb0 igb1 igb2 igb3.
ifconfig: inet6: bad value
Starting Network: igb1.
ifconfig: inet6: bad value
Starting Network: igb2.
ifconfig: inet6: bad value
Starting Network: igb3.
ums0 on uhub4
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 7> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
route: bad keyword: inet6
route: usage: route [-46dnqtv] command [[modifiers] args]
route: bad keyword: inet6
route: usage: route [-46dnqtv] command [[modifiers] args]
route: bad keyword: inet6
route: usage: route [-46dnqtv] command [[modifiers] args]
route: bad keyword: inet6
route: usage: route [-46dnqtv] command [[modifiers] args]
route: bad keyword: inet6
route: usage: route [-46dnqtv] command [[modifiers] args]

Tue Nov 15 19:51:09 MSK 2016
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address	= 0xe0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff809fe487
stack pointer	        = 0x28:0xfffffe085f5d4560
frame pointer	        = 0x28:0xfffffe085f5d45b0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= resume, IOPL = 0
current process		= 0 (pkg)

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	GENERIC
machine	amd64
cpu	HAMMER
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	XENHVM
options	USB_DEBUG
options	ATH_ENABLE_11N
options	AH_AR5416_INTERRUPT_MITIGATION
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	SC_PIXEL_MODE
options	VESA
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	PCI_IOV
options	PCI_HP
options	ACPI_DMAR
options	DEVICE_NUMA
options	SMP
options	MALLOC_DEBUG_MAXZONES=8
options	WITNESS_SKIPSPIN
options	WITNESS
options	INVARIANT_SUPPORT
options	INVARIANTS
options	DEADLKRES
options	GDB
options	FULL_BUF_TRACKING
options	DDB
options	BUF_TRACKING
options	KDB_TRACE
options	KDB
options	RCTL
options	RACCT_DEFAULT_TO_DISABLED
options	RACCT
options	INCLUDE_CONFIG_FILE
options	DDB_CTF
options	KDTRACE_HOOKS
options	KDTRACE_FRAME
options	MAC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD10
options	COMPAT_FREEBSD9
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_RAID
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	MD_ROOT
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	TCP_HHOOK
options	TCP_OFFLOAD
options	IPSEC
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ahci
device	ata
device	mvs
device	siis
device	ahc
device	ahd
device	esp
device	hptiop
device	isp
device	mpt
device	mps
device	mpr
device	sym
device	trm
device	adv
device	adw
device	aic
device	bt
device	isci
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	amr
device	arcmsr
device	ciss
device	dpt
device	hptmv
device	hptnr
device	hptrr
device	hpt27xx
device	iir
device	ips
device	mly
device	twa
device	tws
device	aac
device	aacp
device	aacraid
device	ida
device	mfi
device	mlx
device	mrsas
device	pmspcv
device	twe
device	nvme
device	nvd
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	vt
device	vt_vga
device	vt_efifb
device	agp
device	cbb
device	pccard
device	cardbus
device	uart
device	ppc
device	ppbus
device	lpt
device	ppi
device	puc
device	bxe
device	de
device	em
device	igb
device	ix
device	ixv
device	ixl
device	ixlv
device	le
device	ti
device	txp
device	vx
device	miibus
device	ae
device	age
device	alc
device	ale
device	bce
device	bfe
device	bge
device	cas
device	dc
device	et
device	fxp
device	gem
device	hme
device	jme
device	lge
device	msk
device	nfe
device	nge
device	pcn
device	re
device	rl
device	sf
device	sge
device	sis
device	sk
device	ste
device	stge
device	tl
device	tx
device	vge
device	vr
device	wb
device	xl
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	an
device	ath
device	ath_pci
device	ath_hal
device	ath_rate_sample
device	ipw
device	iwi
device	iwn
device	malo
device	mwl
device	ral
device	wi
device	wpi
device	loop
device	random
device	padlock_rng
device	rdrand_rng
device	ether
device	vlan
device	tun
device	md
device	gif
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	ukbd
device	umass
device	sound
device	snd_cmi
device	snd_csa
device	snd_emu10kx
device	snd_es137x
device	snd_hda
device	snd_ich
device	snd_via8233
device	mmc
device	mmcsd
device	sdhci
device	virtio
device	virtio_pci
device	vtnet
device	virtio_blk
device	virtio_scsi
device	virtio_balloon
device	hyperv
device	xenpci
device	vmx
device	netmap
device	crypto

------------------------------------------------------------------------
ddb capture buffer




More information about the freebsd-current mailing list