Still seeing random vmem(I think) panics

Larry Rosenman ler at lerctr.org
Wed Aug 14 22:33:00 UTC 2013


I went to reboot the system and got this.....

I have 10 of them in the last 72 hours.

Ideas?


borg.lerctr.org dumped core - see /var/crash/vmcore.0

Wed Aug 14 17:01:14 CDT 2013

FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #20 r254262: Mon Aug 12 21:45:40 CDT 2013     root at borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
frame pointer	        = 0x28:0xffffff900d2cb950
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1331 (cupsd)
trap number		= 12
panic: page fault
cpuid = 7
Uptime: 1d3h40m18s
Dumping 7789 out of 64479 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel.old/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel.old/zfs.ko.symbols
Reading symbols from /boot/kernel.old/acl_nfs4.ko.symbols...done.
Loaded symbols for /boot/kernel.old/acl_nfs4.ko.symbols
Reading symbols from /boot/kernel.old/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel.old/opensolaris.ko.symbols
Reading symbols from /boot/kernel.old/linux.ko.symbols...done.
Loaded symbols for /boot/kernel.old/linux.ko.symbols
Reading symbols from /boot/kernel.old/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel.old/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel.old/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel.old/snd_spicds.ko.symbols
Reading symbols from /boot/kernel.old/sound.ko.symbols...done.
Loaded symbols for /boot/kernel.old/sound.ko.symbols
Reading symbols from /boot/kernel.old/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel.old/coretemp.ko.symbols
Reading symbols from /boot/kernel.old/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ichsmb.ko.symbols
Reading symbols from /boot/kernel.old/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel.old/smbus.ko.symbols
Reading symbols from /boot/kernel.old/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ichwd.ko.symbols
Reading symbols from /boot/kernel.old/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel.old/cpuctl.ko.symbols
Reading symbols from /boot/kernel.old/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel.old/crypto.ko.symbols
Reading symbols from /boot/kernel.old/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel.old/cryptodev.ko.symbols
Reading symbols from /boot/kernel.old/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtraceall.ko.symbols
Reading symbols from /boot/kernel.old/profile.ko.symbols...done.
Loaded symbols for /boot/kernel.old/profile.ko.symbols
Reading symbols from /boot/kernel.old/cyclic.ko.symbols...done.
Loaded symbols for /boot/kernel.old/cyclic.ko.symbols
Reading symbols from /boot/kernel.old/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtrace.ko.symbols
Reading symbols from /boot/kernel.old/systrace_freebsd32.ko.symbols...done.
Loaded symbols for /boot/kernel.old/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel.old/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel.old/systrace.ko.symbols
Reading symbols from /boot/kernel.old/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel.old/sdt.ko.symbols
Reading symbols from /boot/kernel.old/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel.old/lockstat.ko.symbols
Reading symbols from /boot/kernel.old/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel.old/fasttrap.ko.symbols
Reading symbols from /boot/kernel.old/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel.old/fbt.ko.symbols
Reading symbols from /boot/kernel.old/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtnfscl.ko.symbols
Reading symbols from /boot/kernel.old/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtmalloc.ko.symbols
Reading symbols from /boot/kernel.old/dtio.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtio.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel.old/fdescfs.ko.symbols...done.
Loaded symbols for /boot/kernel.old/fdescfs.ko.symbols
Reading symbols from /boot/kernel.old/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel.old/uhid.ko.symbols
Reading symbols from /boot/kernel.old/ums.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ums.ko.symbols
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel.old/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel.old/netgraph.ko.symbols
Reading symbols from /boot/kernel.old/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ng_ether.ko.symbols
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:236
236	pcpu.h: No such file or directory.
 	in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:236
#1  0xffffffff8051d780 in kern_reboot (howto=260)
     at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff8051db07 in panic (fmt=<value optimized out>)
     at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80780eca in trap_fatal (frame=<value optimized out>,
     eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:873
#4  0xffffffff807812c9 in trap_pfault (frame=0x0, usermode=0)
     at /usr/src/sys/amd64/amd64/trap.c:731
#5  0xffffffff807808c2 in trap (frame=0xffffff900d2cb880)
     at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff8076ae32 in calltrap ()
     at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff806fde34 in vm_map_entry_unlink (map=0xfffffe001e955000,
     entry=0xfffffe03325b9000) at /usr/src/sys/vm/vm_map.c:997
#8  0xffffffff806fd99e in vm_map_delete (map=0xfffffe001e955000,
     start=<value optimized out>, end=140737488355328)
     at /usr/src/sys/vm/vm_map.c:2741
#9  0xffffffff806ffb19 in vm_map_remove (map=0xfffffe001e955000,
     start=140737488355328, end=18446741875199397888)
     at /usr/src/sys/vm/vm_map.c:2903
#10 0xffffffff806fbe6d in vmspace_exit (td=0xfffffe001d5a9490)
     at /usr/src/sys/vm/vm_map.c:340
#11 0xffffffff804e9b5f in exit1 (td=0xfffffe001d5a9490,
     rv=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:322
#12 0xffffffff804e94fe in sys_sys_exit (td=<value optimized out>,
     uap=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:122
#13 0xffffffff807817c7 in amd64_syscall (td=0xfffffe001d5a9490, traced=0)
     at subr_syscall.c:134
#14 0xffffffff8076b11b in Xfast_syscall ()
     at /usr/src/sys/amd64/amd64/exception.S:391
#15 0x00000008025c96ea in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)

------------------------------------------------------------------------
ps -axl

  UID   PID  PPID CPU PRI NI    VSZ  RSS MWCHAN   STAT TT        TIME COMMAND
    0     0     0   0  -8  0      0    0 -        DLs   -    48:58.19 [kernel]
    0     1     0   0  21  0   9428    0 wait     DLs   -     0:00.39 [init]
    0     2     0   0 -16  0      0    0 ftcl     DL    -     0:00.00 [ftcleanup
    0     3     0   0 -16  0      0    0 crypto_w DL    -     0:00.00 [crypto]
    0     4     0   0 -16  0      0    0 crypto_r DL    -     0:00.00 [crypto re
    0     5     0   0 -16  0      0    0 -        DL    -     0:00.54 [fdc0]
    0     6     0   0  -8  0      0    0 zio->io_ DL    -     2:44.04 [zfskern]
    0     7     0   0 -16  0      0    0 waiting_ DL    -     0:00.00 [sctp_iter
    0     8     0   0 -16  0      0    0 ccb_scan DL    -     0:00.00 [xpt_thrd]
    0     9     0   0 -16  0      0    0 psleep   DL    -     0:00.11 [pagedaemo
    0    10     0   0 -16  0      0    0 audit_wo DL    -     0:00.00 [audit]
    0    11     0   0 155  0      0    0 -        RL    -  1055:16.45 [idle]
    0    12     0   0 -84  0      0    0 -        WL    -     5:32.65 [intr]
    0    13     0   0  -8  0      0    0 -        DL    -     3:31.63 [geom]
    0    14     0   0 -16  0      0    0 -        DL    -     0:13.36 [yarrow]
    0    15     0   0 -68  0      0    0 -        DL    -     0:14.34 [usb]
    0    16     0   0 -20  0      0    0 VBoxIS   DL    -     0:00.00 [TIMER]
    0    17     0   0 -16  0      0    0 psleep   DL    -     0:00.00 [vmdaemon]
    0    18     0   0 155  0      0    0 pgzero   DL    -     0:00.00 [pagezero]
    0    19     0   0 -16  0      0    0 psleep   DL    -     0:00.67 [bufdaemon
    0    20     0   0  16  0      0    0 syncer   DL    -     6:02.52 [syncer]
    0    21     0   0 -16  0      0    0 vlruwt   DL    -     0:03.86 [vnlru]
    0   858     1   0  20  0  13208    0 select   Ds    -     0:00.05 [devd]
    0   871     1   0  52  0  16584    0 select   Ds    -     0:00.00 [moused]
    0  1018     1   0  20  0  14380    0 zio->io_ Ds    -     0:00.56 [syslogd]
    0  1021     1   0 -52  0   6192 2104 nanslp   Ds    -     0:05.71 [watchdogd
    0  1090     1   0  20  0  16468    0 select   Ds    -     0:00.11 [rpcbind]
    0  1125     1   0  52  0  38964    0 select   Ds    -     0:00.02 [mountd]
    0  1131     1   0  52  0  36804    0 select   Ds    -     0:00.04 [nfsd]
    0  1137  1131   0  52  0  12228    0 rpcsvc   D     -     0:00.60 [nfsd]
    0  1141     1   0  52  0  32716    0 select   Ds    -     0:24.28 [apcupsd]
    0  1170     1   0  20  0  25268    0 select   Ds    -     0:04.48 [ntpd]
    0  1191     0   0 -16  0      0    0 sleep    DL    -     0:00.00 [ng_queue]
    0  1203     1   0  20  0  38844    0 nanslp   Ds    -     0:40.22 [perl]
    0  1207     1   0  20  0  30844    0 nanslp   D     -     0:00.61 [smartd]
   70  1215     1   0  20  0  87084    0 select   D     -     0:02.14 [postgres]
   70  1217  1215   0  20  0  87084    0 select   Ds    -     0:01.00 [postgres]
   70  1218  1215   0  20  0  87084    0 select   Ds    -     0:00.74 [postgres]
   70  1219  1215   0  20  0  87084    0 select   Ds    -     0:00.85 [postgres]
   70  1220  1215   0  20  0  87084    0 select   Ds    -     0:02.10 [postgres]
   70  1221  1215   0  20  0  46772    0 select   Ds    -     0:03.68 [postgres]
   26  1322     1   0  20  0  46668    0 select   Ds    -     0:00.04 [exim-4.80
    0  1331     1   0  20  0      0    0 -        REs   -     0:00.83 [cupsd]
1028 53784     1   0 155 19 118984    0 -        RN    -     3:48.21 [wcgrid_fa
1028 53787     1   0 155 19 110596    0 -        RN    -     3:13.08 [wcgrid_fa
    0 53788     1   0  21  0  81592    0 select   Ds    -     0:00.03 [sshd]
1002 53791 53788   0  20  0  81592    0 select   D     -     0:01.01 [sshd]
1002 53792 53791   0  20  0  16944    0 wait     Ds    -     0:00.01 [sh]
    0 53795 53792   0  20  0  51140    0 select   D     -     0:00.01 [sudo]
    0 53796 53795   0  31  0  16944    0 ttyin    D+    -     0:00.01 [sh]
    0 83940     1   0  52  0  16984    0 wait     Ds+   -     0:00.02 [sh]
    0 83943 83940   0  23  0  16984    0 wait     D+    -     0:00.00 [sh]
    0 83945 83943   0  23  0   8160    0 nanslp   D+    -     0:00.00 [sleep]
    0 84039 83940   0  52  0  16984    0 wait     D+    -     0:00.00 [sh]
    0 84044 84039   0  52  0  12260    0 kqread   D+    -     0:00.00 [pwait]

------------------------------------------------------------------------
vmstat -s

188486035 cpu context switches
  16891738 device interrupts
  12882066 software interrupts
609501168 traps
1959237280 system calls
        83 kernel threads created
     92989  fork() calls
    186023 vfork() calls
      4641 rfork() calls
         0 swap pager pageins
         0 swap pager pages paged in
         0 swap pager pageouts
         0 swap pager pages paged out
    134497 vnode pager pageins
    680603 vnode pager pages paged in
     25918 vnode pager pageouts
     52995 vnode pager pages paged out
         0 page daemon wakeups
         0 pages examined by the page daemon
     47730 pages reactivated
   5517586 copy-on-write faults
       912 copy-on-write optimized faults
492347700 zero fill pages zeroed
         0 zero fill pages prezeroed
        71 intransit blocking page faults
501029194 total VM faults taken
    133963 page faults requiring I/O
         0 pages affected by kernel thread creation
   8364551 pages affected by  fork()
  10797982 pages affected by vfork()
  15434414 pages affected by rfork()
         0 pages cached
625905899 pages freed
         0 pages freed by daemon
         0 pages freed by exiting processes
    176372 pages active
    238918 pages inactive
      4756 pages in VM cache
  12348974 pages wired down
   3310348 pages free
      4096 bytes per page
352251095 total name lookups
           cache hits (90% pos + 7% neg) system 0% per-directory
           deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

          Type InUse MemUse HighUse Requests  Size(s)
       kdtrace   501   115K       -   286744  64,256
          kenv    84    11K       -      112  16,32,64,128
        kqueue     2     1K       -     2432  256,512,2048
     proc-args    29     2K       -   175441  16,32,64,128,256
        DEVFSP     3     1K       -       38  64
         hhook     2     1K       -        2  256
       ithread   122    22K       -      122  32,128,256
        KTRACE   100    13K       -      100  128
        linker   401   197K       -      544  16,32,64,128,256,512,1024,2048,4096
         lockf    20     3K       -    60480  64,128
    loginclass     1     1K       -      550  64
         cache     1     1K       -        1  32
        devbuf 18685 36135K       -    19791  16,32,64,128,256,512,1024,2048,4096
          temp    58    34K       - 10356759  16,32,64,128,256,512,1024,2048,4096
        ip6ndp     5     1K       -        6  64,128
        module   301    38K       -      301  128
      mtx_pool     2    16K       -        2
           osd     9     1K       -       43  16,32,64,128
      pmchooks     1     1K       -        1  128
        NFS fh     4     1K       -       18  16,32
          pgrp    26     4K       -     4134  128
       session    24     3K       -     4017  128
          proc     2   256K       -        2
       subproc   237   311K       -   281140  512,4096
          cred   100    16K       -  4515916  64,256
        plimit     9     3K       -    10048  256
       uidinfo     7    33K       -      792  128
        sysctl     0     0K       -    33529  16,32,64
     sysctloid  6788   337K       -     6940  16,32,64,128
     sysctltmp     0     0K       -    91744  16,32,64,128,256,4096
       tidhash     1   256K       -        1
       callout     9  3208K       -        9
          umtx  1014   127K       -     1014  128
      p1003.1b     1     1K       -        1  16
          SWAP     0     0K       -       12  64
           bus   945    90K       -     5432  16,32,64,128,256,512,1024
        bus-sc   141   293K       -     2336  16,32,64,128,256,512,1024,2048,4096
       devstat    34    69K       -       34  32,4096
  eventhandler    90     8K       -       90  64,128
          kobj   180   720K       -     1010  4096
       Per-cpu     1     1K       -        1  32
          rman   360    41K       -      797  16,32,128
          sbuf     0     0K       -     3719  16,32,64,128,256,512,1024,2048,4096
        sglist     3     1K       -        3  32
         stack     0     0K       -        2  256
     taskqueue   109    17K       -      139  16,32,64,256,1024
        Unitno    22     2K       -    45310  32,64
          vmem     3  1153K       -       11  512,1024
      ioctlops     0     0K       -  1098650  16,32,64,128,256,512,1024,2048
        select   159    20K       -      159  128
           iov     0     0K       -  1390276  16,64,128,256,512
           msg     4    30K       -        4  2048,4096
           sem     4   106K       -        4  2048,4096
           shm     7    32K       -     3198  2048
           tty    20    20K       -       29  1024,2048
           pts     1     1K       -        8  256
      mbuf_tag     0     0K       -    23667  32,128
         shmfd     1     8K       -        1
        soname     6     1K       -    67266  16,32,128
           pcb    33  8341K       -     2548  16,32,128,1024,2048
     newnfsmnt     1     1K       -        1  1024
           acl     0     0K       -      145  4096
      vfscache     1 16384K       -        1
      vfs_hash     1  8192K       -        1
        vnodes     1     1K       -        1  256
     pfs_nodes    21     6K       -       21  256
   pfs_vncache    54     4K       -    15332  64
         mount   396    20K       -     1854  16,32,64,128,256,512
          GEOM   338    59K       -     3552  16,32,64,128,256,512,1024,2048
   vnodemarker     0     0K       -   172658  512
           BPF     3     1K       -        3  128
        acpica  1830   187K       -    52091  16,32,64,128,256,512,1024,2048
         ifnet     4     7K       -        4  128,2048
        ifaddr    48    15K       -       48  32,64,128,256,512,2048,4096
   ether_multi    47     3K       -       54  16,32,64
         clone     6     1K       -        6  128
        arpcom     2     1K       -        2  16
       lltable    12     5K       -       25  256,512
          UART     6     5K       -        6  16,1024
       CAM CCB    13    26K       -     1091  2048
      acpitask     1     8K       -        1
       acpisem    22     3K       -       22  128
      CAM path    22     1K       -       68  32
       acpidev    36     3K       -       36  64
      routetbl    35     5K       -    31140  32,64,128,256,512
          igmp     3     1K       -        3  256
      in_multi     2     1K       -        2  256
    CAM periph    16     4K       -       40  16,32,64,128,256
     CAM queue    31     8K       -      100  16,32,512
CAM dev queue     8     1K       -        8  32
     sctp_a_it     0     0K       -        3  16
      sctp_vrf     1     1K       -        1  64
      sctp_ifa     5     1K       -        5  128
      sctp_ifn     2     1K       -        2  128
     sctp_iter     0     0K       -        3  256
     hostcache     1    28K       -        1
      syncache     1    64K       -        1
   in6_mfilter     1     1K       -        1  1024
     in6_multi    27     4K       -       27  32,256
  ip6_moptions     2     1K       -        2  32,256
     raid_data     0     0K       -      552  32,128,256
           mld     3     1K       -        3  128
       CAM SIM     8     2K       -        8  256
       CAM XPT    58     4K       -      354  16,32,64,128,256,1024
       NFS FHA     1     2K       -        1  2048
           rpc    28     9K       -   229273  32,64,128,256,512,1024
audit_evclass   188     6K       -      229  32
     vm_pgdata     1  8192K       -        7  128
       UMAHash     5    42K       -       15  512,1024,2048,4096
       scsi_cd     0     0K       -       11  16
           USB    50   162K       -       56  16,32,64,128,256,512,1024,4096
        USBdev    76    26K       -      112  32,64,128,512,2048,4096
      pci_link    16     2K       -       16  32,64,128
     acpi_perf     8     1K       -        8  64
       memdesc     1     4K       -        1  4096
      atkbddev     2     1K       -        2  64
md_nvidia_data     0     0K       -       90  512
        kbdmux     7    18K       -        7  16,512,1024,2048
           LED     4     1K       -        4  16,128
   md_sii_data     0     0K       -       90  512
       CAM DEV    15    30K       -       24  2048
       ata_pci     1     1K       -        1  64
      acpiintr     1     1K       -        1  64
      ppbusdev     2     1K       -        2  256
        apmdev     1     1K       -        1  128
    madt_table     0     0K       -        1  4096
        isadev     5     1K       -        5  128
random_adaptors     1     1K       -        1  32
       entropy  1024    64K       -     1024  64
        DEVFS3   223    56K       -      282  256
       io_apic     2     4K       -        2  2048
        DEVFS1   194    97K       -      243  512
           MCA     8     1K       -        8  128
          cdev     8     2K       -        8  256
           msi     2     1K       -        2  128
      nexusdev     4     1K       -        4  16
         DEVFS    42     1K       -       43  16,128
      filedesc    95   233K       -   281056  16,32,64,128,2048,4096
filedesc_to_leader     2     1K       -     4845  64
         sigio     1     1K       -        1  64
      filecaps     0     0K       -       18  16,64
         linux    20     2K       -     8353  32,64
         mixer     1     4K       -        1  4096
        feeder    16     2K       -       18  32,128
    kstat_data     5     1K       -        5  64
       solaris 3722414 45963257K       - 429496517  16,32,64,128,256,512,1024,2048,4096
         futex     0     0K       -    53393  128
        spicds     7     1K       -        7  128
      envy24ht    15   195K       -       15  64,2048
        cpuctl     1     1K       -        9  64,4096
        crypto     1     1K       -        1  512
         xform     0     0K       -     9527  16,32
        cyclic    32     3K       -       32  16,64,128
      futex wp     0     0K       -    26711  32
           fbt     1   256K       -        1
      iprtheap    23    56K       -       23  32,64,128,256,2048
        nvidia   179   862K       -      182  16,32,64,128,256,512,1024,2048,4096
   fdesc_mount     1     1K       -        1  16
netgraph_node     4     1K       -        4  128,256
      netgraph     2     1K       -        2  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     172,       8,     172,   0,   0
UMA Zones:             1664,      0,     188,       0,     188,   0,   0
UMA Slabs:               80,      0, 1088327,     473,14366575,   0,   0
UMA RCntSlabs:           88,      0,    1945,      35,    1945,   0,   0
UMA Hash:               256,      0,       5,      10,      10,   0,   0
4 Bucket:                32,      0,     426,    5324, 4108981,33285,   0
8 Bucket:                64,      0,     586,   21362, 1602256,416395,   0
16 Bucket:              128,      0,     319,    5292,  123057, 319,   0
32 Bucket:              256,      0,    7244,    6706,  294466, 999,   0
64 Bucket:              512,      0,     809,    6391,  741894,27478,   0
128 Bucket:            1024,      0,    5145,    3731, 1582420,478309,   0
vmem btag:               56,      0,  652182,  103258,  988336,10662,   0
VM OBJECT:              256,      0,  189525,    2085, 5111404,   0,   0
RADIX NODE:             144,      0,  795442,   26249,46129201,  39,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,      13,     514,      13,   2,   0
MAP ENTRY:              128,      0,    1108,    3046,14837980,   0,   0
VMSPACE:                408,      0,      33,     381,  281080,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     288,       0,     288,   0,   0
16:                      16,      0,       5,     246,       5,   0,   0
16:                      16,      0,    2467,     545,    2622,   0,   0
16:                      16,      0,       2,     249,       2,   0,   0
16:                      16,      0,      90,     914,  503989,   0,   0
16:                      16,      0,      32,     972,   32986,   0,   0
16:                      16,      0,     651,     855,   91769,   0,   0
16:                      16,      0,      48,     956,      90,   0,   0
16:                      16,      0,  245446,    2040,22351211,   0,   0
32:                      32,      0,       2,     123,       2,   0,   0
32:                      32,      0,    2634,     741,    2732,   0,   0
32:                      32,      0,      10,     115,      10,   0,   0
32:                      32,      0,     105,    1395,  340076,   0,   0
32:                      32,      0,     250,    1000,   27674,   0,   0
32:                      32,      0,     336,    1164,   28769,   0,   0
32:                      32,      0,      76,     924,   48855,   0,   0
32:                      32,      0,  172142,    8483,28192547,   0,   0
64:                      64,      0,       9,     549,      13,   0,   0
64:                      64,      0,     324,     730,     340,   0,   0
64:                      64,      0,      14,     172,      14,   0,   0
64:                      64,      0,     770,    1028,  290809,   0,   0
64:                      64,      0,      74,    1352,  283785,   0,   0
64:                      64,      0,    9136,    1218, 2622013,   0,   0
64:                      64,      0,    1078,    1030,    5921,   0,   0
64:                      64,      0, 1147282,   29602,149282978,   0,   0
128:                    128,      0,      11,     764,      11,   0,   0
128:                    128,      0,    2007,     876,    3280,   0,   0
128:                    128,      0,      31,     992,    4139,   0,   0
128:                    128,      0,    1341,     798,   73481,   0,   0
128:                    128,      0,    1380,     635,   54872,   0,   0
128:                    128,      0,    2029,    1257,   80833,   0,   0
128:                    128,      0,       3,     276,       3,   0,   0
128:                    128,      0,  137189,    9782,64781032,   0,   0
256:                    256,      0,       2,      73,       2,   0,   0
256:                    256,      0,      20,     475,   10085,   0,   0
256:                    256,      0,       6,      69,       6,   0,   0
256:                    256,      0,     242,     463,     915,   0,   0
256:                    256,      0,     532,     413,    3481,   0,   0
256:                    256,      0,     393,     762, 2288033,   0,   0
256:                    256,      0,       1,      74,       1,   0,   0
256:                    256,      0,  192192,  191613,77229314,   0,   0
512:                    512,      0,       4,     316,  229245,   0,   0
512:                    512,      0,      39,     169,     219,   0,   0
512:                    512,      0,       0,     232,  172658,   0,   0
512:                    512,      0,     235,     197,    2418,   0,   0
512:                    512,      0,      18,     102,      18,   0,   0
512:                    512,      0,     403,     293, 2255668,   0,   0
512:                    512,      0,       0,       0,       0,   0,   0
512:                    512,      0,  935935,    3297,39073172,   0,   0
1024:                  1024,      0,       4,      36,       4,   0,   0
1024:                  1024,      0,       1,      39,     100,   0,   0
1024:                  1024,      0,       5,      23,       5,   0,   0
1024:                  1024,      0,       6,      74,    1720,   0,   0
1024:                  1024,      0,      20,      84,      27,   0,   0
1024:                  1024,      0,      26,     262, 3281265,   0,   0
1024:                  1024,      0,       1,      15,       1,   0,   0
1024:                  1024,      0,   10473,     151,  184746,   0,   0
2048:                  2048,      0,       6,      56,    3197,   0,   0
2048:                  2048,      0,      32,      40,    1119,   0,   0
2048:                  2048,      0,       1,       5,       1,   0,   0
2048:                  2048,      0,      15,      31,    3211,   0,   0
2048:                  2048,      0,      55,      99,  280961,   0,   0
2048:                  2048,      0,      24,      52,    1405,   0,   0
2048:                  2048,      0,       5,       1,       5,   0,   0
2048:                  2048,      0,   17353,     141,  500775,   0,   0
4096:                  4096,      0,       1,       0,       1,   0,   0
4096:                  4096,      0,       2,       0,       2,   0,   0
4096:                  4096,      0,     180,       3,    1010,   0,   0
4096:                  4096,      0,      12,       8,     159,   0,   0
4096:                  4096,      0,       6,       9,      47,   0,   0
4096:                  4096,      0,     111,      82, 1897800,   0,   0
4096:                  4096,      0,       0,       0,       0,   0,   0
4096:                  4096,      0,  234256,   90044,38442227,   0,   0
uint64 pcpu:              8,      0,    1386,     150,    1386,   0,   0
SLEEPQUEUE:              80,      0,     508,     763,     508,   0,   0
Files:                   80,      0,     147,    1353,29585745,   0,   0
TURNSTILE:              136,      0,     508,     292,     508,   0,   0
rl_entry:                40,      0,     148,     852,     148,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,      54,     126,  283736,   0,   0
THREAD:                1168,      0,     445,      62,    3006,   0,   0
cpuset:                  72,      0,     286,     869,     499,   0,   0
cyclic_id_cache:         64,      0,       0,       0,       0,   0,   0
audit_record:          1240,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 25727505,    1023,    1762, 5379045, 274,   0
mbuf:                   256, 25727505,       4,    3012,36465277,   5,   0
mbuf_cluster:          2048, 4019922,    2774,     710, 6762550,1664,   0
mbuf_jumbo_page:       4096, 2009961,       0,     203, 3242534,  28,   0
mbuf_jumbo_9k:         9216, 1786632,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 1339972,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
dtrace_state_cache:    4096,      0,       0,       0,       0,   0,   0
ttyinq:                 160,      0,      75,     600,     975,   0,   0
ttyoutq:                256,      0,      39,     516,     512,   0,   0
g_bio:                  248,      0,      90,     774,40777282,   0,   0
ata_request:            336,      0,       0,     242,   33238,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
nv_stack_t:           12288,      0,       2,       2,      12,   0,   0
FPU_save_area:          512,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1660,  246792,   0,   0
VNODE:                  472,      0,  373642,    7998, 1278219,   0,   0
VNODEPOLL:              112,      0,       1,     139,       1,   0,   0
BUF TRIE:               144,      0,       6,  105942,  209047,   0,   0
NAMEI:                 1024,      0,       0,     144,57831005,   0,   0
S VFS Cache:            108,      0,  367903,    6002, 3981082,   0,   0
STS VFS Cache:          148,      0,       3,     127,       3,   0,   0
L VFS Cache:            328,      0,    6045,  178551,  327629,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       4,      24,       4,   0,   0
space_seg_cache:         64,      0,  367795,   49713,14905535,   0,   0
zio_cache:              944,      0,    1952,  148884,158506640,   0,   0
zio_link_cache:          48,      0,    2621,  149933,155524857,   0,   0
sa_cache:                80,      0,  373353,    8547, 1262552,   0,   0
dnode_t:                856,      0,  887208,    1784, 1004453,   0,   0
dmu_buf_impl_t:         224,      0, 1770003,    3148, 4769896,   0,   0
arc_buf_hdr_t:          216,      0, 1388652,     876, 2016087,   0,   0
arc_buf_t:               72,      0,  917147,    2013, 3693027,   0,   0
zil_lwb_cache:          192,      0,      14,    3046,   18583,   0,   0
zfs_znode_cache:        368,      0,  373353,    8057, 1262552,   0,   0
Mountpoints:            816,      0,      30,      35,      30,   0,   0
pipe:                   744,      0,       8,     137,   27919,   0,   0
ksiginfo:               112,      0,     168,    1722, 8407744,   0,   0
itimer:                 352,      0,       1,      32,       1,   0,   0
KNOTE:                  128,      0,       2,     897,   12066,   0,   0
socket:                 680, 2063352,      56,     142,   11187,   0,   0
unpcb:                  240, 2063360,      14,     546,    6408,   0,   0
ipq:                     56, 125670,       0,       0,       0,   0,   0
udp_inpcb:              392, 2063360,      19,     301,    3993,   0,   0
udpcb:                   16, 2063471,      19,     985,    3993,   0,   0
tcp_inpcb:              392, 2063360,      22,     248,     779,   0,   0
tcpcb:                 1024, 2063352,      22,     122,     779,   0,   0
tcptw:                   72,  27775,       0,     550,      24,   0,   0
syncache:               152,  15366,       0,     312,     256,   0,   0
hostcache:              136,  15370,       5,     430,      23,   0,   0
tcpreass:                40, 251300,       0,     800,     346,   0,   0
sackhole:                32,      0,       0,    1250,    3843,   0,   0
sctp_ep:               1408, 2063352,       0,       0,       0,   0,   0
sctp_asoc:             2344,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     415,       4,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
ripcb:                  392, 2063360,       0,      30,       1,   0,   0
rtentry:                200,      0,      17,     303,      17,   0,   0
selfd:                   56,      0,     216,    1204, 2042917,   0,   0
SWAPMETA:               288, 8039850,       0,       0,       0,   0,   0
NetGraph items:          72,   4123,       0,       0,       0,   0,   0
NetGraph data items:     72,    527,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                           4          0
irq6: fdc0                            22          0
irq14: ata0                        66453        197
irq17: uhci0 ehci0               1080647       3206
irq19: uhci1 ahci0+             12144841      36038
cpu0:timer                     108831020     322940
irq256: em0                      3599771      10681
cpu4:timer                     100857418     299280
cpu1:timer                     105851766     314100
cpu2:timer                     105278924     312400
cpu3:timer                     105435022     312863
cpu5:timer                     100589435     298484
cpu6:timer                     100423790     297993
cpu7:timer                     100426639     298001
Total                          844585752    2506189

------------------------------------------------------------------------
pstat -T

147/2063348 files
0M/0M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
             ada0             ada1             ada2             cpu
   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  17.63 6757 116.37  16.29 6378 101.44  16.51 6688 107.83   2 89  1  0  8

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME 
m        65536      5432001 --rw------- pgsql    pgsql    pgsql    pgsql               5     41279488         1215         1215 13:15:35 16:52:18 13:15:35

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME 
s        65536      5432001 --rw------- pgsql    pgsql    pgsql    pgsql              17 13:15:35 13:15:35
s        65537      5432002 --rw------- pgsql    pgsql    pgsql    pgsql              17 13:15:35 13:15:35
s        65538      5432003 --rw------- pgsql    pgsql    pgsql    pgsql              17 13:15:35 13:15:35
s        65539      5432004 --rw------- pgsql    pgsql    pgsql    pgsql              17 13:15:35 13:15:35
s        65540      5432005 --rw------- pgsql    pgsql    pgsql    pgsql              17 13:15:35 13:15:35
s        65541      5432006 --rw------- pgsql    pgsql    pgsql    pgsql              17 13:15:35 13:15:35
s        65542      5432007 --rw------- pgsql    pgsql    pgsql    pgsql              17  0:20:07 13:15:35


------------------------------------------------------------------------
ipcs -T

msginfo:
 	msgmax:        16384	(max characters in a message)
 	msgmni:           40	(# of message queues)
 	msgmnb:         2048	(max characters in a message queue)
 	msgtql:           40	(max # of messages in system)
 	msgssz:            8	(size of a message segment)
 	msgseg:         2048	(# of message segments in system)

shminfo:
 	shmmax:    536870912	(max shared memory segment size)
 	shmmin:            1	(min shared memory segment size)
 	shmmni:          192	(max number of shared memory identifiers)
 	shmseg:          128	(max shared memory segments per process)
 	shmall:       131072	(max amount of shared memory in pages)

seminfo:
 	semmni:           50	(# of semaphore identifiers)
 	semmns:          340	(# of semaphores in system)
 	semmnu:          150	(# of undo structures in system)
 	semmsl:          340	(max # of semaphores per id)
 	semopm:          100	(max # of operations per semop call)
 	semume:           50	(max # of undo entries per process)
 	semusz:          632	(size in bytes of undo structure)
 	semvmx:        32767	(semaphore maximum value)
 	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         2         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         1         1         0         0
Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0         4
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
         0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
         0         0         0         0         0         0         0         0

Server Info:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         0         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         0         0         0         0
Server Ret-Failed
                 0
Server Faults
             0
Server Cache Stats:
    Inprog      Idem  Non-idem    Misses
         0         0         0         0
Server Write Gathering:
  WriteOps  WriteRPC   Opsaved
         0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
 	4234330 packets sent
 		2839952 data packets (25524979416 bytes)
 		209941 data packets (441552508 bytes) retransmitted
 		60 data packets unnecessarily retransmitted
 		0 resends initiated by MTU discovery
 		1157083 ack-only packets (11495 delayed)
 		0 URG only packets
 		0 window probe packets
 		26787 window update packets
 		567 control packets
 	5767647 packets received
 		3382438 acks (for 25528786994 bytes)
 		14082 duplicate acks
 		0 acks for unsent data
 		2473972 packets (13686909478 bytes) received in-sequence
 		2342 completely duplicate packets (328888 bytes)
 		0 old duplicate packets
 		0 packets with some dup. data (0 bytes duped)
 		346 out-of-order packets (450068 bytes)
 		0 packets (0 bytes) of data after window
 		0 window probes
 		26083 window update packets
 		6 packets received after close
 		0 discarded for bad checksums
 		0 discarded for bad header offset fields
 		0 discarded because packet too short
 		0 discarded due to memory problems
 	258 connection requests
 	249 connection accepts
 	0 bad connection attempts
 	0 listen queue overflows
 	0 ignored RSTs in the windows
 	504 connections established (including accepts)
 	761 connections closed (including 203 drops)
 		464 connections updated cached RTT on close
 		486 connections updated cached RTT variance on close
 		185 connections updated cached ssthresh on close
 	3 embryonic connections dropped
 	3377046 segments updated rtt (of 2245502 attempts)
 	3658 retransmit timeouts
 		8 connections dropped by rexmit timeout
 	0 persist timeouts
 		0 connections dropped by persist timeout
 	0 Connections (fin_wait_2) dropped because of timeout
 	94 keepalive timeouts
 		85 keepalive probes sent
 		9 connections dropped by keepalive
 	2780158 correct ACK header predictions
 	2335136 correct data packet header predictions
 	256 syncache entries added
 		6 retransmitted
 		0 dupsyn
 		0 dropped
 		249 completed
 		0 bucket overflow
 		0 cache overflow
 		5 reset
 		1 stale
 		0 aborted
 		0 badack
 		1 unreach
 		0 zone failures
 	256 cookies sent
 	0 cookies received
 	23 hostcache entries added
 		0 bucket overflow
 	3473 SACK recovery episodes
 	154091 segment rexmits in SACK recovery episodes
 	222989388 byte rexmits in SACK recovery episodes
 	172100 SACK options (SACK blocks) received
 	335 SACK options (SACK blocks) sent
 	0 SACK scoreboard overflow
 	0 packets with ECN CE bit set
 	0 packets with ECN ECT(0) bit set
 	0 packets with ECN ECT(1) bit set
 	0 successful ECN handshakes
 	0 times ECN reduced the congestion window
udp:
 	111943 datagrams received
 	0 with incomplete header
 	0 with bad data length field
 	0 with bad checksum
 	0 with no checksum
 	2 dropped due to no socket
 	83434 broadcast/multicast datagrams undelivered
 	0 dropped due to full socket buffers
 	0 not for hashed pcb
 	28507 delivered
 	28512 datagrams output
 	0 times multicast source filter matched
ip:
 	5859235 total packets received
 	0 bad header checksums
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with ip length > max ip packet size
 	0 with header length < data size
 	0 with data length < header length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 packets reassembled ok
 	5855932 packets for this host
 	3303 packets for unknown/unsupported protocol
 	0 packets forwarded (0 packets fast forwarded)
 	0 packets not forwardable
 	0 packets received for unknown multicast group
 	0 redirects sent
 	4240487 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	0 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 tunneling packets that can't find gif
 	0 datagrams with bad address in header
icmp:
 	2 calls to icmp_error
 	0 errors not generated in response to an icmp message
 	Output histogram:
 		destination unreachable: 2
 	0 messages with bad code fields
 	0 messages less than the minimum length
 	0 messages with bad checksum
 	0 messages with bad length
 	0 multicast echo requests ignored
 	0 multicast timestamp requests ignored
 	Input histogram:
 		destination unreachable: 2
 	0 message responses generated
 	0 invalid return addresses
 	0 no return routes
igmp:
 	3301 messages received
 	0 messages received with too few bytes
 	0 messages received with wrong TTL
 	0 messages received with bad checksum
 	797 V1/V2 membership queries received
 	0 V3 membership queries received
 	0 membership queries received with invalid field(s)
 	797 general queries received
 	0 group queries received
 	0 group-source queries received
 	0 group-source queries dropped
 	796 membership reports received
 	0 membership reports received with invalid field(s)
 	0 membership reports received for groups to which we belong
 	0 V3 reports received without Router Alert
 	0 membership reports sent
arp:
 	17 ARP requests sent
 	1262 ARP replies sent
 	23696 ARP requests received
 	17 ARP replies received
 	23713 ARP packets received
 	2 total packets dropped due to no ARP entry
 	13 ARP entrys timed out
 	0 Duplicate IPs seen
ip6:
 	23658 total packets received
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 fragments that exceeded limit
 	0 packets reassembled ok
 	23658 packets for this host
 	0 packets forwarded
 	0 packets not forwardable
 	0 redirects sent
 	23665 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	11 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 packets that violated scope rules
 	0 multicast packets which we don't join
 	Input histogram:
 		UDP: 23658
 	Mbuf statistics:
 		13934 one mbuf
 		12451 one ext mbuf
 		0 two or more ext mbuf
 	0 packets whose headers are not contiguous
 	0 tunneling packets that can't find gif
 	0 packets discarded because of too many headers
 	3 failures of source address selection
 	source addresses on a non-outgoing I/F
 		3 addresses scope=f
 	Source addresses selection rule applied:
 		3 same address
icmp6:
 	0 calls to icmp6_error
 	0 errors not generated in response to an icmp6 message
 	0 errors not generated because of rate limitation
 	Output histogram:
 		neighbor solicitation: 1
 	0 messages with bad code fields
 	0 messages < minimum length
 	0 bad checksums
 	0 messages with bad length
 	Histogram of error messages to be generated:
 		0 no route
 		0 administratively prohibited
 		0 beyond scope
 		0 address unreachable
 		0 port unreachable
 		0 packet too big
 		0 time exceed transit
 		0 time exceed reassembly
 		0 erroneous header field
 		0 unrecognized next header
 		0 unrecognized option
 		0 redirect
 		0 unknown
 	0 message responses generated
 	0 messages with too many ND options
 	0 messages with bad ND options
 	0 bad neighbor solicitation messages
 	0 bad neighbor advertisement messages
 	0 bad router solicitation messages
 	0 bad router advertisement messages
 	0 bad redirect messages
 	0 path MTU changes
rip6:
 	0 messages received
 	0 checksum calculations on inbound
 	0 messages with bad checksum
 	0 messages dropped due to no socket
 	0 multicast messages dropped due to no socket
 	0 messages dropped due to full socket buffers
 	0 delivered
 	0 datagrams output

------------------------------------------------------------------------
netstat -m

1027/4774/5801 mbufs in use (current/cache/total)
1012/2472/3484/4019922 mbuf clusters in use (current/cache/total/max)
1023/1762 mbuf+clusters out of packet secondary zone in use (current/cache)
0/203/203/2009961 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/1786632 9k jumbo clusters in use (current/cache/total/max)
0/0/0/1339972 16k jumbo clusters in use (current/cache/total/max)
2280K/6949K/9230K bytes allocated to network (current/cache/total)
5/1664/274 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
28/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
em0    1500 <Link#1>      00:30:48:f2:29:9c  4293739     0     0  2642846     0     0    0 
em0    1500 192.168.200.0 borg               5842774     -     - 10316939     -     -    - 
em0    1500 fe80::230:48f fe80::230:48ff:fe        0     -     -        3     -     -    - 
em1*   1500 <Link#2>      00:30:48:f2:29:9d        0     0     0        0     0     0    0 
lo0   16384 <Link#3>                         1615751     0     0  1615744     0     0    0 
lo0   16384 localhost     ::1                  23658     -     -    23658     -     -    - 
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 
lo0   16384 your-net      localhost              200     -     -  1592089     -     -    -

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.200.11     UGS         0   581717    em0
127.0.0.1          link#3             UH          0      200    lo0
192.168.200.0/24   link#1             U           0  2066670    em0
192.168.200.4      link#1             UHS         0  1591900    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em0/64                     link#1                        U           em0
fe80::230:48ff:fef2:299c%em0      link#1                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff01::%em0/32                     fe80::230:48ff:fef2:299c%em0  U           em0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%em0/32                     fe80::230:48ff:fef2:299c%em0  U           em0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffffe00ae250800 tcp4       0      0 192.168.200.4.22   192.168.200.87.506 ESTABLISHED
fffffe00ae250c00 tcp4       0      0 192.168.200.4.2575 198.20.8.241.443   CLOSED
fffffe04a8bb4800 tcp4       0     29 192.168.200.4.1814 170.224.160.205.44 CLOSE_WAIT
fffffe04a8bb4c00 tcp4       0     29 192.168.200.4.4998 170.224.160.205.44 CLOSE_WAIT
fffffe00ae22c800 tcp4       0      0 192.168.200.4.1616 198.20.8.241.443   CLOSED
fffffe00ae252c00 tcp4       0      0 192.168.200.4.5884 198.20.8.241.443   CLOSED
fffffe00ae263400 tcp4       0     29 192.168.200.4.5008 198.20.8.246.443   CLOSE_WAIT
fffffe033262f400 tcp4       0      0 192.168.200.4.664  192.168.200.23.204 CLOSE_WAIT
fffffe00ae2d8c00 tcp4       0      0 192.168.200.4.5766 198.20.8.246.80    CLOSED
fffffe0332660800 tcp4       0      0 127.0.0.1.587      *.*                LISTEN
fffffe0332660c00 tcp4       0      0 127.0.0.1.25       *.*                LISTEN
fffffe0332661000 tcp4       0      0 192.168.200.4.587  *.*                LISTEN
fffffe0332661400 tcp4       0      0 192.168.200.4.25   *.*                LISTEN
fffffe001ed60800 tcp4       0      0 127.0.0.1.5432     *.*                LISTEN
fffffe001ed60c00 tcp6       0      0 ::1.5432           *.*                LISTEN
fffffe00ae265400 tcp4       0      0 *.3551             *.*                LISTEN
fffffe033200b000 tcp6       0      0 *.2049             *.*                LISTEN
fffffe033200b400 tcp4       0      0 *.2049             *.*                LISTEN
fffffe00ae22b800 tcp4       0      0 *.804              *.*                LISTEN
fffffe00ae22bc00 tcp6       0      0 *.804              *.*                LISTEN
fffffe033200b800 tcp4       0      0 *.111              *.*                LISTEN
fffffe033200bc00 tcp6       0      0 *.111              *.*                LISTEN
fffffe001ed31000 udp6       0      0 ::1.55170          ::1.55170 
fffffe00ae0de620 udp4       0      0 127.0.0.1.123      *.* 
fffffe00ae0de7a8 udp6       0      0 fe80:3::1.123      *.* 
fffffe00ae0de930 udp6       0      0 ::1.123            *.* 
fffffe00ae0deab8 udp6       0      0 fe80:1::230:48ff.1 *.* 
fffffe00ae0dec40 udp4       0      0 192.168.200.4.123  *.* 
fffffe00ae0dedc8 udp6       0      0 *.123              *.* 
fffffe00ae0df000 udp4       0      0 *.123              *.* 
fffffe001ed31188 udp6       0      0 *.2049             *.* 
fffffe001ed31310 udp4       0      0 *.2049             *.* 
fffffe001ecfb7a8 udp4       0      0 *.804              *.* 
fffffe001ecfb930 udp6       0      0 *.804              *.* 
fffffe00ae0df188 udp6       0      0 *.*                *.* 
fffffe001ed31498 udp4       0      0 *.645              *.* 
fffffe001ed31620 udp4       0      0 *.111              *.* 
fffffe001ed317a8 udp6       0      0 *.957              *.* 
fffffe001ed31930 udp6       0      0 *.111              *.* 
fffffe00ae067310 udp4       0      0 *.514              *.* 
fffffe00ae067498 udp6       0      0 *.514              *.* 
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffffe04a8b084b0 stream      0      0        0 fffffe04a8b085a0        0        0
fffffe04a8b085a0 stream      0      0        0 fffffe04a8b084b0        0        0
fffffe00ae04ad20 stream      0      0 fffffe0332277b10        0        0        0 /tmp/.s.PGSQL.5432
fffffe00ae04b780 stream      0      0 fffffe0315f06000        0        0        0 /var/run/rpcbind.sock
fffffe00ae04b870 stream      0      0 fffffe001ec93938        0        0        0 /var/run/devd.pipe
fffffe00ae0aac30 dgram       0      0        0        0        0        0
fffffe00ae0abb40 dgram       0      0        0 fffffe00ae0705a0        0 fffffe03321714b0
fffffe03321714b0 dgram       0      0        0 fffffe00ae0705a0        0        0
fffffe0332088960 dgram       0      0        0 fffffe00ae0704b0        0 fffffe00ae056a50
fffffe00ae056a50 dgram       0      0        0 fffffe00ae0704b0        0 fffffe00ae0f9000
fffffe00ae0f9000 dgram       0      0        0 fffffe00ae0704b0        0 fffffe00ae056b40
fffffe00ae056b40 dgram       0      0        0 fffffe00ae0704b0        0        0
fffffe00ae0704b0 dgram       0      0 fffffe001ebb0b10        0 fffffe0332088960        0 /var/run/logpriv
fffffe00ae0705a0 dgram       0      0 fffffe001e9e7938        0 fffffe00ae0abb40        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address 
tcp4  0/0/20         localhost.submission 
tcp4  0/0/20         localhost.smtp 
tcp4  0/0/20         borg.submission 
tcp4  0/0/20         borg.smtp 
tcp4  0/0/128        localhost.postgresql 
tcp6  0/0/128        localhost.postgresql 
tcp4  0/0/5          *.3551 
tcp6  0/0/5          *.nfsd 
tcp4  0/0/5          *.nfsd 
tcp4  0/0/128        *.804 
tcp6  0/0/128        *.804 
tcp4  0/0/128        *.sunrpc 
tcp6  0/0/128        *.sunrpc 
unix  0/0/128        /tmp/.s.PGSQL.5432
unix  0/0/128        /var/run/rpcbind.sock
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read file 13 at 0xfffffffffffffff
fstat: can't read file 14 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read file 13 at 0xfffffffffffffff
fstat: can't read file 14 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read file 13 at 0xfffffffffffffff
fstat: can't read file 14 at 0x78
fstat: can't read file 16 at 0xffff
fstat: can't read file 19 at 0xfffffffffffffff
fstat: can't read file 20 at 0x78
fstat: can't read file 22 at 0xffff
fstat: can't read file 23 at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read file 13 at 0xfffffffffffffff
fstat: can't read file 14 at 0x78
fstat: can't read file 16 at 0xffff
fstat: can't read file 19 at 0xfffffffffffffff
fstat: can't read file 20 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     pwait      84044 root -         -       error    -
root     pwait      84044   wd -         -       error    -
root     pwait      84044 text -         -       error    -
root     pwait      84044 ctty /dev         13 crw-------  console rw
root     pwait      84044    0 /dev         13 crw-------  console rw
root     sh         84039 root -         -       error    -
root     sh         84039   wd -         -       error    -
root     sh         84039 text -         -       error    -
root     sh         84039 ctty /dev         13 crw-------  console rw
root     sh         84039    0 /dev         13 crw-------  console rw
root     sh         84039    6 /dev         13 crw-------  console rw
root     sleep      83945 root -         -       error    -
root     sleep      83945   wd -         -       error    -
root     sleep      83945 text -         -       error    -
root     sleep      83945 ctty /dev         13 crw-------  console rw
root     sleep      83945    0 /dev         30 crw-rw-rw-    null  r
root     sh         83943 root -         -       error    -
root     sh         83943   wd -         -       error    -
root     sh         83943 text -         -       error    -
root     sh         83943 ctty /dev         13 crw-------  console rw
root     sh         83943    0 /dev         30 crw-rw-rw-    null  r
root     sh         83940 root -         -       error    -
root     sh         83940   wd -         -       error    -
root     sh         83940 text -         -       error    -
root     sh         83940 ctty /dev         13 crw-------  console rw
root     sh         83940    0 /dev         13 crw-------  console rw
root     sh         83940    6 /dev         13 crw-------  console rw
root     sh         53796 root -         -       error    -
root     sh         53796   wd -         -       error    -
root     sh         53796 text -         -       error    -
root     sh         53796 ctty /dev        169 crw--w----   pts/0 rw
root     sh         53796    0 /dev        169 crw--w----   pts/0 rw
root     sh         53796    6 /dev        169 crw--w----   pts/0 rw
root     sudo       53795 root -         -       error    -
root     sudo       53795   wd -         -       error    -
root     sudo       53795 text -         -       error    -
root     sudo       53795 ctty /dev        169 crw--w----   pts/0 rw
root     sudo       53795    0 /dev        169 crw--w----   pts/0 rw
ler      sh         53792 root -         -       error    -
ler      sh         53792   wd -         -       error    -
ler      sh         53792 text -         -       error    -
ler      sh         53792 ctty /dev        169 crw--w----   pts/0 rw
ler      sh         53792    0 /dev        169 crw--w----   pts/0 rw
ler      sh         53792    6 /dev        169 crw--w----   pts/0 rw
ler      sshd       53791 root -         -       error    -
ler      sshd       53791   wd -         -       error    -
ler      sshd       53791 text -         -       error    -
ler      sshd       53791    0 /dev         30 crw-rw-rw-    null rw
ler      sshd       53791    6 /dev         30 crw-rw-rw-    null rw
root     sshd       53788 root -         -       error    -
root     sshd       53788   wd -         -       error    -
root     sshd       53788 text -         -       error    -
root     sshd       53788    0 /dev         30 crw-rw-rw-    null  r
root     sshd       53788    6 /dev         30 crw-rw-rw-    null rw
boinc    wcgrid_fahv_vina_p 53787 root -         -       error    -
boinc    wcgrid_fahv_vina_p 53787   wd -         -       error    -
boinc    wcgrid_fahv_vina_p 53787 text -         -       error    -
boinc    wcgrid_fahv_vina_p 53787    0 /dev         30 crw-rw-rw-    null rw
boinc    wcgrid_fahv_vina_p 53787    6 -         -       error    -
boinc    wcgrid_fahv_vina_p 53787   12 -         -       error    -
boinc    wcgrid_fahv_vina_p 53784 root -         -       error    -
boinc    wcgrid_fahv_vina_p 53784   wd -         -       error    -
boinc    wcgrid_fahv_vina_p 53784 text -         -       error    -
boinc    wcgrid_fahv_vina_p 53784    0 /dev         30 crw-rw-rw-    null rw
boinc    wcgrid_fahv_vina_p 53784    6 -         -       error    -
boinc    wcgrid_fahv_vina_p 53784   12 -         -       error    -
mailnull exim-4.80.1-2  1322 root -         -       error    -
mailnull exim-4.80.1-2  1322   wd -         -       error    -
mailnull exim-4.80.1-2  1322 text -         -       error    -
mailnull exim-4.80.1-2  1322    0 /dev         30 crw-rw-rw-    null rw
mailnull exim-4.80.1-2  1322    6 /dev         30 crw-rw-rw-    null rw
pgsql    postgres    1221 root -         -       error    -
pgsql    postgres    1221   wd -         -       error    -
pgsql    postgres    1221 text -         -       error    -
pgsql    postgres    1221    0 /dev         30 crw-rw-rw-    null  r
pgsql    postgres    1221    6 -         -         bad    -
pgsql    postgres    1220 root -         -       error    -
pgsql    postgres    1220   wd -         -       error    -
pgsql    postgres    1220 text -         -       error    -
pgsql    postgres    1220    0 /dev         30 crw-rw-rw-    null  r
pgsql    postgres    1220    6 -         -         bad    -
pgsql    postgres    1219 root -         -       error    -
pgsql    postgres    1219   wd -         -       error    -
pgsql    postgres    1219 text -         -       error    -
pgsql    postgres    1219    0 /dev         30 crw-rw-rw-    null  r
pgsql    postgres    1219    6 -         -         bad    -
pgsql    postgres    1218 root -         -       error    -
pgsql    postgres    1218   wd -         -       error    -
pgsql    postgres    1218 text -         -       error    -
pgsql    postgres    1218    0 /dev         30 crw-rw-rw-    null  r
pgsql    postgres    1218    6 -         -         bad    -
pgsql    postgres    1217 root -         -       error    -
pgsql    postgres    1217   wd -         -       error    -
pgsql    postgres    1217 text -         -       error    -
pgsql    postgres    1217    0 /dev         30 crw-rw-rw-    null  r
pgsql    postgres    1217    6 -         -         bad    -
pgsql    postgres    1215 root -         -       error    -
pgsql    postgres    1215   wd -         -       error    -
pgsql    postgres    1215 text -         -       error    -
pgsql    postgres    1215    0 /dev         30 crw-rw-rw-    null  r
pgsql    postgres    1215    6 -         -         bad    -
root     smartd      1207 root -         -       error    -
root     smartd      1207   wd -         -       error    -
root     smartd      1207 text -         -       error    -
root     smartd      1207    0 /dev         30 crw-rw-rw-    null rw
root     perl        1203 root -         -       error    -
root     perl        1203   wd -         -       error    -
root     perl        1203 text -         -       error    -
root     perl        1203    0 -         -       error    -
root     ng_queue    1191 root -         -       error    -
root     ng_queue    1191   wd -         -       error    -
root     ntpd        1170 root -         -       error    -
root     ntpd        1170   wd -         -       error    -
root     ntpd        1170 text -         -       error    -
root     ntpd        1170    0 /dev         30 crw-rw-rw-    null rw
root     ntpd        1170    6 /dev         30 crw-rw-rw-    null rw
root     ntpd        1170   12 /dev         30 crw-rw-rw-    null rw
root     ntpd        1170   18* local dgram fffffe00ae0f9000 <-> fffffe00ae0704b0
root     apcupsd     1141 root -         -       error    -
root     apcupsd     1141   wd -         -       error    -
root     apcupsd     1141 text -         -       error    -
root     apcupsd     1141    0 /dev         30 crw-rw-rw-    null  r
root     apcupsd     1141    6 /dev         30 crw-rw-rw-    null  r
root     nfsd        1137 root -         -       error    -
root     nfsd        1137   wd -         -       error    -
root     nfsd        1137 text -         -       error    -
root     nfsd        1137    0 /dev         30 crw-rw-rw-    null rw
root     nfsd        1131 root -         -       error    -
root     nfsd        1131   wd -         -       error    -
root     nfsd        1131 text -         -       error    -
root     nfsd        1131    0 /dev         30 crw-rw-rw-    null rw
root     nfsd        1131    6 /dev         30 crw-rw-rw-    null rw
root     mountd      1125 root -         -       error    -
root     mountd      1125   wd -         -       error    -
root     mountd      1125 text -         -       error    -
root     mountd      1125    0 /dev         30 crw-rw-rw-    null rw
root     mountd      1125    6 /dev         30 crw-rw-rw-    null rw
root     rpcbind     1090 root -         -       error    -
root     rpcbind     1090   wd -         -       error    -
root     rpcbind     1090 text -         -       error    -
root     rpcbind     1090    0 /dev         30 crw-rw-rw-    null rw
root     rpcbind     1090    6 /dev         30 crw-rw-rw-    null rw
root     watchdogd   1021 root -         -       error    -
root     watchdogd   1021   wd -         -       error    -
root     watchdogd   1021 text -         -       error    -
root     watchdogd   1021    0 /dev         30 crw-rw-rw-    null rw
root     syslogd     1018 root -         -       error    -
root     syslogd     1018   wd -         -       error    -
root     syslogd     1018 text -         -       error    -
root     syslogd     1018    0 /dev         30 crw-rw-rw-    null rw
root     syslogd     1018    6 /dev         30 crw-rw-rw-    null rw
root     syslogd     1018   12 /dev         30 crw-rw-rw-    null rw
root     syslogd     1018   18 -         -       error    -
root     moused       871 root -         -       error    -
root     moused       871   wd -         -       error    -
root     moused       871 text -         -       error    -
root     moused       871    0 /dev         30 crw-rw-rw-    null rw
root     devd         858 root -         -       error    -
root     devd         858   wd -         -       error    -
root     devd         858 text -         -       error    -
root     devd         858    0 /dev         30 crw-rw-rw-    null rw
root     devd         858    6 /dev         30 crw-rw-rw-    null rw
root     init           1 root -         -       error    -
root     init           1   wd -         -       error    -
root     init           1 text -         -       error    -
root     kernel         0 root -         -       error    -
root     kernel         0   wd -         -       error    -

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #20 r254262: Mon Aug 12 21:45:40 CDT 2013
     root at borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz (2327.54-MHz K8-class CPU)
   Origin = "GenuineIntel"  Id = 0x10676  Family = 0x6  Model = 0x17  Stepping = 6
   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 65660014592 (62618 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
  cpu4 (AP): APIC ID:  4
  cpu5 (AP): APIC ID:  5
  cpu6 (AP): APIC ID:  6
  cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
netmap: loaded module
random: <Software, Yarrow> initialized
cryptosoft0: <software crypto> on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2000-0x201f mem 0xd9220000-0xd923ffff,0xd9200000-0xd921ffff irq 18 at device 0.0 on pci6
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:48:f2:29:9c
001.000009 netmap_attach [2244] success for em0
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2020-0x203f mem 0xd9260000-0xd927ffff,0xd9240000-0xd925ffff irq 19 at device 0.1 on pci6
em1: Using an MSI interrupt
em1: Ethernet address: 00:30:48:f2:29:9d
001.000010 netmap_attach [2244] success for em1
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> port 0x3000-0x307f mem 0xd8000000-0xd8ffffff,0xc0000000-0xc7ffffff,0xc8000000-0xc9ffffff irq 16 at device 0.0 on pci8
nvidia0: <GeForce GT 610> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
pci8: <multimedia, HDA> at device 0.1 (no driver attached)
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> irq 16 at device 0.0 on pci10
pci11: <PCI bus> on pcib11
pcm0: <Envy24HT audio (Generic)> port 0x4080-0x409f,0x4000-0x407f irq 16 at device 0.0 on pci11
pcm0: system configuration
   SubVendorID: 0x1412, SubDeviceID: 0x2403
   XIN2 Clock Source: 24.576MHz(96kHz*256)
   MPU-401 UART(s) #: not implemented
   ADC #: 1 and SPDIF receiver connected
   DAC #: 4
   Multi-track converter type: AC'97(SDATA_OUT:packed)
   S/PDIF(IN/OUT): 1/1 ID# 0x00
   GPIO(mask/dir/state): 0xff/0xff/0xff
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
usbus0 on uhci0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
usbus1 on uhci1
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
usbus2 on uhci2
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd9600400-0xd96007ff irq 17 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci12: <ACPI PCI bus> on pcib12
vgapci1: <VGA-compatible display> port 0x5000-0x50ff mem 0xd0000000-0xd7ffffff,0xd9300000-0xd930ffff irq 18 at device 1.0 on pci12
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ahci0: <Intel ESB2 AHCI SATA controller> port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xd9600800-0xd9600bff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x1100-0x111f irq 19 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ichwd0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x38e offMax=0x4b4
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ata0: DMA limited to UDMA33, controller found non-ATA66 cable
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
ada5: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad14
cd0 at ata0 bus 0 scbus0 target 0 lun 0
cd0: <Memorex DVD+-RAM 510L v1 MWS7> Removable CD-ROM SCSI-0 device 
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #4 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3
uhub3: 6 ports with 6 removable, self powered
Root mount waiting for: usbus3
Root mount waiting for: usbus3
ugen3.2: <Peppercon AG> at usbus3
ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
kbd2 at ukbd0
Trying to mount root from zfs:zroot/ROOT/default []...
ugen0.2: <Microsoft> at usbus0
ugen0.3: <American Power Conversion> at usbus0
ffclock reset: HPET (14318180 Hz), time = 1376417561.500000000
Setting hostuuid: 53d19f64-d663-a017-8922-0030488e9ff3.
Setting hostid: 0xf53a926e.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
Mounting local file systems:.
Writing entropy file:.
Setting hostname: borg.lerctr.org.
Starting Network: lo0 em0 em1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 	inet6 ::1 prefixlen 128
 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
 	inet 127.0.0.1 netmask 0xff000000
 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9c
 	inet 192.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255
 	inet6 fe80::230:48ff:fef2:299c%em0 prefixlen 64 scopeid 0x1
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9d
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
Starting devd.
Starting Network: em1.
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9d
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
uhid0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
ums0: <Microsoft Microsoft Trackball Optical, class 0/0, rev 1.10/1.21, addr 2> on usbus0
ums0: 5 buttons and [XYZ] coordinates ID=0
add net default: gateway 192.168.200.11
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Mounting NFS file systems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/dbmail /usr/local/lib/event2 /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/virtualbox
32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat
Creating and/or trimming log files.
Starting syslogd.
Starting watchdogd.
savecore: reboot after panic: page fault
Aug 13 13:12:58 borg savecore: reboot after panic: page fault
savecore: writing core to /var/crash/vmcore.9
Writing crash summary to /var/crash/core.txt.9.
Additional ABI support: linux.
Starting rpcbind.
NFS access cache time=60
Clearing /tmp (X related).
Starting mountd.
NFSv4 is disabled
Starting nfsd.
Starting apcupsd.
Updating motd:.
Mounting late file systems:.
Starting ntpd.
Starting sshblock.
Starting smartd.
pg_ctl: another server might be running; trying to start server anyway
Updating cpucodes...
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl0 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl1 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl2 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl3 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl4 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl5 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl6 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl7 from rev 0x60c to rev 0x60f... done.
Done.
Starting exim.
Starting cupsd.
Starting boinc_client.
Starting bacula_sd.
Starting bacula_fd.
Starting bacula_dir.
Performing sanity check on sshd configuration.
Starting sshd.
Configuring syscons: blanktime.
Starting cron.
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
Starting inetd.
Starting background file system checks in 60 seconds.

Tue Aug 13 13:15:43 CDT 2013
Aug 14 16:52:42 borg shutdown: reboot by ler: 
Stopping inetd.
Waiting for PIDS: 1386.
Stopping cron.
Waiting for PIDS: 1364.
Stopping sshd.
Waiting for PIDS: 1352.
Stopping bacula_dir.
Waiting for PIDS: 1347.
Stopping bacula_fd.
Waiting for PIDS: 1344.
Stopping bacula_sd.
Waiting for PIDS: 1341Aug 14 16:52:43 borg bacula-dir: Shutting down Bacula service: borg-dir ...
Aug 14 16:52:43 borg bacula-fd: Shutting down Bacula service: borg-fd ...
Aug 14 16:52:43 borg bacula-sd: Shutting down Bacula service: borg-sd ...
.
Stopping boinc_client.
Waiting for PIDS: 1337.
Stopping cupsd.
Waiting for PIDS: 1331


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address	= 0x0
fault code		= supervisor write data, page not present
instruction pointer	= 0x20:0xffffffff806fde34
stack pointer	        = 0x28:0xffffff900d2cb930
frame pointer	        = 0x28:0xffffff900d2cb950
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1331 (cupsd)
trap number		= 12
panic: page fault
cpuid = 7
Uptime: 1d3h40m18s
Dumping 7789 out of 64479 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	BORG-DTRACE
machine	amd64
cpu	HAMMER
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	FFCLOCK
options	USB_DEBUG
options	RDRAND_RNG
options	PADLOCK_RNG
options	ATH_ENABLE_11N
options	AH_AR5416_INTERRUPT_MITIGATION
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	SC_PIXEL_MODE
options	VESA
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	ATA_STATIC_ID
options	SMP
options	MALLOC_DEBUG_MAXZONES=8
options	KDB_TRACE
options	INCLUDE_CONFIG_FILE
options	DDB_CTF
options	KDTRACE_HOOKS
options	KDTRACE_FRAME
options	MAC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_RAID
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	QUOTA
options	SCTP
options	TCP_OFFLOAD
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ahci
device	ata
device	esp
device	isci
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	hptnr
device	aacraid
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	agp
device	uart
device	ppc
device	ppbus
device	lpt
device	ppi
device	em
device	miibus
device	cas
device	gem
device	hme
device	nfe
device	nge
device	loop
device	random
device	ether
device	vlan
device	tun
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	ukbd
device	umass
device	virtio
device	virtio_pci
device	vtnet
device	virtio_blk
device	virtio_scsi
device	virtio_balloon
device	netmap

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: ler at lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688


More information about the freebsd-current mailing list