bhyve tty / login problems / panic
Dan Mack
mack at macktronics.com
Sat Mar 30 19:17:45 UTC 2013
Here you go:
- uni-processor output:
root at olive:~ # bhyvectl --vm=coco --get-stats --cpu=0
vcpu0
vm exits due to external interrupt 1330
number of times hlt was intercepted 676
number of times %cr access was intercepted 1
number of times rdmsr was intercepted 0
number of times wrmsr was intercepted 4
number of monitor trap exits 0
number of times pause was intercepted 428499
vm exits due to interrupt window opening 1231
vm exits due to nmi window opening 0
number of times in/out was intercepted 423985
number of times cpuid was intercepted 22
vm exits due to nested page fault 7523
number of vm exits for unknown reason 0
number of times astpending at exit 38
number of vm exits handled in userspace 853290
number of NMIs delivered to vcpu 0
vcpu total runtime 6954314530
number of ticks vcpu was idle 59572
vcpu migration across host cpus 29
number of times hlt was ignored 0
total number of vm exits 863271
And here is when running with vcpu==2:
root at olive:~ # bhyvectl --vm=coco --get-stats --cpu=0
vcpu0
vm exits due to external interrupt 19234
number of times hlt was intercepted 1965
number of times %cr access was intercepted 1
number of times rdmsr was intercepted 0
number of times wrmsr was intercepted 4
number of monitor trap exits 0
number of times pause was intercepted 3907709
vm exits due to interrupt window opening 8105
vm exits due to nmi window opening 0
number of times in/out was intercepted 457998
number of times cpuid was intercepted 50
vm exits due to nested page fault 940909
number of vm exits for unknown reason 0
number of times astpending at exit 102
number of vm exits handled in userspace 4367825
number of NMIs delivered to vcpu 0
vcpu total runtime 51834598990
number of ticks vcpu was idle 887
vcpu migration across host cpus 25
number of times hlt was ignored 0
total number of vm exits 5335975
root at olive:~ # bhyvectl --vm=coco --get-stats --cpu=1
vcpu1
vm exits due to external interrupt 51249
number of times hlt was intercepted 3676
number of times %cr access was intercepted 2
number of times rdmsr was intercepted 0
number of times wrmsr was intercepted 4
number of monitor trap exits 0
number of times pause was intercepted 4667527
vm exits due to interrupt window opening 32453
vm exits due to nmi window opening 0
number of times in/out was intercepted 57278
number of times cpuid was intercepted 63
vm exits due to nested page fault 266091
number of vm exits for unknown reason 0
number of times astpending at exit 164
number of vm exits handled in userspace 4728481
number of NMIs delivered to vcpu 0
vcpu total runtime 53833466076
number of ticks vcpu was idle 965
vcpu migration across host cpus 16
number of times hlt was ignored 0
total number of vm exits 5078343
The panic comes eventually like this:
login: panic: deadlkres: possible deadlock detected for 0xfffffe0090fb5490, blocked for 217798 ticks
cpuid = 0
Uptime: 255d6h29m56s
Automatic reboot in 15 seconds - press a key on the console to abort
Hope this helps, and FYI, here's a dmesg from the host:
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #7 r248935: Sat Mar 30 13:19:43 CDT 2013
root at olive.example.com:/usr/obj/usr/src/sys/MACKGEN amd64
FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.19-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d
Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x1fbee3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 68719476736 (65536 MB)
avail memory = 63123800064 (60199 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL DX79SI >
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
cpu8 (AP): APIC ID: 8
cpu9 (AP): APIC ID: 9
cpu10 (AP): APIC ID: 10
cpu11 (AP): APIC ID: 11
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic0 <Version 2.0> irqs 0-23 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd1 at kbdmux0
acpi0: <INTEL DX79SI> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of f0000000, 18000000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed03fff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_button0: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
xhci0: <NEC uPD720200 USB 3.0 controller> mem 0xe3400000-0xe3401fff irq 16
at device 0.0 on pci2
usbus0: waiting for BIOS to give up control
xhci0: 32 byte context size.
usbus0 on xhci0
pcib3: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci3: <ACPI PCI bus> on pcib3
vgapci0: <VGA-compatible display> port 0x3000-0x307f mem
0xe2000000-0xe2ffffff,0xd0000000-0xdfffffff,0xe0000000-0xe1ffffff irq 16
at device 0.0 on pci3
pci3: <multimedia, HDA> at device 0.1 (no driver attached)
pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci0: <base peripheral> at device 5.0 (no driver attached)
pci0: <base peripheral> at device 5.2 (no driver attached)
pcib5: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pci0: <simple comms> at device 22.0 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection 7.3.7> port 0x4040-0x405f mem
0xe3500000-0xe351ffff,0xe3521000-0xe3521fff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 4c:72:b9:d1:b2:d2
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xe3601000-0xe36013ff irq
16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib6: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci7: <ACPI PCI bus> on pcib7
pci7: <serial bus, FireWire> at device 0.0 (no driver attached)
pcib8: <ACPI PCI-PCI bridge> at device 28.3 on pci0
pci8: <ACPI PCI bus> on pcib8
xhci1: <NEC uPD720200 USB 3.0 controller> mem 0xe3200000-0xe3201fff irq 19
at device 0.0 on pci8
usbus2: waiting for BIOS to give up control
xhci1: 32 byte context size.
usbus2 on xhci1
pcib9: <ACPI PCI-PCI bridge> at device 28.5 on pci0
pci9: <ACPI PCI bus> on pcib9
em1: <Intel(R) PRO/1000 Network Connection 7.3.7> port 0x1000-0x101f mem
0xe3100000-0xe311ffff,0xe3120000-0xe3123fff irq 17 at device 0.0 on pci9
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 4c:72:b9:d1:b2:d3
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xe3602000-0xe36023ff irq
23 at device 29.0 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci10: <ACPI PCI bus> on pcib10
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Patsburg AHCI SATA controller> port
0x4068-0x406f,0x4074-0x4077,0x4060-0x4067,0x4070-0x4073,0x4020-0x403f mem
0xe3523000-0xe35237ff irq 18 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
orm0: <ISA Option ROMs> at iomem 0xd1000-0xd1fff,0xd2000-0xd2fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
est8: <Enhanced SpeedStep Frequency Control> on cpu8
p4tcc8: <CPU Frequency Thermal Control> on cpu8
est9: <Enhanced SpeedStep Frequency Control> on cpu9
p4tcc9: <CPU Frequency Thermal Control> on cpu9
est10: <Enhanced SpeedStep Frequency Control> on cpu10
p4tcc10: <CPU Frequency Thermal Control> on cpu10
est11: <Enhanced SpeedStep Frequency Control> on cpu11
p4tcc11: <CPU Frequency Thermal Control> on cpu11
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 5.0Gbps Super Speed USB v3.0
usbus3: 480Mbps High Speed USB v2.0
ugen1.1: <Intel> at usbus1
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen0.1: <0x1033> at usbus0
uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen3.1: <Intel> at usbus3
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <0x1033> at usbus2
uhub3: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <OCZ-VERTEX4 1.5> ATA-9 SATA 3.x device
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 122104MB (250069680 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <INTEL SSDSC2CW120A3 400i> ATA-9 SATA 3.x device
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
ada5: <Hitachi HUS724020ALE640 MJ6OA3B0> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: Previously was known as ad14
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
lapic4: Forcing LINT1 to edge trigger
SMP: AP CPU #4 Launched!
lapic11: Forcing LINT1 to edge trigger
SMP: AP CPU #11 Launched!
lapic7: Forcing LINT1 to edge trigger
SMP: AP CPU #7 Launched!
lapic5: Forcing LINT1 to edge trigger
SMP: AP CPU #5 Launched!
lapic9: Forcing LINT1 to edge trigger
SMP: AP CPU #9 Launched!
lapic2: Forcing LINT1 to edge trigger
SMP: AP CPU #2 Launched!
lapic3: Forcing LINT1 to edge trigger
SMP: AP CPU #3 Launched!
lapic10: Forcing LINT1 to edge trigger
SMP: AP CPU #10 Launched!
lapic6: Forcing LINT1 to edge trigger
SMP: AP CPU #6 Launched!
lapic8: Forcing LINT1 to edge trigger
SMP: AP CPU #8 Launched!
uhub3: 4 ports with 4 removable, self powered
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus3 usbus1
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3 usbus1
ugen1.2: <vendor 0x8087> at usbus1
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on
usbus1
ugen3.2: <vendor 0x8087> at usbus3
uhub5: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on
usbus3
Root mount waiting for: usbus3 usbus1
uhub4: 6 ports with 6 removable, self powered
uhub5: 8 ports with 8 removable, self powered
ugen1.3: <vendor 0x060b> at usbus1
ukbd0: <vendor 0x060b USB Keyboard, class 0/0, rev 2.00/1.10, addr 3> on
usbus1
kbd2 at ukbd0
ugen3.3: <Ralink> at usbus3
Root mount waiting for: usbus3
ugen3.4: <vendor 0x0db0> at usbus3
Trying to mount root from zfs:tron []...
ums0: <vendor 0x060b USB Keyboard, class 0/0, rev 2.00/1.10, addr 3> on
usbus1
ums0: 5 buttons and [XYZ] coordinates ID=0
run0: <1.0> on usbus3
run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address
94:db:c9:e3:6e:c8
ubt0: <vendor 0x0db0 product 0xa871, class 224/1, rev 2.00/52.76, addr 4>
on usbus3
WARNING: attempt to domain_add(bluetooth) after domainfinalize()
WARNING: attempt to domain_add(netgraph) after domainfinalize()
tap0: Ethernet address: 00:bd:8c:54:07:00
tap0: link state changed to UP
bridge0: Ethernet address: 02:4c:89:ce:33:00
tap0: promiscuous mode enabled
bridge0: link state changed to UP
em1: link state changed to DOWN
em1: promiscuous mode enabled
em1: link state changed to UP
tap0: link state changed to DOWN
tap0: link state changed to UP
tap0: link state changed to DOWN
tap0: link state changed to UP
>
> Yep, no problem. I saw the checkin and started a rebuild a few minutes ago
> :-)
>
> Thanks,
>
> Dan
>
> On Sat, 30 Mar 2013, Neel Natu wrote:
>
>> Hi Dan,
>>
>> On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack <mack at macktronics.com> wrote:
>>
>>>
>>>
>>> Here you go Neel:
>>>
>>> root at olive:~ # bhyvectl --vm=coco --get-stats --cpu=0
>>> vcpu0
>>> vm exits due to external interrupt 45051
>>> number of times hlt was intercepted 2277
>>> number of NMIs delivered to vcpu 0
>>> vcpu total runtime 105001652482
>>> number of ticks vcpu was idle 919
>>> vcpu migration across host cpus 13
>>> number of times hlt was ignored 0
>>> total number of vm exits 9240925
>>> root at olive:~ #
>>> root at olive:~ #
>>> root at olive:~ # bhyvectl --vm=coco --get-stats --cpu=1
>>> vcpu1
>>> vm exits due to external interrupt 149431
>>> number of times hlt was intercepted 4222
>>> number of NMIs delivered to vcpu 0
>>> vcpu total runtime 106876622528
>>> number of ticks vcpu was idle 869
>>> vcpu migration across host cpus 20
>>> number of times hlt was ignored 0
>>> total number of vm exits 8065909
>>>
>>>
>> That did not help a whole lot because we were not keeping track of all the
>> reasons a vcpu could exit.
>>
>> I have fixed that in r248935:
>> http://svnweb.freebsd.org/base?view=revision&revision=248935
>>
>> Do you mind updating your vmm.ko with the change and getting the stats
>> again?
>>
>> best
>> Neel
>>
>>
>>> Dan
>>>
>>>
>>>
>>> On Fri, 29 Mar 2013, Neel Natu wrote:
>>>
>>> Hi Dan,
>>>>
>>>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack <mack at macktronics.com> wrote:
>>>>
>>>>
>>>>> I ran the procsystime dtrace script on bhyve when it was chewing up all
>>>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime:
>>>>>
>>>>> root at olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT
>>>>>
>>>>> Tracing... Hit Ctrl-C to end...
>>>>> dtrace: 158536 dynamic variable drops with non-empty dirty list
>>>>> dtrace: 207447 dynamic variable drops with non-empty dirty list
>>>>> dtrace: 189205 dynamic variable drops with non-empty dirty list
>>>>> dtrace: 164341 dynamic variable drops with non-empty dirty list
>>>>> dtrace: 246307 dynamic variable drops with non-empty dirty list
>>>>> dtrace: 187640 dynamic variable drops with non-empty dirty list
>>>>> dtrace: 214771 dynamic variable drops with non-empty dirty list
>>>>> dtrace: 221265 dynamic variable drops with non-empty dirty list
>>>>> ^C
>>>>> dtrace: 243468 dynamic variable drops with non-empty dirty list
>>>>>
>>>>> Elapsed Times for processes bhyve,
>>>>>
>>>>> SYSCALL TIME (ns)
>>>>> _umtx_op 18349
>>>>> writev 135712
>>>>> preadv 16175267
>>>>> pwritev 22924378
>>>>> ioctl 4353897920
>>>>> TOTAL: 4393151626
>>>>>
>>>>> CPU Times for processes bhyve,
>>>>>
>>>>> SYSCALL TIME (ns)
>>>>> _umtx_op 8815
>>>>> writev 103145
>>>>> pwritev 10647023
>>>>> preadv 15159976
>>>>> ioctl 3943399889
>>>>> TOTAL: 3969318848
>>>>>
>>>>> Syscall Counts for processes bhyve,
>>>>>
>>>>> SYSCALL COUNT
>>>>> _umtx_op 17
>>>>> writev 43
>>>>> pwritev 639
>>>>> preadv 1183
>>>>> ioctl 652782
>>>>> TOTAL: 654664
>>>>>
>>>>> Not sure if that helps or not. Hotkernel shows the cpu in the
>>>>> kernel`acpi_cpu_c1 call most of the time, normal ?
>>>>>
>>>>> Dan
>>>>>
>>>>> root at olive:/usr/share/dtrace/****toolkit # ./hotkernel
>>>>>
>>>>> Sampling... Hit Ctrl-C to end.
>>>>> ^C
>>>>> FUNCTION COUNT PCNT
>>>>> zfs.ko`space_map_sync 1 0.0%
>>>>> kernel`syscall_thread_exit 1 0.0%
>>>>> kernel`should_yield 1 0.0%
>>>>> zfs.ko`lzjb_compress 1 0.0%
>>>>> zfs.ko`dbuf_rele 1 0.0%
>>>>> zfs.ko`dsl_pool_tempreserve_****space 1 0.0%
>>>>>
>>>>> kernel`_bus_dmamap_load_buffer 1 0.0%
>>>>> kernel`cpu_idleclock 1 0.0%
>>>>> kernel`_rw_runlock_cookie 1 0.0%
>>>>> kernel`pmap_enter 1 0.0%
>>>>> kernel`__lockmgr_args 1 0.0%
>>>>> zfs.ko`vdev_geom_io_start 1 0.0%
>>>>> kernel`ahci_ch_intr 1 0.0%
>>>>> kernel`__rw_rlock 1 0.0%
>>>>> kernel`VOP_UNLOCK_APV 1 0.0%
>>>>> zfs.ko`arc_buf_thaw 1 0.0%
>>>>> kernel`atomic_set_int 1 0.0%
>>>>> kernel`g_io_schedule_down 1 0.0%
>>>>> kernel`__mtx_lock_flags 1 0.0%
>>>>> kernel`_callout_stop_safe 1 0.0%
>>>>> kernel`sched_idletd 1 0.0%
>>>>> kernel`_sx_slock 1 0.0%
>>>>> vmm.ko`lapic_set_intr 1 0.0%
>>>>> kernel`cpusetobj_ffs 1 0.0%
>>>>> zfs.ko`dbuf_hold 1 0.0%
>>>>> vmm.ko`vlapic_intr_accepted 1 0.0%
>>>>> zfs.ko`sa_attr_op 1 0.0%
>>>>> zfs.ko`dmu_buf_hold_array_by_****dnode 1 0.0%
>>>>>
>>>>> zfs.ko`taskq_run_safe 1 0.0%
>>>>> kernel`vm_reserv_free_page 1 0.0%
>>>>> kernel`_sx_sunlock 1 0.0%
>>>>> kernel`vn_rlimit_fsize 1 0.0%
>>>>> zfs.ko`vdev_queue_io_to_issue 1 0.0%
>>>>> kernel`vn_write 1 0.0%
>>>>> kernel`lock_init 1 0.0%
>>>>> vmm.ko`vmx_resume 1 0.0%
>>>>> zfs.ko`zio_gang_tree_free 1 0.0%
>>>>> zfs.ko`dbuf_rele_and_unlock 1 0.0%
>>>>> kernel`mnt_vnode_next_active 1 0.0%
>>>>> kernel`vm_page_remove 1 0.0%
>>>>> zfs.ko`vdev_queue_deadline_****compare 1 0.0%
>>>>>
>>>>> kernel`zone_free_item 1 0.0%
>>>>> vmm.ko`vmx_launch 1 0.0%
>>>>> kernel`m_pullup 1 0.0%
>>>>> kernel`pmap_change_wiring 1 0.0%
>>>>> kernel`sched_userret 1 0.0%
>>>>> zfs.ko`dbuf_write_done 1 0.0%
>>>>> zfs.ko`zio_execute 1 0.0%
>>>>> kernel`PHYS_TO_VM_PAGE 2 0.0%
>>>>> zfs.ko`avl_rotation 2 0.0%
>>>>> kernel`ehci_device_isoc_fs_****enter 2 0.0%
>>>>>
>>>>> vmm.ko`vm_exitinfo 2 0.0%
>>>>> zfs.ko`zfs_freebsd_write 2 0.0%
>>>>> vmm.ko`vcpu_stats 2 0.0%
>>>>> kernel`atomic_add_long 2 0.0%
>>>>> vmm.ko`0xffffffff81 2 0.0%
>>>>> kernel`vm_reserv_alloc_page 2 0.0%
>>>>> kernel`pagezero 2 0.0%
>>>>> kernel`bzero 2 0.0%
>>>>> zfs.ko`fletcher_2_native 2 0.0%
>>>>> vmm.ko`vmx_getreg 2 0.0%
>>>>> vmm.ko`vm_guest_msrs 3 0.0%
>>>>> vmm.ko`vmx_setreg 3 0.0%
>>>>> vmm.ko`vm_get_register 3 0.0%
>>>>> kernel`sx_try_xlock_ 3 0.0%
>>>>> kernel`vm_radix_lookup_le 3 0.0%
>>>>> vmm.ko`lapic_pending_intr 4 0.0%
>>>>> kernel`syscall_thread_enter 4 0.0%
>>>>> kernel`cpu_set_syscall_retval 4 0.0%
>>>>> kernel`__mnt_vnode_next_all 4 0.0%
>>>>> kernel`vfs_stdsync 4 0.0%
>>>>> vmm.ko`vmm_fetch_instruction 5 0.0%
>>>>> vmm.ko`vlapic_op_mem_read 6 0.0%
>>>>> kernel`_sx_xunlock 7 0.0%
>>>>> vmm.ko`vmm_emulate_instruction 7 0.0%
>>>>> vmm.ko`vlapic_op_mem_write 7 0.0%
>>>>> kernel`__mtx_lock_spin_flags 8 0.0%
>>>>> vmm.ko`lapic_timer_tick 8 0.0%
>>>>> zfs.ko`list_next 8 0.0%
>>>>> zfs.ko`l2arc_feed_thread 8 0.0%
>>>>> kernel`userret 9 0.0%
>>>>> vmm.ko`vm_gpa2hpa 9 0.0%
>>>>> vmm.ko`vmm_decode_instruction 11 0.0%
>>>>> vmm.ko`ept_vmmmap_get 11 0.0%
>>>>> vmm.ko`vm_lapic 11 0.0%
>>>>> kernel`spinlock_enter 14 0.0%
>>>>> kernel`__mtx_lock_sleep 15 0.0%
>>>>> vmm.ko`vcpu_set_state 16 0.0%
>>>>> vmm.ko`vm_nmi_pending 16 0.0%
>>>>> kernel`malloc_type_zone_****allocated 16 0.0%
>>>>>
>>>>> kernel`devfs_ioctl_f 16 0.0%
>>>>> kernel`pmap_kextract 17 0.0%
>>>>> kernel`sys_ioctl 18 0.0%
>>>>> kernel`malloc 21 0.0%
>>>>> kernel`cpu_fetch_syscall_args 24 0.0%
>>>>> kernel`uma_zfree_arg 28 0.0%
>>>>> vmm.ko`vlapic_pending_intr 29 0.0%
>>>>> kernel`trap 30 0.0%
>>>>> kernel`hpet_get_timecount 31 0.0%
>>>>> dtrace.ko`dtrace_trap 35 0.0%
>>>>> vmm.ko`vlapic_update_ppr 39 0.0%
>>>>> vmm.ko`vmmdev_ioctl 40 0.0%
>>>>> kernel`copyout 43 0.0%
>>>>> kernel`critical_enter 44 0.0%
>>>>> kernel`copyin 51 0.1%
>>>>> vmm.ko`vmcs_read 53 0.1%
>>>>> kernel`fpusave 61 0.1%
>>>>> kernel`uma_zalloc_arg 62 0.1%
>>>>> kernel`devvn_refthread 69 0.1%
>>>>> kernel`atomic_load_acq_int 70 0.1%
>>>>> kernel`fpurestore 70 0.1%
>>>>> vmm.ko`vlapic_timer_tick 75 0.1%
>>>>> kernel`amd64_syscall 77 0.1%
>>>>> kernel`bcopy 91 0.1%
>>>>> kernel`fpudna 103 0.1%
>>>>> kernel`free 103 0.1%
>>>>> kernel`critical_exit 105 0.1%
>>>>> kernel`dev_relthread 108 0.1%
>>>>> kernel`spinlock_exit 119 0.1%
>>>>> kernel`kern_ioctl 138 0.1%
>>>>> kernel`fpuexit 242 0.3%
>>>>> kernel`0xffffffff80 306 0.3%
>>>>> vmm.ko`restore_host_msrs 398 0.4%
>>>>> vmm.ko`vm_run 428 0.4%
>>>>> vmm.ko`restore_guest_msrs 433 0.5%
>>>>> kernel`cpu_idle 906 1.0%
>>>>> vmm.ko`vmx_run 9266 9.7%
>>>>> kernel`acpi_cpu_c1 81134 85.2%
>>>>>
>>>>> Hope this helps, let me know if you have any ideas on what to try next.
>>>>>
>>>>>
>>>>> Could you try the following command to get vcpu stats?
>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0
>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1
>>>>
>>>> best
>>>> Neel
>>>>
>>>>
>>>> Dan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 29 Mar 2013, Dan Mack wrote:
>>>>>
>>>>> On Thu, 28 Mar 2013, Peter Grehan wrote:
>>>>>
>>>>>>
>>>>>> Hi Dan,
>>>>>>
>>>>>>>
>>>>>>> Any ideas?
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6
>>>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest.
>>>>>>>
>>>>>>>
>>>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM.
>>>>>>
>>>>>> The issue you are seeing looks a bit like what happens when systems
>>>>>> are
>>>>>>
>>>>>>> heavily oversubscribed and vCPUs can't get enough run time.
>>>>>>>
>>>>>>>
>>>>>> I was running in on an idle system; only bhyve was running.
>>>>>>
>>>>>>
>>>>>> How much spare RAM is on the host when you started the vm ? Also, is
>>>>>>
>>>>>>> there much running on the host when the guest is started ?
>>>>>>>
>>>>>>>
>>>>>> 64GB
>>>>>>
>>>>>> Here's some more information (sorry for the formatting):
>>>>>>
>>>>>> Case 1: single vCPU via:
>>>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47
>>>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s
>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
>>>>>>
>>>>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB
>>>>>> free RAM, nothing else clocking CPU cycles):
>>>>>>
>>>>>> root at olive:~ # top -bH
>>>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31
>>>>>> 14:11:51
>>>>>> 24 processes: 1 running, 23 sleeping
>>>>>>
>>>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free
>>>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other
>>>>>> Swap:
>>>>>>
>>>>>>
>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>>>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00%
>>>>>> bhyve{coco vcpu 0}
>>>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd
>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd
>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh
>>>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00%
>>>>>> bhyve{coco mevent}
>>>>>>
>>>>>>
>>>>>> So, case #1 all is normal. I was able to build a kernel with very good
>>>>>> performance.
>>>>>>
>>>>>> The boot sequence looks like the following:
>>>>>>
>>>>>> root at olive:/vms/coco # ./vmrun.sh coco
>>>>>> Launching virtual machine "coco" ...
>>>>>> Consoles: userboot
>>>>>>
>>>>>> FreeBSD/amd64 User boot, Revision 1.1
>>>>>> (root at olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178
>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08]
>>>>>>
>>>>>> /
>>>>>> ______ ____ _____ _____
>>>>>> | ____| | _ \ / ____| __ \
>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | |
>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | |
>>>>>> | | | | | __/ __/| |_) |____) | |__| |
>>>>>> | | | | | | || | | |
>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ```
>>>>>> `
>>>>>> s` `.....---.......--.```
>>>>>> -/
>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:`
>>>>>> +.
>>>>>> | | yo`:. :o
>>>>>> `+-
>>>>>> | 1. Boot Multi User [Enter] | y/ -/`
>>>>>> -o/
>>>>>> | 2. Boot [S]ingle User | .-
>>>>>> ::/sy+:.
>>>>>> | 3. [Esc]ape to loader prompt | / `--
>>>>>> /
>>>>>> | 4. Reboot | `:
>>>>>> :`
>>>>>> | | `:
>>>>>> :`
>>>>>> | Options: | /
>>>>>> /
>>>>>> | 5. Configure Boot [O]ptions... | .-
>>>>>> -.
>>>>>> | | --
>>>>>> -.
>>>>>> | | `:` `:`
>>>>>> | | .-- `--.
>>>>>> | | .---.....----.
>>>>>> +-----------------------------****------------+
>>>>>>
>>>>>>
>>>>>>
>>>>>> Booting...
>>>>>> Copyright (c) 1992-2013 The FreeBSD Project.
>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
>>>>>> 1994
>>>>>> The Regents of the University of California. All rights
>>>>>> reserved.
>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
>>>>>> root at cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64
>>>>>>
>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class
>>>>>> CPU)
>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d
>>>>>> Stepping = 7
>>>>>> Features=0x8fa3ab7f<FPU,VME,****DE,PSE,TSC,MSR,PAE,CX8,APIC,**
>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE>
>>>>>> Features2=0x83bee217<SSE3,****PCLMULQDQ,DTES64,DS_CPL,SSSE3,****
>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,**
>>>>>> TSCDLT,**AESNI,HV>
>>>>>> AMD Features=0x2c100800<SYSCALL,****NX,Page1GB,RDTSCP,LM>
>>>>>>
>>>>>> AMD Features2=0x1<LAHF>
>>>>>> TSC: P-state invariant
>>>>>> real memory = 5368709120 (5120 MB)
>>>>>> avail memory = 3957534720 (3774 MB)
>>>>>> Event timer "LAPIC" quality 400
>>>>>> ACPI APIC Table: <BHYVE BVMADT >
>>>>>> random device not loaded; using insecure entropy
>>>>>> ioapic0: Changing APIC ID to 1
>>>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard
>>>>>> kbd0 at kbdmux0
>>>>>> acpi0: <BHYVE BVXSDT> on motherboard
>>>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
>>>>>> Event timer "RTC" frequency 32768 Hz quality 0
>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
>>>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>>>>>> pci0: <ACPI PCI bus> on pcib0
>>>>>> pcib0: no PRT entry for 0.31.INTA
>>>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem
>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0
>>>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0
>>>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>>> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df
>>>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem
>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0
>>>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1
>>>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>>> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>>> vtblk0: 8192MB (16777216 512 byte sectors)
>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on
>>>>>> pci0
>>>>>> uart2: console (9600,n,8,1)
>>>>>> Timecounters tick every 10.000 msec
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000
>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf.
>>>>>>
>>>>>> Setting hostid: 0x24bfd025.
>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart.
>>>>>> Starting file system checks:
>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
>>>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7%
>>>>>> fragmentation)
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> Mounting local file systems:.
>>>>>> Writing entropy file:.
>>>>>> Setting hostname: cocopuff.example.com.
>>>>>> vtnet0: link state changed to UP
>>>>>> Starting Network: lo0 vtnet0.
>>>>>> lo0: flags=8049<UP,LOOPBACK,****RUNNING,MULTICAST> metric 0 mtu 16384
>>>>>> options=600003<RXCSUM,TXCSUM,****RXCSUM_IPV6,TXCSUM_IPV6>
>>>>>>
>>>>>> inet6 ::1 prefixlen 128
>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>>>> inet 127.0.0.1 netmask 0xff000000
>>>>>> nd6 options=21<PERFORMNUD,AUTO_****LINKLOCAL>
>>>>>> vtnet0: flags=8943<UP,BROADCAST,****RUNNING,PROMISC,SIMPLEX,****
>>>>>> MULTICAST>
>>>>>> metric 0 mtu 1500
>>>>>> options=80028<VLAN_MTU,JUMBO_****MTU,LINKSTATE>
>>>>>>
>>>>>> ether 00:a0:98:ac:0d:df
>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
>>>>>> nd6 options=29<PERFORMNUD,****IFDISABLED,AUTO_LINKLOCAL>
>>>>>>
>>>>>> media: Ethernet 1000baseT <full-duplex>
>>>>>> status: active
>>>>>> Starting devd.
>>>>>> add net default: gateway 172.22.22.252
>>>>>> add net ::ffff:0.0.0.0: gateway ::1
>>>>>> add net ::0.0.0.0: gateway ::1
>>>>>> add net fe80::: gateway ::1
>>>>>> add net ff02::: gateway ::1
>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
>>>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>>>> Creating and/or trimming log files.
>>>>>> Starting syslogd.
>>>>>> Clearing /tmp (X related).
>>>>>> Updating motd:.
>>>>>> Performing sanity check on sshd configuration.
>>>>>> Starting sshd.
>>>>>> Starting cron.
>>>>>> Starting background file system checks in 60 seconds.
>>>>>>
>>>>>> Fri Mar 29 14:09:20 CDT 2013
>>>>>>
>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console)
>>>>>>
>>>>>> login:
>>>>>>
>>>>>>
>>>>>> ====== Next, boot the same image with two (2) vCPU:
>>>>>>
>>>>>> Command run by vmrun.sh:
>>>>>>
>>>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42
>>>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s
>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
>>>>>>
>>>>>> Top shows it running out of control:
>>>>>>
>>>>>> root at olive:~ # top -bH
>>>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12
>>>>>> 14:17:32
>>>>>> 25 processes: 3 running, 22 sleeping
>>>>>>
>>>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free
>>>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M
>>>>>> Other
>>>>>> Swap:
>>>>>>
>>>>>>
>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>>>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00%
>>>>>> bhyve{coco vcpu 1}
>>>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48%
>>>>>> bhyve{coco vcpu 0}
>>>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd
>>>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd
>>>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd
>>>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron
>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd
>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh
>>>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00%
>>>>>> bhyve{coco mevent}
>>>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh
>>>>>>
>>>>>> Even though, it's just sitting at the login prompt:
>>>>>>
>>>>>> root at olive:/vms/coco # ./vmrun.sh coco
>>>>>> Launching virtual machine "coco" ...
>>>>>> Consoles: userboot
>>>>>>
>>>>>> FreeBSD/amd64 User boot, Revision 1.1
>>>>>> (root at olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178
>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08]
>>>>>>
>>>>>> /
>>>>>> ______ ____ _____ _____
>>>>>> | ____| | _ \ / ____| __ \
>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | |
>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | |
>>>>>> | | | | | __/ __/| |_) |____) | |__| |
>>>>>> | | | | | | || | | |
>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ```
>>>>>> `
>>>>>> s` `.....---.......--.```
>>>>>> -/
>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:`
>>>>>> +.
>>>>>> | | yo`:. :o
>>>>>> `+-
>>>>>> | 1. Boot Multi User [Enter] | y/ -/`
>>>>>> -o/
>>>>>> | 2. Boot [S]ingle User | .-
>>>>>> ::/sy+:.
>>>>>> | 3. [Esc]ape to loader prompt | / `--
>>>>>> /
>>>>>> | 4. Reboot | `:
>>>>>> :`
>>>>>> | | `:
>>>>>> :`
>>>>>> | Options: | /
>>>>>> /
>>>>>> | 5. Configure Boot [O]ptions... | .-
>>>>>> -.
>>>>>> | | --
>>>>>> -.
>>>>>> | | `:` `:`
>>>>>> | | .-- `--.
>>>>>> | | .---.....----.
>>>>>> +-----------------------------****------------+
>>>>>>
>>>>>>
>>>>>>
>>>>>> Booting...
>>>>>> Copyright (c) 1992-2013 The FreeBSD Project.
>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
>>>>>> 1994
>>>>>> The Regents of the University of California. All rights
>>>>>> reserved.
>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
>>>>>> root at cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64
>>>>>>
>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class
>>>>>> CPU)
>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d
>>>>>> Stepping = 7
>>>>>> Features=0x8fa3ab7f<FPU,VME,****DE,PSE,TSC,MSR,PAE,CX8,APIC,**
>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE>
>>>>>> Features2=0x83bee217<SSE3,****PCLMULQDQ,DTES64,DS_CPL,SSSE3,****
>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,**
>>>>>> TSCDLT,**AESNI,HV>
>>>>>> AMD Features=0x2c100800<SYSCALL,****NX,Page1GB,RDTSCP,LM>
>>>>>>
>>>>>> AMD Features2=0x1<LAHF>
>>>>>> TSC: P-state invariant
>>>>>> real memory = 5368709120 (5120 MB)
>>>>>> avail memory = 3957534720 (3774 MB)
>>>>>> Event timer "LAPIC" quality 400
>>>>>> ACPI APIC Table: <BHYVE BVMADT >
>>>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
>>>>>> FreeBSD/SMP: 2 package(s) x 1 core(s)
>>>>>> cpu0 (BSP): APIC ID: 0
>>>>>> cpu1 (AP): APIC ID: 1
>>>>>> random device not loaded; using insecure entropy
>>>>>> ioapic0: Changing APIC ID to 2
>>>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard
>>>>>> kbd0 at kbdmux0
>>>>>> acpi0: <BHYVE BVXSDT> on motherboard
>>>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
>>>>>> Event timer "RTC" frequency 32768 Hz quality 0
>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
>>>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>>>>>> pci0: <ACPI PCI bus> on pcib0
>>>>>> pcib0: no PRT entry for 0.31.INTA
>>>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem
>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0
>>>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0
>>>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>>> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df
>>>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem
>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0
>>>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1
>>>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>>> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>>> vtblk0: 8192MB (16777216 512 byte sectors)
>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on
>>>>>> pci0
>>>>>> uart2: console (9600,n,8,1)
>>>>>> Timecounters tick every 10.000 msec
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> SMP: AP CPU #1 Launched!
>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf.
>>>>>>
>>>>>> Setting hostid: 0x24bfd025.
>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart.
>>>>>> Starting file system checks:
>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
>>>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7%
>>>>>> fragmentation)
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>>> Mounting local file systems:.
>>>>>> Writing entropy file:.
>>>>>> Setting hostname: cocopuff.example.com.
>>>>>> vtnet0: link state changed to UP
>>>>>> Starting Network: lo0 vtnet0.
>>>>>> lo0: flags=8049<UP,LOOPBACK,****RUNNING,MULTICAST> metric 0 mtu 16384
>>>>>> options=600003<RXCSUM,TXCSUM,****RXCSUM_IPV6,TXCSUM_IPV6>
>>>>>>
>>>>>> inet6 ::1 prefixlen 128
>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>>>> inet 127.0.0.1 netmask 0xff000000
>>>>>> nd6 options=21<PERFORMNUD,AUTO_****LINKLOCAL>
>>>>>> vtnet0: flags=8943<UP,BROADCAST,****RUNNING,PROMISC,SIMPLEX,****
>>>>>> MULTICAST>
>>>>>> metric 0 mtu 1500
>>>>>> options=80028<VLAN_MTU,JUMBO_****MTU,LINKSTATE>
>>>>>>
>>>>>> ether 00:a0:98:ac:0d:df
>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
>>>>>> nd6 options=29<PERFORMNUD,****IFDISABLED,AUTO_LINKLOCAL>
>>>>>>
>>>>>> media: Ethernet 1000baseT <full-duplex>
>>>>>> status: active
>>>>>> Starting devd.
>>>>>> add net default: gateway 172.22.22.252
>>>>>> add net ::ffff:0.0.0.0: gateway ::1
>>>>>> add net ::0.0.0.0: gateway ::1
>>>>>> add net fe80::: gateway ::1
>>>>>> add net ff02::: gateway ::1
>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
>>>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>>>> Creating and/or trimming log files.
>>>>>> Starting syslogd.
>>>>>> syslogd: timed out waiting for child
>>>>>> /etc/rc: WARNING: failed to start syslogd
>>>>>> Clearing /tmp (X related).
>>>>>> Updating motd:.
>>>>>> Performing sanity check on sshd configuration.
>>>>>> Starting sshd.
>>>>>> Starting cron.
>>>>>> Starting background file system checks in 60 seconds.
>>>>>>
>>>>>> Thu Apr 18 02:06:43 CDT 2013
>>>>>>
>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console)
>>>>>>
>>>>>> login:
>>>>>>
>>>>>>
>>>>>> ========== end of boot sequence
>>>>>>
>>>>>> So, something is broken with SMP on the guest, I'd be happy to help
>>>>>> just
>>>>>> let me know.
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>> ______________________________****_________________
>>>>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org> <
>>>>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization at freebsd.org>>mailing
>>>>>> list
>>>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****
>>>>>> virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>
>>>>>> <http://lists.**freebsd.org/mailman/listinfo/**freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
>>>>>>>
>>>>>> To unsubscribe, send any mail to "freebsd-virtualization-**
>>>>>> unsubscribe at freebsd.org <freebsd-virtualization-**
>>>>>> unsubscribe at freebsd.org<freebsd-virtualization-unsubscribe at freebsd.org>
>>>>>>> "
>>>>>>
>>>>>> ______________________________****_________________
>>>>>>
>>>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org> <
>>>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization at freebsd.org>>mailing
>>>>> list
>>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>
>>>>> <http://lists.**freebsd.org/mailman/listinfo/**freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
>>>>>>
>>>>> To unsubscribe, send any mail to "freebsd-virtualization-**
>>>>> unsubscribe at freebsd.org <freebsd-virtualization-**
>>>>> unsubscribe at freebsd.org <freebsd-virtualization-unsubscribe at freebsd.org>
>>>>>> "
>>>>>
>>>>>
>>>>
>>
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscribe at freebsd.org"
>
More information about the freebsd-virtualization
mailing list