bhyve hangs on going multi user

Thu Feb 7 07:24:57 UTC 2013

Hi Sevan,

At the risk of stating the obvious:

> pid 17 (stty), uid 0: exited on signal 4
> Illegal instruction
> pid 19 (ps), uid 0: exited on signal 4
> Illegal instruction

The userland processes launched by /etc/rc are getting a SIGILL but it
is not clear why.

Could you do some experiments on your end to narrow this down a bit more:

- Boot to single user and then execute a variety of commands to figure
out whether the problem happens randomly or does is follow a
particular executable (for e.g. execute "ps", "stty", "fsck", "find",
"ls" etc).

- Did this problem start happening only recently?

- If you have a different host system does it happen there too? If you
have a different virtual machine does it happen there as well?

best
Neel

On Wed, Feb 6, 2013 at 7:55 PM, Sevan / Venture37 <venture37 at gmail.com> wrote:
> Hi,
> Attempting to start a VM with the following command results in a hang when
> the system tries to go multiuser
> sudo bhyveload -m 768 -M 8192 -d ./diskdev guest0 && sudo bhyve -c 1 -a -A
> -m 768 -M 8192 -I -H -g 0 -s 0:0,hostbridge -s 1:0,virtio-blk,diskdev -S
> 31,uart,stdio gues0
>
>
> /boot/loader.conf
> autoboot_delay=5
> kernel="kernel"
>
> virtio_load="YES"
> if_vtnet_load="YES"
> virtio_pci_load="YES"
> virtio_blk_load="YES"
>
> # Avoid clock drift
> kern.timecounter.hardware="TSC"
> kern.timecounter.invariant_tsc="1"
> # Tell the guest it is virtualized
> #smbios.bios.vendor="Bochs"
> smbios.bios.vendor="BHYVE"
>
> console="userboot"
> bootverbose="1"
>
>
> /etc/fstab
> /dev/vtbd0s1a / ufs rw 1 1
>
>
>
>
> Booting...
> GDB: no debug ports present
> KDB: debugger backends: ddb
> KDB: current backend: ddb
> SMAP type=01 base=0000000000000000 len=00000000000a0000
> SMAP type=01 base=0000000000100000 len=000000002ff00000
> SMAP type=01 base=0000000100000000 len=0000000200000000
> Table 'APIC' at 0xf0500
> APIC: Found table at 0xf0500APIC: Using the MADT enumerator.
> MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
> SMP: Added CPU 0 (AP)
> Copyright (c) 1992-2013 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 10.0-CURRENT #0 r246446: Thu Feb  7 02:36:17 UTC 2013
>     you at dev.foo.local:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
> WARNING: WITNESS option enabled, expect reduced performance.
> Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff81822000.
> Hypervisor: Origin = "BHyVBHyVBHyV"
> Calibrating TSC clock ... TSC clock: 3399738220 Hz
> CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (3399.74-MHz K8-class CPU)
>   Origin = "GenuineIntel"  Id = 0x306a9  Family = 0x6  Model = 0x3a Stepping
> = 9
>
> Features=0x8fa3ab7f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,APIC,SEP,PGE,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,PBE>
>
> Features2=0xe3bae257<SSE3,PCLMULQDQ,DTES64,DS_CPL,SMX,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,F16C,RDRAND,HV>
>   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>   AMD Features2=0x1<LAHF>
>   TSC: P-state invariant
> real memory  = 12884901888 (12288 MB)
> Physical memory chunk(s):
> 0x0000000000001000 - 0x000000000009bfff, 634880 bytes (155 pages)
> 0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages)
> 0x000000000184f000 - 0x000000002fffffff, 779816960 bytes (190385 pages)
> 0x0000000100000000 - 0x00000002e96e7fff, 8211300352 bytes (2004712 pages)
> avail memory = 8932638720 (8518 MB)
> Event timer "LAPIC" quality 400
> ACPI APIC Table: <BHYVE  BVMADT  >
> x86bios:  IVT 0x000000-0x0004ff at 0xfffffe0000000000
> x86bios: SSEG 0x001000-0x001fff at 0xffffff8000223000
> x86bios:  ROM 0x0a0000-0x0fefff at 0xfffffe00000a0000
> APIC: CPU 0 has ACPI ID 0
> random device not loaded; using insecure entropy
> ULE: setup cpu 0
> ACPI: RSDP 0xf0400 00024 (v02 BHYVE )
> ACPI: XSDT 0xf0480 00034 (v01 BHYVE  BVXSDT   00000001 INTL 20130117)
> ACPI: APIC 0xf0500 0004A (v01 BHYVE  BVMADT   00000001 INTL 20130117)
> ACPI: FACP 0xf0600 0010C (v05 BHYVE  BVFACP   00000001 INTL 20130117)
> ACPI: DSDT 0xf0800 000F2 (v02 BHYVE  BVDSDT   00000001 INTL 20130117)
> ACPI: FACS 0xf0780 00040
> MADT: Found IO APIC ID 1, Interrupt 0 at 0xfec00000
> ioapic0: Changing APIC ID to 1
> ioapic0: Routing external 8259A's -> intpin 0
> MADT: Interrupt override: source 9, irq 9
> ioapic0: intpin 9 trigger: level
> ioapic0: intpin 9 polarity: low
> ioapic0 <Version 1.1> irqs 0-16 on motherboard
> cpu0 BSP:
>      ID: 0x00000000   VER: 0x00000011 LDR: 0x00000000 DFR: 0xffffffff
>   lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
>   timer: 0x000100ef therm: 0x00010000 err: 0x000000f0
> snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
> feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=5
> feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
> wlan: <802.11 Link Layer>
> nfslock: pseudo-device
> null: <null device, zero device>
> random: <entropy source, Hardware, Intel IvyBridge+ RNG>
> module_register_init: MOD_LOAD (vesa, 0xffffffff80c24d10, 0) error 19
> io: <I/O>
> kbd0 at kbdmux0
> ctl: CAM Target Layer loaded
> acpi0: <BHYVE BVXSDT> on motherboard
> ACPI: All ACPI Tables successfully acquired
> ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48
> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
> atrtc0: registered as a time-of-day clock (resolution 1000000us, adjustment
> 0.500000000s)
> ioapic0: routing intpin 8 (ISA IRQ 8) to lapic 0 vector 49
> Event timer "RTC" frequency 32768 Hz quality 0
> ACPI timer: 1/5 1/4 1/5 1/4 1/3 1/4 1/5 1/4 1/5 1/5 -> 10
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pcib0: decoding 4 range 0-0xcf7
> pcib0: decoding 4 range 0xd00-0xffff
> pcib0: could not get PCI interrupt routing table for \_SB_.PCI0 -
> AE_NOT_FOUND
> pci0: <ACPI PCI bus> on pcib0
> pci0: domain=0, physical bus=0
> found-> vendor=0x1275, dev=0x1275, revid=0x00
>         domain=0, bus=0, slot=0, func=0
>         class=06-00-00, hdrtype=0x01, mfdev=0
>         cmdreg=0x0007, statreg=0x0000, cachelnsz=0 (dwords)
>         lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
> found-> vendor=0x1af4, dev=0x1001, revid=0x00
>         domain=0, bus=0, slot=1, func=0
>         class=01-00-00, hdrtype=0x00, mfdev=0
>         cmdreg=0x0007, statreg=0x0010, cachelnsz=0 (dwords)
>         lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
>         MSI-X supports 2 messages in map 0x14
>         map[10]: type I/O Port, range 32, base 0x2000, size  6, enabled
> pcib0: allocated type 4 (0x2000-0x203f) for rid 10 of pci0:0:1:0
>         map[14]: type Memory, range 32, base 0x30000000, size 13, enabled
> found-> vendor=0x131f, dev=0x2000, revid=0x00
>         domain=0, bus=0, slot=31, func=0
>         class=07-00-00, hdrtype=0x00, mfdev=0
>         cmdreg=0x0007, statreg=0x0000, cachelnsz=0 (dwords)
>         lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
>         intpin=a, irq=4
>         map[10]: type I/O Port, range 32, base 0x3f8, size  3, enabled
> pcib0: allocated type 4 (0x3f8-0x3ff) for rid 10 of pci0:0:31:0
> pcib0: no PRT entry for 0.31.INTA
> virtio_pci0: <VirtIO PCI Block adapter> port 0x2000-0x203f mem
> 0x30000000-0x30001fff at device 1.0 on pci0
> vtblk0: <VirtIO Block Adapter> on virtio_pci0
> virtio_pci0: host features: 0x10000004 <RingIndirect,MaxNumSegs>
> virtio_pci0: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
> virtio_pci0: attempting to allocate 2 MSI-X vectors (2 supported)
> msi: routing MSI-X IRQ 256 to local APIC 0 vector 50
> msi: routing MSI-X IRQ 257 to local APIC 0 vector 51
> virtio_pci0: using IRQs 256-257 for MSI-X
> virtio_pci0: using per VQ MSIX interrupts
> vtblk0: 20480MB (41943040 512 byte sectors)
> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0
> ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 52
> uart2: fast interrupt
> uart2: console (9600,n,8,1)
> acpi0: wakeup code va 0xffffff8000299000 pa 0x4000
> Device configuration finished.
> procfs registered
> lapic: Divisor 2, Frequency 1700037000 Hz
> Timecounters tick every 10.000 msec
> vlan: initialized, using hash tables with chaining
> tcp_init: net.inet.tcp.tcbhashsize auto tuned to 131072
> lo0: bpf attached
> hptrr: no controller detected.
> hpt27xx: no controller detected.
> GEOM: new disk vtbd0
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> TSC timecounter discards lower 1 bit(s)
> Timecounter "TSC-low" frequency 1699869110 Hz quality 1000
> WARNING: WITNESS option enabled, expect reduced performance.
> Trying to mount root from ufs:/dev/vtbd0s1a [rw]...
> start_init: trying /sbin/init
> pid 17 (stty), uid 0: exited on signal 4
> Illegal instruction
> pid 19 (ps), uid 0: exited on signal 4
> Illegal instruction
> Setting hostuuid: 524a76a9-70d9-11e2-9241-9f81272ed682.
> Setting hostid: 0x7c395229.
> No suitable dump device was found.
> Entropy harvesting:.
> Starting file system checks:
> /dev/vtbd0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
> pid 50 (fsck_ufs), uid 0: exited on signal 4
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
> fsck: /dev/vtbd0s1a: Illegal instruction
> Unknown error; help!
> ERROR: ABORTING BOOT (sending SIGTERM to parent)!
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to
> "freebsd-virtualization-unsubscribe at freebsd.org"