bhyve tty / login problems / panic
Dan Mack
mack at macktronics.com
Sat Mar 30 18:00:28 UTC 2013
Yep, no problem. I saw the checkin and started a rebuild a few minutes
ago :-)
Thanks,
Dan
On Sat, 30 Mar 2013, Neel Natu wrote:
> Hi Dan,
>
> On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack <mack at macktronics.com> wrote:
>
>>
>>
>> Here you go Neel:
>>
>> root at olive:~ # bhyvectl --vm=coco --get-stats --cpu=0
>> vcpu0
>> vm exits due to external interrupt 45051
>> number of times hlt was intercepted 2277
>> number of NMIs delivered to vcpu 0
>> vcpu total runtime 105001652482
>> number of ticks vcpu was idle 919
>> vcpu migration across host cpus 13
>> number of times hlt was ignored 0
>> total number of vm exits 9240925
>> root at olive:~ #
>> root at olive:~ #
>> root at olive:~ # bhyvectl --vm=coco --get-stats --cpu=1
>> vcpu1
>> vm exits due to external interrupt 149431
>> number of times hlt was intercepted 4222
>> number of NMIs delivered to vcpu 0
>> vcpu total runtime 106876622528
>> number of ticks vcpu was idle 869
>> vcpu migration across host cpus 20
>> number of times hlt was ignored 0
>> total number of vm exits 8065909
>>
>>
> That did not help a whole lot because we were not keeping track of all the
> reasons a vcpu could exit.
>
> I have fixed that in r248935:
> http://svnweb.freebsd.org/base?view=revision&revision=248935
>
> Do you mind updating your vmm.ko with the change and getting the stats
> again?
>
> best
> Neel
>
>
>> Dan
>>
>>
>>
>> On Fri, 29 Mar 2013, Neel Natu wrote:
>>
>> Hi Dan,
>>>
>>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack <mack at macktronics.com> wrote:
>>>
>>>
>>>> I ran the procsystime dtrace script on bhyve when it was chewing up all
>>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime:
>>>>
>>>> root at olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT
>>>>
>>>> Tracing... Hit Ctrl-C to end...
>>>> dtrace: 158536 dynamic variable drops with non-empty dirty list
>>>> dtrace: 207447 dynamic variable drops with non-empty dirty list
>>>> dtrace: 189205 dynamic variable drops with non-empty dirty list
>>>> dtrace: 164341 dynamic variable drops with non-empty dirty list
>>>> dtrace: 246307 dynamic variable drops with non-empty dirty list
>>>> dtrace: 187640 dynamic variable drops with non-empty dirty list
>>>> dtrace: 214771 dynamic variable drops with non-empty dirty list
>>>> dtrace: 221265 dynamic variable drops with non-empty dirty list
>>>> ^C
>>>> dtrace: 243468 dynamic variable drops with non-empty dirty list
>>>>
>>>> Elapsed Times for processes bhyve,
>>>>
>>>> SYSCALL TIME (ns)
>>>> _umtx_op 18349
>>>> writev 135712
>>>> preadv 16175267
>>>> pwritev 22924378
>>>> ioctl 4353897920
>>>> TOTAL: 4393151626
>>>>
>>>> CPU Times for processes bhyve,
>>>>
>>>> SYSCALL TIME (ns)
>>>> _umtx_op 8815
>>>> writev 103145
>>>> pwritev 10647023
>>>> preadv 15159976
>>>> ioctl 3943399889
>>>> TOTAL: 3969318848
>>>>
>>>> Syscall Counts for processes bhyve,
>>>>
>>>> SYSCALL COUNT
>>>> _umtx_op 17
>>>> writev 43
>>>> pwritev 639
>>>> preadv 1183
>>>> ioctl 652782
>>>> TOTAL: 654664
>>>>
>>>> Not sure if that helps or not. Hotkernel shows the cpu in the
>>>> kernel`acpi_cpu_c1 call most of the time, normal ?
>>>>
>>>> Dan
>>>>
>>>> root at olive:/usr/share/dtrace/****toolkit # ./hotkernel
>>>>
>>>> Sampling... Hit Ctrl-C to end.
>>>> ^C
>>>> FUNCTION COUNT PCNT
>>>> zfs.ko`space_map_sync 1 0.0%
>>>> kernel`syscall_thread_exit 1 0.0%
>>>> kernel`should_yield 1 0.0%
>>>> zfs.ko`lzjb_compress 1 0.0%
>>>> zfs.ko`dbuf_rele 1 0.0%
>>>> zfs.ko`dsl_pool_tempreserve_****space 1 0.0%
>>>>
>>>> kernel`_bus_dmamap_load_buffer 1 0.0%
>>>> kernel`cpu_idleclock 1 0.0%
>>>> kernel`_rw_runlock_cookie 1 0.0%
>>>> kernel`pmap_enter 1 0.0%
>>>> kernel`__lockmgr_args 1 0.0%
>>>> zfs.ko`vdev_geom_io_start 1 0.0%
>>>> kernel`ahci_ch_intr 1 0.0%
>>>> kernel`__rw_rlock 1 0.0%
>>>> kernel`VOP_UNLOCK_APV 1 0.0%
>>>> zfs.ko`arc_buf_thaw 1 0.0%
>>>> kernel`atomic_set_int 1 0.0%
>>>> kernel`g_io_schedule_down 1 0.0%
>>>> kernel`__mtx_lock_flags 1 0.0%
>>>> kernel`_callout_stop_safe 1 0.0%
>>>> kernel`sched_idletd 1 0.0%
>>>> kernel`_sx_slock 1 0.0%
>>>> vmm.ko`lapic_set_intr 1 0.0%
>>>> kernel`cpusetobj_ffs 1 0.0%
>>>> zfs.ko`dbuf_hold 1 0.0%
>>>> vmm.ko`vlapic_intr_accepted 1 0.0%
>>>> zfs.ko`sa_attr_op 1 0.0%
>>>> zfs.ko`dmu_buf_hold_array_by_****dnode 1 0.0%
>>>>
>>>> zfs.ko`taskq_run_safe 1 0.0%
>>>> kernel`vm_reserv_free_page 1 0.0%
>>>> kernel`_sx_sunlock 1 0.0%
>>>> kernel`vn_rlimit_fsize 1 0.0%
>>>> zfs.ko`vdev_queue_io_to_issue 1 0.0%
>>>> kernel`vn_write 1 0.0%
>>>> kernel`lock_init 1 0.0%
>>>> vmm.ko`vmx_resume 1 0.0%
>>>> zfs.ko`zio_gang_tree_free 1 0.0%
>>>> zfs.ko`dbuf_rele_and_unlock 1 0.0%
>>>> kernel`mnt_vnode_next_active 1 0.0%
>>>> kernel`vm_page_remove 1 0.0%
>>>> zfs.ko`vdev_queue_deadline_****compare 1 0.0%
>>>>
>>>> kernel`zone_free_item 1 0.0%
>>>> vmm.ko`vmx_launch 1 0.0%
>>>> kernel`m_pullup 1 0.0%
>>>> kernel`pmap_change_wiring 1 0.0%
>>>> kernel`sched_userret 1 0.0%
>>>> zfs.ko`dbuf_write_done 1 0.0%
>>>> zfs.ko`zio_execute 1 0.0%
>>>> kernel`PHYS_TO_VM_PAGE 2 0.0%
>>>> zfs.ko`avl_rotation 2 0.0%
>>>> kernel`ehci_device_isoc_fs_****enter 2 0.0%
>>>>
>>>> vmm.ko`vm_exitinfo 2 0.0%
>>>> zfs.ko`zfs_freebsd_write 2 0.0%
>>>> vmm.ko`vcpu_stats 2 0.0%
>>>> kernel`atomic_add_long 2 0.0%
>>>> vmm.ko`0xffffffff81 2 0.0%
>>>> kernel`vm_reserv_alloc_page 2 0.0%
>>>> kernel`pagezero 2 0.0%
>>>> kernel`bzero 2 0.0%
>>>> zfs.ko`fletcher_2_native 2 0.0%
>>>> vmm.ko`vmx_getreg 2 0.0%
>>>> vmm.ko`vm_guest_msrs 3 0.0%
>>>> vmm.ko`vmx_setreg 3 0.0%
>>>> vmm.ko`vm_get_register 3 0.0%
>>>> kernel`sx_try_xlock_ 3 0.0%
>>>> kernel`vm_radix_lookup_le 3 0.0%
>>>> vmm.ko`lapic_pending_intr 4 0.0%
>>>> kernel`syscall_thread_enter 4 0.0%
>>>> kernel`cpu_set_syscall_retval 4 0.0%
>>>> kernel`__mnt_vnode_next_all 4 0.0%
>>>> kernel`vfs_stdsync 4 0.0%
>>>> vmm.ko`vmm_fetch_instruction 5 0.0%
>>>> vmm.ko`vlapic_op_mem_read 6 0.0%
>>>> kernel`_sx_xunlock 7 0.0%
>>>> vmm.ko`vmm_emulate_instruction 7 0.0%
>>>> vmm.ko`vlapic_op_mem_write 7 0.0%
>>>> kernel`__mtx_lock_spin_flags 8 0.0%
>>>> vmm.ko`lapic_timer_tick 8 0.0%
>>>> zfs.ko`list_next 8 0.0%
>>>> zfs.ko`l2arc_feed_thread 8 0.0%
>>>> kernel`userret 9 0.0%
>>>> vmm.ko`vm_gpa2hpa 9 0.0%
>>>> vmm.ko`vmm_decode_instruction 11 0.0%
>>>> vmm.ko`ept_vmmmap_get 11 0.0%
>>>> vmm.ko`vm_lapic 11 0.0%
>>>> kernel`spinlock_enter 14 0.0%
>>>> kernel`__mtx_lock_sleep 15 0.0%
>>>> vmm.ko`vcpu_set_state 16 0.0%
>>>> vmm.ko`vm_nmi_pending 16 0.0%
>>>> kernel`malloc_type_zone_****allocated 16 0.0%
>>>>
>>>> kernel`devfs_ioctl_f 16 0.0%
>>>> kernel`pmap_kextract 17 0.0%
>>>> kernel`sys_ioctl 18 0.0%
>>>> kernel`malloc 21 0.0%
>>>> kernel`cpu_fetch_syscall_args 24 0.0%
>>>> kernel`uma_zfree_arg 28 0.0%
>>>> vmm.ko`vlapic_pending_intr 29 0.0%
>>>> kernel`trap 30 0.0%
>>>> kernel`hpet_get_timecount 31 0.0%
>>>> dtrace.ko`dtrace_trap 35 0.0%
>>>> vmm.ko`vlapic_update_ppr 39 0.0%
>>>> vmm.ko`vmmdev_ioctl 40 0.0%
>>>> kernel`copyout 43 0.0%
>>>> kernel`critical_enter 44 0.0%
>>>> kernel`copyin 51 0.1%
>>>> vmm.ko`vmcs_read 53 0.1%
>>>> kernel`fpusave 61 0.1%
>>>> kernel`uma_zalloc_arg 62 0.1%
>>>> kernel`devvn_refthread 69 0.1%
>>>> kernel`atomic_load_acq_int 70 0.1%
>>>> kernel`fpurestore 70 0.1%
>>>> vmm.ko`vlapic_timer_tick 75 0.1%
>>>> kernel`amd64_syscall 77 0.1%
>>>> kernel`bcopy 91 0.1%
>>>> kernel`fpudna 103 0.1%
>>>> kernel`free 103 0.1%
>>>> kernel`critical_exit 105 0.1%
>>>> kernel`dev_relthread 108 0.1%
>>>> kernel`spinlock_exit 119 0.1%
>>>> kernel`kern_ioctl 138 0.1%
>>>> kernel`fpuexit 242 0.3%
>>>> kernel`0xffffffff80 306 0.3%
>>>> vmm.ko`restore_host_msrs 398 0.4%
>>>> vmm.ko`vm_run 428 0.4%
>>>> vmm.ko`restore_guest_msrs 433 0.5%
>>>> kernel`cpu_idle 906 1.0%
>>>> vmm.ko`vmx_run 9266 9.7%
>>>> kernel`acpi_cpu_c1 81134 85.2%
>>>>
>>>> Hope this helps, let me know if you have any ideas on what to try next.
>>>>
>>>>
>>>> Could you try the following command to get vcpu stats?
>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0
>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1
>>>
>>> best
>>> Neel
>>>
>>>
>>> Dan
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, 29 Mar 2013, Dan Mack wrote:
>>>>
>>>> On Thu, 28 Mar 2013, Peter Grehan wrote:
>>>>
>>>>>
>>>>> Hi Dan,
>>>>>
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6
>>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest.
>>>>>>
>>>>>>
>>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM.
>>>>>
>>>>> The issue you are seeing looks a bit like what happens when systems are
>>>>>
>>>>>> heavily oversubscribed and vCPUs can't get enough run time.
>>>>>>
>>>>>>
>>>>> I was running in on an idle system; only bhyve was running.
>>>>>
>>>>>
>>>>> How much spare RAM is on the host when you started the vm ? Also, is
>>>>>
>>>>>> there much running on the host when the guest is started ?
>>>>>>
>>>>>>
>>>>> 64GB
>>>>>
>>>>> Here's some more information (sorry for the formatting):
>>>>>
>>>>> Case 1: single vCPU via:
>>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47
>>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s
>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
>>>>>
>>>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB
>>>>> free RAM, nothing else clocking CPU cycles):
>>>>>
>>>>> root at olive:~ # top -bH
>>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31
>>>>> 14:11:51
>>>>> 24 processes: 1 running, 23 sleeping
>>>>>
>>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free
>>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other
>>>>> Swap:
>>>>>
>>>>>
>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00%
>>>>> bhyve{coco vcpu 0}
>>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd
>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd
>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh
>>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00%
>>>>> bhyve{coco mevent}
>>>>>
>>>>>
>>>>> So, case #1 all is normal. I was able to build a kernel with very good
>>>>> performance.
>>>>>
>>>>> The boot sequence looks like the following:
>>>>>
>>>>> root at olive:/vms/coco # ./vmrun.sh coco
>>>>> Launching virtual machine "coco" ...
>>>>> Consoles: userboot
>>>>>
>>>>> FreeBSD/amd64 User boot, Revision 1.1
>>>>> (root at olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178
>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08]
>>>>>
>>>>> /
>>>>> ______ ____ _____ _____
>>>>> | ____| | _ \ / ____| __ \
>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | |
>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | |
>>>>> | | | | | __/ __/| |_) |____) | |__| |
>>>>> | | | | | | || | | |
>>>>> |_| |_| \___|\___||____/|_____/|_____/ ```
>>>>> `
>>>>> s` `.....---.......--.```
>>>>> -/
>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:`
>>>>> +.
>>>>> | | yo`:. :o
>>>>> `+-
>>>>> | 1. Boot Multi User [Enter] | y/ -/` -o/
>>>>> | 2. Boot [S]ingle User | .-
>>>>> ::/sy+:.
>>>>> | 3. [Esc]ape to loader prompt | / `--
>>>>> /
>>>>> | 4. Reboot | `:
>>>>> :`
>>>>> | | `:
>>>>> :`
>>>>> | Options: | /
>>>>> /
>>>>> | 5. Configure Boot [O]ptions... | .-
>>>>> -.
>>>>> | | -- -.
>>>>> | | `:` `:`
>>>>> | | .-- `--.
>>>>> | | .---.....----.
>>>>> +-----------------------------****------------+
>>>>>
>>>>>
>>>>>
>>>>> Booting...
>>>>> Copyright (c) 1992-2013 The FreeBSD Project.
>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>>>> The Regents of the University of California. All rights
>>>>> reserved.
>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
>>>>> root at cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64
>>>>>
>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU)
>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d
>>>>> Stepping = 7
>>>>> Features=0x8fa3ab7f<FPU,VME,****DE,PSE,TSC,MSR,PAE,CX8,APIC,**
>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE>
>>>>> Features2=0x83bee217<SSE3,****PCLMULQDQ,DTES64,DS_CPL,SSSE3,****
>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,**
>>>>> TSCDLT,**AESNI,HV>
>>>>> AMD Features=0x2c100800<SYSCALL,****NX,Page1GB,RDTSCP,LM>
>>>>>
>>>>> AMD Features2=0x1<LAHF>
>>>>> TSC: P-state invariant
>>>>> real memory = 5368709120 (5120 MB)
>>>>> avail memory = 3957534720 (3774 MB)
>>>>> Event timer "LAPIC" quality 400
>>>>> ACPI APIC Table: <BHYVE BVMADT >
>>>>> random device not loaded; using insecure entropy
>>>>> ioapic0: Changing APIC ID to 1
>>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard
>>>>> kbd0 at kbdmux0
>>>>> acpi0: <BHYVE BVXSDT> on motherboard
>>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
>>>>> Event timer "RTC" frequency 32768 Hz quality 0
>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
>>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>>>>> pci0: <ACPI PCI bus> on pcib0
>>>>> pcib0: no PRT entry for 0.31.INTA
>>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem
>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0
>>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0
>>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df
>>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem
>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0
>>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1
>>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>> vtblk0: 8192MB (16777216 512 byte sectors)
>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on
>>>>> pci0
>>>>> uart2: console (9600,n,8,1)
>>>>> Timecounters tick every 10.000 msec
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000
>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf.
>>>>>
>>>>> Setting hostid: 0x24bfd025.
>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart.
>>>>> Starting file system checks:
>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
>>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7%
>>>>> fragmentation)
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> Mounting local file systems:.
>>>>> Writing entropy file:.
>>>>> Setting hostname: cocopuff.example.com.
>>>>> vtnet0: link state changed to UP
>>>>> Starting Network: lo0 vtnet0.
>>>>> lo0: flags=8049<UP,LOOPBACK,****RUNNING,MULTICAST> metric 0 mtu 16384
>>>>> options=600003<RXCSUM,TXCSUM,****RXCSUM_IPV6,TXCSUM_IPV6>
>>>>>
>>>>> inet6 ::1 prefixlen 128
>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>>> inet 127.0.0.1 netmask 0xff000000
>>>>> nd6 options=21<PERFORMNUD,AUTO_****LINKLOCAL>
>>>>> vtnet0: flags=8943<UP,BROADCAST,****RUNNING,PROMISC,SIMPLEX,****
>>>>> MULTICAST>
>>>>> metric 0 mtu 1500
>>>>> options=80028<VLAN_MTU,JUMBO_****MTU,LINKSTATE>
>>>>>
>>>>> ether 00:a0:98:ac:0d:df
>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
>>>>> nd6 options=29<PERFORMNUD,****IFDISABLED,AUTO_LINKLOCAL>
>>>>>
>>>>> media: Ethernet 1000baseT <full-duplex>
>>>>> status: active
>>>>> Starting devd.
>>>>> add net default: gateway 172.22.22.252
>>>>> add net ::ffff:0.0.0.0: gateway ::1
>>>>> add net ::0.0.0.0: gateway ::1
>>>>> add net fe80::: gateway ::1
>>>>> add net ff02::: gateway ::1
>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
>>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>>> Creating and/or trimming log files.
>>>>> Starting syslogd.
>>>>> Clearing /tmp (X related).
>>>>> Updating motd:.
>>>>> Performing sanity check on sshd configuration.
>>>>> Starting sshd.
>>>>> Starting cron.
>>>>> Starting background file system checks in 60 seconds.
>>>>>
>>>>> Fri Mar 29 14:09:20 CDT 2013
>>>>>
>>>>> FreeBSD/amd64 (cocopuff.example.com) (console)
>>>>>
>>>>> login:
>>>>>
>>>>>
>>>>> ====== Next, boot the same image with two (2) vCPU:
>>>>>
>>>>> Command run by vmrun.sh:
>>>>>
>>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42
>>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s
>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco
>>>>>
>>>>> Top shows it running out of control:
>>>>>
>>>>> root at olive:~ # top -bH
>>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12
>>>>> 14:17:32
>>>>> 25 processes: 3 running, 22 sleeping
>>>>>
>>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free
>>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other
>>>>> Swap:
>>>>>
>>>>>
>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00%
>>>>> bhyve{coco vcpu 1}
>>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48%
>>>>> bhyve{coco vcpu 0}
>>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd
>>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd
>>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd
>>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron
>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd
>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh
>>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00%
>>>>> bhyve{coco mevent}
>>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh
>>>>>
>>>>> Even though, it's just sitting at the login prompt:
>>>>>
>>>>> root at olive:/vms/coco # ./vmrun.sh coco
>>>>> Launching virtual machine "coco" ...
>>>>> Consoles: userboot
>>>>>
>>>>> FreeBSD/amd64 User boot, Revision 1.1
>>>>> (root at olive.example.com, Thu Mar 28 21:20:49 CDT 2013)
>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178
>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08]
>>>>>
>>>>> /
>>>>> ______ ____ _____ _____
>>>>> | ____| | _ \ / ____| __ \
>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | |
>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | |
>>>>> | | | | | __/ __/| |_) |____) | |__| |
>>>>> | | | | | | || | | |
>>>>> |_| |_| \___|\___||____/|_____/|_____/ ```
>>>>> `
>>>>> s` `.....---.......--.```
>>>>> -/
>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:`
>>>>> +.
>>>>> | | yo`:. :o
>>>>> `+-
>>>>> | 1. Boot Multi User [Enter] | y/ -/` -o/
>>>>> | 2. Boot [S]ingle User | .-
>>>>> ::/sy+:.
>>>>> | 3. [Esc]ape to loader prompt | / `--
>>>>> /
>>>>> | 4. Reboot | `:
>>>>> :`
>>>>> | | `:
>>>>> :`
>>>>> | Options: | /
>>>>> /
>>>>> | 5. Configure Boot [O]ptions... | .-
>>>>> -.
>>>>> | | -- -.
>>>>> | | `:` `:`
>>>>> | | .-- `--.
>>>>> | | .---.....----.
>>>>> +-----------------------------****------------+
>>>>>
>>>>>
>>>>>
>>>>> Booting...
>>>>> Copyright (c) 1992-2013 The FreeBSD Project.
>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>>>> The Regents of the University of California. All rights
>>>>> reserved.
>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013
>>>>> root at cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64
>>>>>
>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU)
>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d
>>>>> Stepping = 7
>>>>> Features=0x8fa3ab7f<FPU,VME,****DE,PSE,TSC,MSR,PAE,CX8,APIC,**
>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE>
>>>>> Features2=0x83bee217<SSE3,****PCLMULQDQ,DTES64,DS_CPL,SSSE3,****
>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,**
>>>>> TSCDLT,**AESNI,HV>
>>>>> AMD Features=0x2c100800<SYSCALL,****NX,Page1GB,RDTSCP,LM>
>>>>>
>>>>> AMD Features2=0x1<LAHF>
>>>>> TSC: P-state invariant
>>>>> real memory = 5368709120 (5120 MB)
>>>>> avail memory = 3957534720 (3774 MB)
>>>>> Event timer "LAPIC" quality 400
>>>>> ACPI APIC Table: <BHYVE BVMADT >
>>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
>>>>> FreeBSD/SMP: 2 package(s) x 1 core(s)
>>>>> cpu0 (BSP): APIC ID: 0
>>>>> cpu1 (AP): APIC ID: 1
>>>>> random device not loaded; using insecure entropy
>>>>> ioapic0: Changing APIC ID to 2
>>>>> ioapic0 <Version 1.1> irqs 0-16 on motherboard
>>>>> kbd0 at kbdmux0
>>>>> acpi0: <BHYVE BVXSDT> on motherboard
>>>>> atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
>>>>> Event timer "RTC" frequency 32768 Hz quality 0
>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
>>>>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>>>>> pci0: <ACPI PCI bus> on pcib0
>>>>> pcib0: no PRT entry for 0.31.INTA
>>>>> virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem
>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0
>>>>> vtnet0: <VirtIO Networking Adapter> on virtio_pci0
>>>>> virtio_pci0: host features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>> virtio_pci0: negotiated features: 0x18020 <Status,MrgRxBuf,MacAddress>
>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df
>>>>> virtio_pci1: <VirtIO PCI Block adapter> port 0x2040-0x207f mem
>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0
>>>>> vtblk0: <VirtIO Block Adapter> on virtio_pci1
>>>>> virtio_pci1: host features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>> virtio_pci1: negotiated features: 0x10000004 <RingIndirect,MaxNumSegs>
>>>>> vtblk0: 8192MB (16777216 512 byte sectors)
>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on
>>>>> pci0
>>>>> uart2: console (9600,n,8,1)
>>>>> Timecounters tick every 10.000 msec
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> SMP: AP CPU #1 Launched!
>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]...
>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf.
>>>>>
>>>>> Setting hostid: 0x24bfd025.
>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart.
>>>>> Starting file system checks:
>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
>>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7%
>>>>> fragmentation)
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
>>>>> Mounting local file systems:.
>>>>> Writing entropy file:.
>>>>> Setting hostname: cocopuff.example.com.
>>>>> vtnet0: link state changed to UP
>>>>> Starting Network: lo0 vtnet0.
>>>>> lo0: flags=8049<UP,LOOPBACK,****RUNNING,MULTICAST> metric 0 mtu 16384
>>>>> options=600003<RXCSUM,TXCSUM,****RXCSUM_IPV6,TXCSUM_IPV6>
>>>>>
>>>>> inet6 ::1 prefixlen 128
>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>>> inet 127.0.0.1 netmask 0xff000000
>>>>> nd6 options=21<PERFORMNUD,AUTO_****LINKLOCAL>
>>>>> vtnet0: flags=8943<UP,BROADCAST,****RUNNING,PROMISC,SIMPLEX,****
>>>>> MULTICAST>
>>>>> metric 0 mtu 1500
>>>>> options=80028<VLAN_MTU,JUMBO_****MTU,LINKSTATE>
>>>>>
>>>>> ether 00:a0:98:ac:0d:df
>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255
>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1
>>>>> nd6 options=29<PERFORMNUD,****IFDISABLED,AUTO_LINKLOCAL>
>>>>>
>>>>> media: Ethernet 1000baseT <full-duplex>
>>>>> status: active
>>>>> Starting devd.
>>>>> add net default: gateway 172.22.22.252
>>>>> add net ::ffff:0.0.0.0: gateway ::1
>>>>> add net ::0.0.0.0: gateway ::1
>>>>> add net fe80::: gateway ::1
>>>>> add net ff02::: gateway ::1
>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
>>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>>> Creating and/or trimming log files.
>>>>> Starting syslogd.
>>>>> syslogd: timed out waiting for child
>>>>> /etc/rc: WARNING: failed to start syslogd
>>>>> Clearing /tmp (X related).
>>>>> Updating motd:.
>>>>> Performing sanity check on sshd configuration.
>>>>> Starting sshd.
>>>>> Starting cron.
>>>>> Starting background file system checks in 60 seconds.
>>>>>
>>>>> Thu Apr 18 02:06:43 CDT 2013
>>>>>
>>>>> FreeBSD/amd64 (cocopuff.example.com) (console)
>>>>>
>>>>> login:
>>>>>
>>>>>
>>>>> ========== end of boot sequence
>>>>>
>>>>> So, something is broken with SMP on the guest, I'd be happy to help just
>>>>> let me know.
>>>>>
>>>>> Dan
>>>>>
>>>>> ______________________________****_________________
>>>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org> <
>>>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization at freebsd.org>>mailing
>>>>> list
>>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****
>>>>> virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>
>>>>> <http://lists.**freebsd.org/mailman/listinfo/**freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
>>>>>>
>>>>> To unsubscribe, send any mail to "freebsd-virtualization-**
>>>>> unsubscribe at freebsd.org <freebsd-virtualization-**
>>>>> unsubscribe at freebsd.org<freebsd-virtualization-unsubscribe at freebsd.org>
>>>>>> "
>>>>>
>>>>> ______________________________****_________________
>>>>>
>>>> freebsd-virtualization@**freeb**sd.org <http://freebsd.org> <
>>>> freebsd-virtualization@**freebsd.org<freebsd-virtualization at freebsd.org>>mailing
>>>> list
>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization<http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization>
>>>> <http://lists.**freebsd.org/mailman/listinfo/**freebsd-virtualization<http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>
>>>>>
>>>> To unsubscribe, send any mail to "freebsd-virtualization-**
>>>> unsubscribe at freebsd.org <freebsd-virtualization-**
>>>> unsubscribe at freebsd.org <freebsd-virtualization-unsubscribe at freebsd.org>
>>>>> "
>>>>
>>>>
>>>
>
More information about the freebsd-virtualization
mailing list