Panics when booting i386 paravirtualized FreeBSD

Sun Apr 6 17:08:10 UTC 2014

On 05/04/14 11:41, Jonni Nakari wrote:
> Hello,
> I'm having problems booting the i386 paravirtualized (KERNCONF=XEN)
> kernel (10.0R) on Xen 4.4. My hardware is an old machine with two, dual
> core AMD Opteron 285s. Most times the booting the DomU panics with
> message "panic: can't schedule timer" but sometimes it goes a bit
> further and panics with "panic: removing pages from non-current pmap". I
> added the full boot log of both panics and Xen Domu configuration to the
> end of this message.
> 
> What could be causing those panics? Are there any recent developments of
> PV FreeBSD that I'm not aware of? I can compile custom kernels and help
> testing experimental patches.
> 
> The machine is not capable of running HVM virtualized (no VT-x/AMD-V) so
> I would like to get paravirtualized version of FreeBSD running, but it
> seems to me that the state of PV FreeBSD is quite bad currently. If I
> manage to get the DomU running, can I expect it to support multiple
> processors?

No, the Xen i386 PV port only supports UP.

> Can it handle more than 4 GB of RAM?

Theoretically yes, Xen only supports 32bit guests with PAE (which
implies support for more than 4GB of RAM).

> I read from a previous
> post to this list that multiprocessor support might be broken, but that
> message was from over a year ago. Also the maximum RAM support support
> comes from an old source. Ideally I would like the FreeBSD DomU to serve
> ZVOLs and NFS to other DomUs.
> 
> example of "can't schedule timer" panic:
> jonni at tara:/etc/xen$ sudo xl create maggie.pvfreebsd -c
> Parsing config from maggie.pvfreebsd
> WARNING: loader(8) metadata is missing!
> GDB: no debug ports present
> KDB: debugger backends: ddb
> KDB: current backend: ddb
> APIC: Using the MPTable enumerator.
> SMP: Added CPU 0 (BSP)
> Copyright (c) 1992-2014 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 10.0-RELEASE #0: Sat Apr  5 07:11:56 EEST 2014
>     jonni at maggie:/usr/obj/usr/src/sys/XEN i386
> FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
> WARNING: WITNESS option enabled, expect reduced performance.
> Xen reported: 2612.074 MHz processor.
> CPU: Dual Core AMD Opteron(tm) Processor 285 (2612.07-MHz 686-class CPU)
>   Origin = "AuthenticAMD"  Id = 0x20f12  Family = 0xf  Model = 0x21
> Stepping = 2
> Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE3
> 
> 6,MMX,FXSR,SSE,SSE2,HTT>
>   Features2=0x1<SSE3>
>   AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
>   AMD Features2=0x3<LAHF,CMP>
> Data TLB: 32 entries, fully associative
> Instruction TLB: 32 entries, fully associative
> L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
> L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way
> associative
> L2 internal cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 8-way
> associative
> WARNING: This architecture revision has known SMP hardware bugs which
> may cause random insta
> bility
> real memory  = 536870912 (512 MB)
> Physical memory chunk(s):
> 0x0000000000873000 - 0x000000001f6a0fff, 518184960 bytes (126510 pages)
> avail memory = 515543040 (491 MB)
> random device not loaded; using insecure entropy
> ULE: setup cpu 0
> nfslock: pseudo-device
> io: <I/O>
> Falling back to <Software, Yarrow> random adaptor
> random: <Software, Yarrow> initialized
> mem: <memory>
> Pentium Pro MTRR support enabled
> null: <null device, zero device>
> xc0: <Xen Console> on motherboard
> xenstore0: <XenStore> on motherboard
> Grant table initialized
> xen_et0: <Xen PV Clock> on motherboard
> Event timer "XENTIMER" frequency 1000000000 Hz quality 950
> Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
> xen_et0: registered as a time-of-day clock (resolution 10000000us,
> adjustment 5.000000000s)
> Device configuration finished.
> procfs registered
> panic: can't schedule timer
> cpuid = 0
> KDB: enter: panic
> [ thread pid 0 tid 100000 ]
> Stopped at      kdb_enter+0x3d: movl    $0,kdb_why
> db> bt
> Tracing pid 0 tid 100000 td 0xc068b710
> kdb_enter(c042ffc7,c042ffc7,c041c2b4,c0748bd4,8b3b77b1,...) at
> kdb_enter+0x3d/frame 0xc0748b80
> kassert_panic(c0748bd4,c0748c28,c008a9ca,c041c2b4,c01d3940,...) at
> kassert_panic+0x233/frame 0xc0748bbc
> panic(c041c2b4,c01d3940,246,c0633418,c08e1480,...) at panic+0x12/frame
> 0xc0748bc8
> network_connect(c286162c,a3d70a,0,0,0,0,0) at
> network_connect+0x3e4a/frame 0xc0748c28
> et_start(c286162c,a3d70a,0,0,0,c286162c,c0872cbc,229a5,1,1f4,0,3ce580,3ce580)
> at et_start+0x10b/frame 0xc0748c58
> cpu_idleclock(1,0,c046c817,1f4,c26fda40,...) at
> cpu_idleclock+0x3eb/frame 0xc0748ca0
> cpu_initclocks_bsp(c068bba8,0,c046c817,2a1,c04713b4,...) at
> cpu_initclocks_bsp+0xa83/frame 0xc0748cf8
> cpu_initclocks_bsp(c04b59a4,c042754f,0,0,c0748d58,...) at
> cpu_initclocks_bsp+0x3af/frame 0xc0748d18
> profclock_cnt(0,0,0,0,c0471534,...) at profclock_cnt+0x5bf/frame 0xc0748d30
> mi_startup(0,0,742000,0,0,...) at mi_startup+0xf7/frame 0xc0748d58
> btext() at btext+0x95
> db>

Not sure where this is coming from, the PV timer is shared with PVHVM
guests, and it seems to be working fine there.

> 
> example of "removing pages from non-current pmap" panic:
> jonni at tara:/etc/xen$ sudo xl create maggie.pvfreebsd -c
> Parsing config from maggie.pvfreebsd
> WARNING: loader(8) metadata is missing!
> GDB: no debug ports present
> KDB: debugger backends: ddb
> KDB: current backend: ddb
> APIC: Using the MPTable enumerator.
> SMP: Added CPU 0 (BSP)
> Copyright (c) 1992-2014 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 10.0-RELEASE #0: Sat Apr  5 07:11:56 EEST 2014
>     jonni at maggie:/usr/obj/usr/src/sys/XEN i386
> FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
> WARNING: WITNESS option enabled, expect reduced performance.
> Xen reported: 2612.074 MHz processor.
> CPU: Dual Core AMD Opteron(tm) Processor 285 (2612.07-MHz 686-class CPU)
>   Origin = "AuthenticAMD"  Id = 0x20f12  Family = 0xf  Model = 0x21
> Stepping = 2
> Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE3
> 
> 6,MMX,FXSR,SSE,SSE2,HTT>
>   Features2=0x1<SSE3>
>   AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
>   AMD Features2=0x3<LAHF,CMP>
> Data TLB: 32 entries, fully associative
> Instruction TLB: 32 entries, fully associative
> L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
> L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way
> associative
> L2 internal cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 8-way
> associative
> WARNING: This architecture revision has known SMP hardware bugs which
> may cause random insta
> bility
> real memory  = 536870912 (512 MB)
> Physical memory chunk(s):
> 0x0000000000873000 - 0x000000001f6a0fff, 518184960 bytes (126510 pages)
> avail memory = 515543040 (491 MB)
> random device not loaded; using insecure entropy
> ULE: setup cpu 0
> nfslock: pseudo-device
> io: <I/O>
> Falling back to <Software, Yarrow> random adaptor
> random: <Software, Yarrow> initialized
> mem: <memory>
> Pentium Pro MTRR support enabled
> null: <null device, zero device>
> xc0: <Xen Console> on motherboard
> xenstore0: <XenStore> on motherboard
> Grant table initialized
> xen_et0: <Xen PV Clock> on motherboard
> Event timer "XENTIMER" frequency 1000000000 Hz quality 950
> Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
> xen_et0: registered as a time-of-day clock (resolution 10000000us,
> adjustment 5.000000000s)
> Device configuration finished.
> procfs registered
> Timecounters tick every 10.000 msec
> tcp_init: net.inet.tcp.tcbhashsize auto tuned to 4096
> lo0: bpf attached
> xctrl0: <Xen Control Device> on xenstore0
> xenbusb_front0: <Xen Frontend Devices> on xenstore0
> xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
> xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
> xn0: bpf attached
> xn0: Ethernet address: 00:16:3e:7c:a1:9a
> xenbusb_back0: <Xen Backend Devices> on xenstore0
> xn0: backend features: feature-sg feature-gso-tcp4
> xbd0: 10240MB <Virtual Block Device> at device/vbd/832 on xenbusb_front0
> xbd0: attaching as ada1
> GEOM: new disk ada1
> random: unblocking device.
> TSC timecounter discards lower 1 bit(s)
> Timecounter "TSC-low" frequency 1306037000 Hz quality 800
> WARNING: WITNESS option enabled, expect reduced performance.
> Trying to mount root from ufs:ada1p2 []...
> start_init: trying /sbin/init
> pid 26 (sh), uid 0: exited on signal 4
> panic: removing pages from non-current pmap
> cpuid = 0
> KDB: enter: panic
> [ thread pid 26 tid 100035 ]
> Stopped at      kdb_enter+0x3d: movl    $0,kdb_why
> db> bt
> Tracing pid 26 tid 100035 td 0xc2a73000
> kdb_enter(c042ffc7,c042ffc7,c046acbe,d93b1984,c046acbe,...) at
> kdb_enter+0x3d/frame 0xd93b1918
> kassert_panic(c04b7148,100,c046acbe,d93b1984,d93b1984,...) at
> kassert_panic+0x233/frame 0xd93b1954
> kassert_panic(c046acbe,c0469fe4,e06,806,c0605668,...) at
> kassert_panic+0xea/frame 0xd93b1978
> pmap_remove_pages(c08eaa3c,8,c04296b8,c08eaa3c,c2a79000,...) at
> pmap_remove_pages+0xb3/frame 0xd93b19e8
> vmspace_exit(c2a73000,0,c04296b8,13f,4,...) at vmspace_exit+0xb1/frame
> 0xd93b1a10
> exit1(c2a73000,4,1a,c2a79218,0,...) at exit1+0x654/frame 0xd93b1a70
> sigexit(c2a73000,4,c0430060,b05,c2a78c48,...) at sigexit+0xbe4/frame
> 0xd93b1c20
> postsig(4,0,c0436965,10d,0,...) at postsig+0x3b2/frame 0xd93b1cdc
> ast(d93b1d18) at ast+0x388/frame 0xd93b1d0c
> vm86_biosret() at vm86_biosret+0x9c/frame 0xbf7fcbc0

I'm aware of this error, it can be solved by reverting r244237, although
there's something clearly wrong in the i386 Xen pmap code.

> Xen configuration maggie.pvfreebsd
> kernel = "/root/freebsd_10.0_pv_kernel"
> extra = "
> vfs.root.mountfrom=ufs:ada1p2,machdep.idle_mwait=0,boot_verbose=1,kern.hz=100"

The kern.hz=100 can be removed AFAIK.

> 
> memory = 512
> name = "maggie.pvfreebsd"
> vif = [ 'bridge=xenbr0']
> disk = [
>         'phy:/dev/mapper/tara--vg-maggie,hdb,w'
> ]
> vcpus = 1
> 