my experience with FreeBSD Xen guest

Wed Jun 17 10:55:23 UTC 2009

Hello

I've spent last four days trying to get a stable 8.0-CURRENT on a CentOS
5.3. I am not very versed in BSD, so some of the problems may come from
applying generic Unix approach to BSD environment.

* disk operations are unstable. I attach dmesg from boot of a freshly
fscked filesystem (16G LVM on host), the two stack backtraces are
routinely happening during boot sequence. Also massive operations like
buildworld usually lead to more syscall errors and routine filesystem
corruption

* sometimes the guest just crashes to kernel debugger, I have not
isolated any particular reason why it happens

* during the four days I only managed to successfully compile kernel
once and world twice, installworld always fails, the last one with
following error:

[...]
===> secure (install)
===> secure/lib (install)
===> secure/lib/libcrypto (install)
".depend", line 19879: Unassociated shell command "��ɫ�������'�F/^�>�
闆���
".depend", line 19880: Need an operator
".depend", line 19881: Need an operator
".depend", line 19883: Missing dependency operator
".depend", line 19884: Need an operator
No closing parenthesis in archive specification
".depend", line 19885: Error in archive specification: "%
����Z��F�����_o����o�x;����w,�
�Fގ�/7�͚��"
No closing parenthesis in archive specification
".depend", line 19886: Error in archive specification: "�����"
".depend", line 19888: Need an operator
".depend", line 19889: Need an operator
".depend", line 19893: Need an operator
".depend", line 19894: Missing dependency operator
No closing parenthesis in archive specification
".depend", line 19899: Error in archive specification:
"6pH�EX��
��]����9�䝺8�"
".depend", line 19910: Need an operator
make: fatal errors encountered -- cannot continue
*** Error code 1

Stop in /usr/src/secure/lib.
*** Error code 1

Stop in /usr/src/secure.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.

Also, during the mentioned buildworld the following syscall error
happened:

lock order reversal:
 1st 0xc393b81c snaplk (snaplk)
@ /usr/home/alex/bsd/sys/kern/vfs_vnops.c:295
 2nd 0xc441037c ufs (ufs)
@ /usr/home/alex/bsd/sys/ufs/ffs/ffs_snapshot.c:1587
KDB: stack backtrace:
X_db_sym_numargs(c0356957,e4d2f640,c0116b95,c01072bb,c035979e,...) at
X_db_sym_numargs+0x146
kdb_backtrace(c01072bb,c035979e,c36ccf10,c36ccbd0,e4d2f69c,...) at
kdb_backtrace+0x29
witness_display_spinlock(c035979e,c441037c,c034c8dc,c36ccbd0,c0371652,...) at witness_display_spinlock+0x75
witness_checkorder(c441037c,9,c0371652,633,0,...) at witness_checkorder
+0x839
__lockmgr_args(c441037c,80000,0,0,0,...) at __lockmgr_args+0x797
ffs_snapremove(c4410324,0,e4d2f7d8,e4d2f7d8,c035e67e,...) at
ffs_snapremove+0x11f
ffs_truncate(c4410324,0,0,c00,0,...) at ffs_truncate+0x5f7
ufs_inactive(e4d2fa58,c4410398,c4410324,c4410398,e4d2fa70,...) at
ufs_inactive+0x218
VOP_INACTIVE_APV(c03b0200,e4d2fa58,c0360aae,925,c03c2160,...) at
VOP_INACTIVE_APV+0xa5
vref(c03b0200,e4d2fa8c,c0360aae,8ab,127,...) at vref+0x136e
vput(c4410324,e4d2fac8,c0361c78,127,0,...) at vput+0x1cd
vn_close(c4410324,1,c3b02600,c3b1f240,c0c74880,...) at vn_close+0x19a
vn_close(c39611f8,c3b1f240,3,0,c39611f8,...) at vn_close+0x294
_fdrop(c39611f8,c3b1f240,e4d2fb94,c01169dc,0,c0507a20,c3b1f2e4,c0507a20,c039b680,c4bd432c,6f4,c034e229,e4d2fbbc,c00de690,c4bd432c,8,c034e229,6f4) at _fdrop+0x43
closef(c39611f8,c3b1f240,6f4,6f1,c3b1f2e4,...) at closef+0x290
fdfree(c3b1f240,0,c034eafc,103,1,...) at fdfree+0x3ea
exit1(c3b1f240,0,e4d2fd3c,c0329e23,c3b1f240,...) at exit1+0x533
sys_exit(c3b1f240,e4d2fd08,4,c035a0f3,c039227c,...) at sys_exit+0x1d
syscall(e4d2fd48) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x22
--- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x28129f63, esp =
0xbf7fe33c, ebp = 0xbf7fe348 ---

I guess that the both of the errors are related to fs corruption that
happened during the course of buildworld (before I started buildworld I
launched the guest from different image and fscked the / clean).

The filesystem problems are severe, and block us from using FreeBSD/Xen.

Regards

Alex
-------------- next part --------------
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #0 r: Tue Jun 16 13:53:39 UTC 2009
    alex at dev32:/usr/obj/usr/home/alex/bsd/sys/XEN
WARNING: WITNESS option enabled, expect reduced performance.
Xen reported: 2327.498 MHz processor.
Timecounter "ixen" frequency 1000000000 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz (2327.50-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x1067a  Stepping = 10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE>
  AMD Features=0x20000000<LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant

Instruction TLB: 4 KB Pages, 4-way set associative, 128 entries
1st-level instruction cache: 32 KB, 8-way set associative, 64 byte line size
1st-level data cache: 32 KB, 8-way set associative, 64 byte line size
L2 cache: 6144 kbytes, 16-way associative, 64 bytes/line
real memory  = 1073741824 (1024 MB)
Physical memory chunk(s):
0x0000000000895000 - 0x000000003edb2fff, 1045553152 bytes (255262 pages)
avail memory = 1041563648 (993 MB)
APIC: Using the MPTable enumerator.
SMP: Added CPU 0 (BSP)
ULE: setup cpu 0
[XEN] IPI cpu=0 irq=128 vector=RESCHEDULE_VECTOR (0)
[XEN] IPI cpu=0 irq=129 vector=CALL_FUNCTION_VECTOR (1)
Event-channel device installed.
mem: <memory>
Pentium Pro MTRR support enabled
null: <null device, zero device>
nfslock: pseudo-device
random: <entropy source, Software, Yarrow>
io: <I/O>
kbd0 at kbdmux0
Grant table initialized
xenbus0: <Xen Devices> on motherboard
xc0: <Xen Console> on motherboard
[XEN] xen_rtc_probe: probing Hypervisor RTC clock
rtc0: <Xen Hypervisor Clock> on motherboard
[XEN] xen_rtc_attach: attaching Hypervisor RTC clock
rtc0: registered as a time-of-day clock (resolution 1000000us)
npx0: INT 16 interface
Device configuration finished.
procfs registered
Timecounters tick every 10.000 msec
[XEN] hypervisor wallclock nudged; nudging TOD.
lo0: bpf attached
xbd0: 16384MB <Virtual Block Device> at device/vbd/51712 on xenbus0
GEOM: new disk xbd0
xbd2: 256MB <Virtual Block Device> at device/vbd/51744 on xenbus0
xn0: <Virtual Network Interface> at device/vif/0 on xenbus0
xn0: bpf attached
xn0: Ethernet address: 00:bd:c4:13:00:01
WARNING: WITNESS option enabled, expect reduced performance.
GEOM: new disk xbd2
Trying to mount root from ufs:xbd0
rtc0: [XEN] xen_rtc_gettime
rtc0: [XEN] xen_rtc_gettime: wallclock 1241776380 sec; 699847734 nsec
rtc0: [XEN] xen_rtc_gettime: uptime 3384929 sec; 293434434 nsec
rtc0: [XEN] xen_rtc_gettime: TOD 1245161309 sec; 993282168 nsec
start_init: trying /sbin/init
lock order reversal:
 1st 0xc3a8b8b8 ufs (ufs) @ /usr/home/alex/bsd/sys/kern/vfs_mount.c:1054
 2nd 0xc3a8d6a0 devfs (devfs) @ /usr/home/alex/bsd/sys/kern/vfs_subr.c:2084
KDB: stack backtrace:
X_db_sym_numargs(c0356957,c35b8824,c0116b95,c01072bb,c035979e,...) at X_db_sym_numargs+0x146
kdb_backtrace(c01072bb,c035979e,c36ccbd0,c36ccb00,c35b8880,...) at kdb_backtrace+0x29
witness_display_spinlock(c035979e,c3a8d6a0,c03489cb,c36ccb00,c0360aae,...) at witness_display_spinlock+0x75
witness_checkorder(c3a8d6a0,9,c0360aae,824,0,...) at witness_checkorder+0x839
__lockmgr_args(c3a8d6a0,80100,c3a8d6bc,0,0,...) at __lockmgr_args+0x797
vop_stdlock(c35b8988,c011693b,c0348c10,80100,c3a8d648,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c038f2e0,c35b8988,c39c2524,c03c21a0,c3a8d648,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c3a8d648,80100,c0360aae,824,8,...) at _vn_lock+0x5e
vget(c3a8d648,80100,c39c2480,15d,c0348b28,...) at vget+0xb9
devfs_allocv(c3952000,c394ca10,c35b8a20,9d,c0534698,...) at devfs_allocv+0x102
devfs_rules_apply(c394ca10,80000,c35b8c3c,42c,0,...) at devfs_rules_apply+0x14a
vfs_donmount(c39c2480,0,c3952100,c3952100,bf7fde39,...) at vfs_donmount+0x14bb
nmount(c39c2480,c35b8d08,c,c,c0394bb8,...) at nmount+0x72
syscall(c35b8d48) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x22
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280e7c3b, esp = 0xbf7fde0c, ebp = 0xbf7fe368 ---
lock order reversal:
 1st 0xd772de30 bufwait (bufwait) @ /usr/home/alex/bsd/sys/kern/vfs_bio.c:2558
 2nd 0xc3991e00 dirhash (dirhash) @ /usr/home/alex/bsd/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
X_db_sym_numargs(c0356957,e4d2fa8c,c0116b95,c01072bb,c035979e,...) at X_db_sym_numargs+0x146
kdb_backtrace(c01072bb,c035979e,c36ca4d0,c36ccc38,e4d2fae8,...) at kdb_backtrace+0x29
witness_display_spinlock(c035979e,c3991e00,c0373b42,c36ccc38,c03737d1,...) at witness_display_spinlock+0x75
witness_checkorder(c3991e00,9,c03737d1,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c3991e00,0,c03737d1,11d,d86a3018,...) at _sx_xlock+0x85
ufsdirhash_enduseful(0,d,c38c4800,d772ddd0,d86a3018,...) at ufsdirhash_enduseful+0x2f5
ufsdirhash_remove(c3b2a0e8,d86a3018,18,e4d2fb78,e4d2fb74,...) at ufsdirhash_remove+0x14
ufs_dirremove(c3b25b84,c3b291d0,500800c,1,9f,...) at ufs_dirremove+0xe5
ufs_itimes(e4d2fc44,e4d2fc54,c3b20860,c3b20860,0,...) at ufs_itimes+0x826
VOP_RMDIR_APV(c03b0200,e4d2fc44,2,0,28216238,...) at VOP_RMDIR_APV+0xa5
kern_rmdirat(c3b16000,ffffff9c,28216238,0,e4d2fc90,...) at kern_rmdirat+0x16b
kern_rmdir(c3b16000,28216238,0,e4d2fd3c,c0329e23,...) at kern_rmdir+0x27
rmdir(c3b16000,e4d2fd08,4,c036e7d5,c039315c,...) at rmdir+0x22
syscall(e4d2fd48) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x22
--- syscall (137, FreeBSD ELF32, rmdir), eip = 0x280d8bfb, esp = 0xbf7febfc, ebp = 0xbf7fec28 ---