Debian 8 CPU stall

[Rajil Saraswat]

On 12/06/2016 10:50 PM, Peter Grehan wrote:
> Hi Rajil,
>
>> I get these messages in Debian 8 VM running in bhyve FreeBSD-11
>> release.  Any idea what could be the issue:
>>
>> INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0,
>> t=11047 jiffies, g=1038939, c=1038938, q=77)
>> INFO: Stall ended before state dump start
>
>  That's a sign that a vCPU wasn't able to run for an amount of time.
>
>  Is the system oversubscribed ? i.e. more vCPUs than physical CPUs ?
> Or, is the guest performing a lot of i/o ?
>
> later,
>
> Peter.
>

No the system is not oversubscribed. I have a 11 vCPU (1 on debian and 1
on ubuntu) on a 24 core machine. The debian jail is running x2go and an
ssh server for remote access, so the I/O shouldnt be an issue. The
ubuntu jail doesnt give out any warning messages though.

Following is the latest error i received in the debian vm:
[152444.353007] INFO: rcu_sched self-detected stall on CPU { 0}  (t=6809
jiffies g=2685598 c=2685597 q=6)
[152444.354261] sending NMI to all CPUs:
[152444.354270] NMI backtrace for cpu 0
[152444.354274] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-4-amd64
#1 Debian 3.16.36-1+deb8u2
[152444.354275] Hardware name:   BHYVE, BIOS 1.00 03/14/2014
[152444.354277] task: ffffffff8181a460 ti: ffffffff81800000 task.ti:
ffffffff81800000
[152444.354278] RIP: 0010:[<ffffffff81047a6d>]  [<ffffffff81047a6d>]
default_send_IPI_mask_sequence_phys+0xad/0xe0
[152444.354293] RSP: 0018:ffff88007fc03e18  EFLAGS: 00000046
[152444.354294] RAX: 0000000000000400 RBX: 000000000000a0ea RCX:
0000000000000000
[152444.354296] RDX: 0000000000000000 RSI: 0000000000000200 RDI:
0000000000000300
[152444.354297] RBP: ffffffff818e29c0 R08: ffffffff818e29c0 R09:
00000000000001bb
[152444.354298] R10: 0000000000000000 R11: ffff88007fc03b96 R12:
0000000000000400
[152444.354299] R13: 0000000000000096 R14: 0000000000000002 R15:
0000000000000000
[152444.354301] FS:  0000000000000000(0000) GS:ffff88007fc00000(0000)
knlGS:0000000000000000
[152444.354303] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[152444.354303] CR2: 00000000019a3240 CR3: 00000000797ca000 CR4:
00000000000406f0
[152444.354305] Stack:
[152444.354306]  0000000000000000 ffff88007fc0d6a0 ffffffff81853800
0000000000000000
[152444.354308]  ffffffff818e2f20 0000000000000006 ffffffff81853800
ffffffff81047cf3
[152444.354309]  ffff88007fc0d6a0 ffffffff810c73ea ffffffff00000007
ffffffff810c8f35
[152444.354311] Call Trace:
[152444.354313]  <IRQ>

[152444.354318]  [<ffffffff81047cf3>] ?
arch_trigger_all_cpu_backtrace+0xc3/0x140
[152444.354327]  [<ffffffff810c73ea>] ? rcu_check_callbacks+0x42a/0x670
[152444.354331]  [<ffffffff810c8f35>] ?
timekeeping_update.constprop.9+0x35/0x70
[152444.354335]  [<ffffffff810d1df0>] ? tick_sched_handle.isra.16+0x60/0x60
[152444.354343]  [<ffffffff81075f80>] ? update_process_times+0x40/0x70
[152444.354345]  [<ffffffff810d1db0>] ? tick_sched_handle.isra.16+0x20/0x60
[152444.354347]  [<ffffffff810d1e2c>] ? tick_sched_timer+0x3c/0x60
[152444.354351]  [<ffffffff8108c667>] ? __run_hrtimer+0x67/0x210
[152444.354353]  [<ffffffff8108ca69>] ? hrtimer_interrupt+0xe9/0x220
[152444.354359]  [<ffffffff8151b46b>] ? smp_apic_timer_interrupt+0x3b/0x50
[152444.354365]  [<ffffffff815194fd>] ? apic_timer_interrupt+0x6d/0x80
[152444.354366]  <EOI>

[152444.354374]  [<ffffffff8101da20>] ? mwait_idle+0xa0/0xa0
[152444.354381]  [<ffffffff81052c02>] ? native_safe_halt+0x2/0x10
[152444.354384]  [<ffffffff8101da39>] ? default_idle+0x19/0xd0
[152444.354389]  [<ffffffff810a9b44>] ? cpu_startup_entry+0x374/0x470
[152444.354392]  [<ffffffff81903076>] ? start_kernel+0x497/0x4a2
[152444.354394]  [<ffffffff81902a04>] ? set_init_arg+0x4e/0x4e
[152444.354396]  [<ffffffff81902120>] ? early_idt_handler_array+0x120/0x120
[152444.354398]  [<ffffffff8190271f>] ? x86_64_start_kernel+0x14d/0x15c
[152444.354399] Code: 8b 0c 25 00 53 5f ff 80 e5 10 75 f2 44 89 f8 c1 e0
18 89 04 25 10 53 5f ff 41 83 fe 02 44 89 e0 41 0f 45 c6 89 04 25 00 53
5f ff <eb> 91 4c 89 ef 57 9d 0f 1f 44 00 00 48 83 c4 08 5b 5d 41 5c 41

I use vm-bhyve for managing the jails which looks like this for debian8:

guest="linux"
loader="grub"
cpu=1
memory=2048M
network0_type="virtio-net"
network0_switch="lannetwork"
disk0_type="virtio-blk"
disk0_name="/dev/zvol/vmpool/os2"
disk0_dev="custom"
passthru0="2/0/0"
passthru1="2/0/1"


Thanks,
Rajil