From nobody Thu Nov 18 00:59:30 2021 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3437418568AF for ; Thu, 18 Nov 2021 00:59:42 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HvhK86BTHz4g33 for ; Thu, 18 Nov 2021 00:59:40 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 1173D3200EAD for ; Wed, 17 Nov 2021 19:59:33 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 17 Nov 2021 19:59:34 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:mime-version:content-type; s= fm1; bh=+cilvJv+cFMjyxbwn4oRvcgOxd7AousRvupP1QyVdGo=; b=h7kool7Z cZ/I0wT5IM/QW1WbS2kr1fmvGml6kjGoGjyxLIAM5LGYmTcnY2T4hQIBCcy0T2cD wV+APXLty5DYy0tjWVA+M0FkZK4mq1JwRRbUvL2L9CSI0pi3evd3J+RobH+04SzQ 9YMcHw7ZbvA30+uvYFkmSZxUZ7Jxtx9CXMNhMjkYoTYImMc7BpzcNFLjIi0yDUem ZhcOUkyDUcabA4UEe6x/NW2cv4qXi5d4CU+tcgOFdEY9gZjOEVl3TfIP7Dt7kEeo 45ZzJ0Rh41/KqhoP+S0wCctQQWChlWUn5UmRWnRkTBSzcnyJgOjnT0USbVFyemvK 5h3KvaivAeR6ZQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=+cilvJv+cFMjyxbwn4oRvcgOxd7Ao usRvupP1QyVdGo=; b=lR4YxPwRLOtoLIUCoUJh4aA77Gcl7VMOlkryTOUzPTWk+ 94Wn1UgHz3wpUwQX9zt+wFRrnNTWtxAhSR+g/0d3IC12BL0q/PScxTcfOkybjKxx kATPZ9ABJHTGOigk9Hb7jUw3a7To/r4+ITqYKLYkb1UT4/ZxWC4BhSxp5tU1LQ9q 5C7P5aNmlLZte3eMltX9h0IcwGnszN9rykJDUlPJy03G3Ka57JJXm4gYIZebItfN pDMuwuNg1d/cYpuelIjRvOtwOl0kafYfuNIu3/U3vvmN3yF2EjZk5y8kYGtasnbm VgaAsYRsNehM8cTHm3dk2o3Y3oy+QKpOu6W+Uw7aQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrfeehgddvkecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehgtderredttd dvnecuhfhrohhmpehtvggthhdqlhhishhtshcuoehtvggthhdqlhhishhtshesiiihgihs thdrnhgvtheqnecuggftrfgrthhtvghrnhepvefghffftdefkeelleehtdejledvhfdvge eijeevfffguddvhfetgeejueejueeinecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepthgvtghhqdhlihhsthhsseiihiigshhtrdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 17 Nov 2021 19:59:33 -0500 (EST) Date: Thu, 18 Nov 2021 00:59:30 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org Subject: cpu soft lockup bhyve host ubuntu guest Message-ID: Mail-Followup-To: freebsd-virtualization@freebsd.org List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="I7ODD5mjxYr1vVEs" Content-Disposition: inline X-Rspamd-Queue-Id: 4HvhK86BTHz4g33 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm1 header.b=h7kool7Z; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=lR4YxPwR; dmarc=none; spf=none (mx1.freebsd.org: domain of tech-lists@zyxst.net has no SPF policy when checking 64.147.123.19) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-6.49 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm1,messagingengine.com:s=fm1]; RWL_MAILSPIKE_POSSIBLE(0.00)[64.147.123.19:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; RCVD_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-0.99)[-0.991]; DMARC_NA(0.00)[zyxst.net]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; SIGNED_PGP(-2.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.19:from] X-ThisMailContainsUnwantedMimeParts: N --I7ODD5mjxYr1vVEs Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, I'm seeing an ubuntu guest freezing on a freebsd (12-stable) bhyve host. No issues are being reported in console.log, messages or all.log or dmesg on the freebsd server that show anything linking to lack of resources or cpu allocation. There's 44 cores, they're not oversubscribed. The guest vm has 4 vcores and 8G vram. but if I go to the guest console: [...] Ubuntu 21.10 foo ttyS0 foo login: [74237.857639] watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [sshd:5452] [74265.874169] watchdog: BUG: soft lockup - CPU#2 stuck for 52s! [sshd:5452] [74270.328733] rcu: INFO: rcu_sched self-detected stall on CPU [74270.329269] rcu: 2-....: (14862 ticks this GP) idle=3D68a/1/0x4000000000000000 softirq=3D1242684/1242684 fqs=3D7432 [74296.627733] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [swapper/1:0] [74297.892134] watchdog: BUG: soft lockup - CPU#2 stuck for 82s! [sshd:5452] hit return a few times and it unlocks itself. I wonder if it's going to sleep?! surely not I see this sort of message in ubuntu's dmesg: [...] [74237.857639] watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [sshd:5452] [74237.858302] Modules linked in: xt_recent dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common rapl input_leds serio_raw mac _hid ip6t_REJECT nf_reject_ipv6 xt_hl ip6_tables ip6t_rt ipt_REJECT nf_reject_ipv4 xt_LOG nf_log_syslog nft_limit xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nft_counter sch_fq_codel nf_tables nfnetlink msr drm ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_net aesni_intel net_failover crypto_simd cryptd psmouse virtio_blk failover [74237.858360] CPU: 2 PID: 5452 Comm: sshd Not tainted 5.13.0-21-generic #21-Ubuntu [74237.858363] Hardware name: BHYVE, BIOS 1.00 03/14/2014 [74237.858365] RIP: 0010:smp_call_function_many_cond+0x11a/0x2c0 [74237.858377] Code: 74 24 08 e8 38 48 49 00 3b 05 76 45 00 02 89 c7 73 22 48 63 c7 49 8b 0c 24 48 03 0c c5 00 59 eb 85 8b 41 08 a8 01 74 0a f3 90 <8b> 51 08 83 e2 01 75 f6 eb ca 48 83 c4 40 5b 41 5c 41 5d 41 5e 41 [74237.858379] RSP: 0018:ffffb92f80c9fab8 EFLAGS: 00000202 [74237.858381] RAX: 0000000000000011 RBX: 0000000000000001 RCX: ffff9a8f37c341a0 [74237.858383] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000000000 [74237.858384] RBP: ffffb92f80c9fb20 R08: 0000000000000000 R09: 0000000000000000 [74237.858385] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9a8f37d2de80 [74237.858386] R13: 0000000000000246 R14: 0000000000000000 R15: ffff9a8f37d2de80 [74237.858388] FS: 00007f7a70523900(0000) GS:ffff9a8f37d00000(0000) knlGS:0000000000000000 [74237.858389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [74237.858390] CR2: 00007f7a70aa64a6 CR3: 000000010b8d2001 CR4: 00000000000706e0 [74237.858393] Call Trace: [74237.858399] ? invalidate_user_asid+0x30/0x30 [74237.858409] on_each_cpu_cond_mask+0x1d/0x20 [74237.858412] flush_tlb_kernel_range+0x41/0xa0 [74237.858414] __purge_vmap_area_lazy+0xbd/0x6f0 [74237.858420] ? do_jit+0xe3a/0x2390 [74237.858424] ? purge_fragmented_blocks+0xc3/0x1c0 [74237.858427] _vm_unmap_aliases.part.0+0x114/0x150 [74237.858430] vm_unmap_aliases+0x27/0x30 [74237.858432] change_page_attr_set_clr+0xb7/0x1b0 [74237.858436] set_memory_ro+0x29/0x30 [74237.858439] bpf_int_jit_compile+0x353/0x3d0 [74237.858442] bpf_prog_select_runtime+0xf7/0x130 [74237.858447] bpf_prepare_filter+0x1cc/0x200 [74237.858456] ? hardlockup_detector_perf_cleanup+0xa0/0xa0 [74237.858461] bpf_prog_create_from_user+0xc4/0x120 [74237.858464] seccomp_set_mode_filter+0x122/0x530 [74237.858467] do_seccomp+0x37/0x1f0 [74237.858469] prctl_set_seccomp+0x2c/0x40 [74237.858471] __do_sys_prctl+0x438/0x6f0 [74237.858476] __x64_sys_prctl+0x21/0x30 [74237.858478] do_syscall_64+0x61/0xb0 [74237.858484] entry_SYSCALL_64_after_hwframe+0x44/0xae [74237.858490] RIP: 0033:0x7f7a70a0d281 [74237.858495] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 08 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 9d 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 17 48 8b 4c 24 18 64 48 2b 0c 25 28 00 00 00 [74237.858497] RSP: 002b:00007fff28e84180 EFLAGS: 00000246 ORIG_RAX: 000000000000009d [74237.858499] RAX: ffffffffffffffda RBX: 000055c0301fe020 RCX: 00007f7a70a0d281 [...] thanks, --=20 J. --I7ODD5mjxYr1vVEs Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAmGVpWgACgkQs8o7QhFz NAX/Fw//dJg2JEga0Q9/YF0w2YutL0Rp+4EPTHFfJOYxKtkLXaCHNl+XLIUBXajv n1/xlDDs5dgt+iJtPWHnc0bAK1hEXgVKg6S/PmOK44aYEs14cGI7cGHw1kv38v6B awkUnEfy+nX8hZPQBqNgaFMLaKnu46vhB1tfjtqKiPQr7bF7AoP/5vRiGAOYGe1z IXeRvhS3UjmPF3C93ltDa6yK8f/QPqTO45pz08PP8NhsqYBPaVUCbrmMIWniz4Ww LyJJPGs/dTWL2YULk7ZdS5oiEvRpUIfwB7uVuGn4h/14a2tEP4Iv8qYqIdhz3POI 0sCuW3068kXR9Jd/EwyX6odLEILcN42cnat3fvpKi9zNrTAynWIdwEaAV3STWhp4 CK9DADwcg4ShALHPLRLtHISKOsLJJXUdPHJmSd+ltk8vVr1LTVim0cjnWnkAxCF+ tQhxvo5ZjoQ2BAr3MwZBtAHMEgShLcuvBlvlqdgCg9ez7z79gcQQXWW1m4RnTVF6 3rqQmibbUuBJIjNY7gl4roxKIXCxyWySWZFtpr0dYEuHQQbtCEh1mjUWZt1TM9Ni zS4pXN0MxPajUvHKE5m8KVtaZukCYgBRjAogtvX41D/Aoz4r79soUWM9VcuTvwbh gVAF5cDyTVAT+fz+FfWS8dzMqYIIQGyy7K91N9aagXJJ92TFDP0= =9W4S -----END PGP SIGNATURE----- --I7ODD5mjxYr1vVEs--