From nobody Tue Oct 18 13:32:13 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MsFCy2BMkz4fYqq for ; Tue, 18 Oct 2022 13:32:46 +0000 (UTC) (envelope-from prvs=283b39a2d=roger.pau@citrix.com) Received: from esa3.hc3370-68.iphmx.com (esa3.hc3370-68.iphmx.com [216.71.145.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.hc3370-68.iphmx.com", Issuer "HydrantID Server CA O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MsFCx0fk7z3Cxs for ; Tue, 18 Oct 2022 13:32:44 +0000 (UTC) (envelope-from prvs=283b39a2d=roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1666099965; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=mPQVn3MbUcON7xDa8JX2YzBqLLO0FGBh6l+Lubv8i8E=; b=eVMQhbUhWxdQUmvaPvfMf4yXU4TEFuMDG22Bx9uslKMlq4ozQVqaLLml +s7ZWUIULjM78v1xbNot0UcRuGzYZf0D6YaoVj2TsFXmU9dk+5IT374xc i/Hiim4mLsXJrGTTIvmG5VPIEK3JxCnEQC6yaivx8zdGLNXl/lfjtF6u6 0=; X-IronPort-RemoteIP: 104.47.55.102 X-IronPort-MID: 82972687 X-IronPort-Reputation: None X-IronPort-Listener: OutboundMail X-IronPort-SenderGroup: RELAY_O365 X-IronPort-MailFlowPolicy: $RELAYED IronPort-Data: A9a23:mCLT7q5hTRuj4Vz7zWRM5QxRtC3GchMFZxGqfqrLsTDasY5as4F+v mceD2iOM/aMZTGhKI1wO9nj9BgB75HSmN5iTgZvpSoyHi5G8cbLO4+Ufxz6V8+wwm8vb2o8t plDNYOQRCwQZiWBzvt4GuG59RGQ7YnRGvymTras1hlZHWdMUD0mhQ9oh9k3i4tphcnRKw6Ws Jb5rta31GWNglaYCUpJrfPewP9TlK6q4mlB5gVuPakjUGL2zBH5MrpOfcldEFOgKmVkNrbSb /rOyri/4lTY838FYj9yuu+mGqGiaue60Tmm0hK6aYD76vRxjnVaPpIAHOgdcS9qZwChxLid/ jnvWauYEm/FNoWU8AgUvoIx/ytWZcWq85efSZSzXFD6I+QrvBIAzt03ZHzaM7H09c5oCGdg0 dA+MgtKaxWS3uer5bOkQbdj05FLwMnDZOvzu1lG5BSAVbMDfsqGRK/Ho9hFwD03m8ZCW+7EY NYUYiZuaxKGZABTPlAQC9Q1m+LAanvXKmUE7g7K4/dqpTGLlmSd05C0WDbRUsaNSshP2F6Ru 0rN/njjAwFcP9uaodaA2iL217CVw3OkMG4UPIae3/FL0UGK+ndNVAJHCQW6hNCWp0HrDrqzL GRRoELCt5Ma91SqQ936RQ+PpGWYlgIQS5xbFKsn62mlxLDZ7gyCCi0UQzhpZsYrucUtSXotz FDhoj/yLTlmsbnQTG3H8L6R9Gu2IXJMdTVEYjIYRwwY5dWluJs0kh/EUtdkFuiyk8HxHjbzh TuNqUDSmokusCLC7I3jlXivvt5mjsKhotIdjukPYl+Y0w== IronPort-HdrOrdr: A9a23:yR4mDKCSZhqAHzXlHeg3sceALOsnbusQ8zAXPh9KJCC9I/bzqy nxpp8mPH/P5wr5lktQ/OxoHJPwOU80kqQFmrX5XI3SJTUO3VHFEGgM1+vfKlHbak7DH6tmpN 1dmstFeaLN5DpB/KHHCWCDer5PoeVvsprY49s2p00dMT2CAJsQizuRZDzrcHGfE2J9dOcE/d enl7x6jgvlXU5SQtWwB3EDUeSGj9rXlKj+aRpDIxI88gGBgR6h9ba/SnGjr18jegIK5Y1n3X nOkgT/6Knmm/anyiXE32uWy5hNgtPuxvZKGcTJoMkILTfHjBquee1aKvS/lQFwhNvqxEchkd HKrRtlF8Nv60nJdmXwmhfp0xmI6kda11bSjXujxVfzq83wQzw3T+Bbg5hCTxff4008+Plhza NixQuixtZqJCKFuB64y8nDVhlsmEbxi2Eli/Qvg3tWVpZbQKNNrLYY4FheHP47bW/HAbgcYa dT5fznlbdrmQvwVQGYgoAv+q3nYp0LJGbIfqBY0fblkAS/nxhCvjklLYIk7zU9HakGOul5Dt T/Q9pVfY51P74rhIJGdZM8qJiMexvwaCOJFl6uCnLaM4xCE07xivfMkcYIDaeRCdc18Kc= X-IronPort-AV: E=Sophos;i="5.95,193,1661832000"; d="scan'208";a="82972687" Received: from mail-mw2nam10lp2102.outbound.protection.outlook.com (HELO NAM10-MW2-obe.outbound.protection.outlook.com) ([104.47.55.102]) by ob1.hc3370-68.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 18 Oct 2022 09:32:21 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lu8hwq4GS4qSczUXdcF3w9457Bpf0kDF0uSXHaBcmEaT4YoC4HbgSZ3u9bVyIqbHhE+YXzD0okPXYSbgIBY9sLi2TfCgA31MsPfu3X24yZxSOzLU61mYji5ibKrt81ifOUvP2p5afScsI/JRa8adzRSApotIMB1l7ex42TSDJf5vQky0y6l6V5nfaehlhTtQ+NMXV7+6U0Ry8qe4Y5tmNs54E9RX280l3a45kyexBlJn2Ga3IdZUdUOR9l9fKHl/le97v7Nw+l+9NVuK9TJ2F0iHU5ShcYp4dRtw3qJGsXo9MfVwKXprL+fsO8Um/wJSmC94KZnB1MZ2Kn4LTs1mUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UdXjnJNimgO0sU3zE/dr8FZuUaX2VmJdtfLKrrIIVAM=; b=ReegZv8ddWo6mMd+SeAz3IWslpwRn3BowAboYB8ozsp6Gl/NFwv3EWX+75OYWIn8mHi7bhVpnixsQtXXf3bAO6+TvT48wqEo0wqxvZo4HC8bM/+mQtmQenUlMk2sesMNmhTheJjuYhopYQOoWm8gUkQjziF0cD4fjjvpFwRyfFmkNV8KdYHNMhObUp6AiB3sN8BJiQ7pLLtD+sGVGoSv9aOg1WbxzHwMsbp4/Z0tx1gfNr3bGEpIWCbt2WCZr6yewAaN2JJZt3MtyKm+iQLDhAFAR1BsBWE6FFiMhLYk7j20RmZTowWCuzgnuk1Ltlm9ospPJnj8p7qUHTDQHIhRog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=citrix.com; dmarc=pass action=none header.from=citrix.com; dkim=pass header.d=citrix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.onmicrosoft.com; s=selector2-citrix-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UdXjnJNimgO0sU3zE/dr8FZuUaX2VmJdtfLKrrIIVAM=; b=H3QlnHQoaZuARFUa4LVVZypoflqbvQMc/LfZYTdd64CzDWYvYJxl0QWLBvr2OWJp3Wz5RemRVhVLonxN7rW+pRnWBNCX4Ln9PBPdOQs4Dv+rswjugL7n16ZC6cJCTM2c9oXZ1kgcUDAGYlogz8Jkvl1G6qrHUNK6fDQ5vLdP6yU= Received: from PH0PR03MB6368.namprd03.prod.outlook.com (2603:10b6:510:aa::21) by MN2PR03MB5103.namprd03.prod.outlook.com (2603:10b6:208:1aa::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5723.34; Tue, 18 Oct 2022 13:32:19 +0000 Received: from PH0PR03MB6368.namprd03.prod.outlook.com ([fe80::3b75:f07d:1914:3e7c]) by PH0PR03MB6368.namprd03.prod.outlook.com ([fe80::3b75:f07d:1914:3e7c%7]) with mapi id 15.20.5723.033; Tue, 18 Oct 2022 13:32:19 +0000 Date: Tue, 18 Oct 2022 15:32:13 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Karl Pielorz Cc: freebsd-xen@freebsd.org Subject: Re: Recently moved to XS 8.2 on new hardware - seen a couple of FreeBSD DomU lock ups? Message-ID: References: <3C47F89ECDBA642ACCC7B932@[10.12.30.106]> <91CF514F28FADFC45E157B96@[10.12.30.106]> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <91CF514F28FADFC45E157B96@[10.12.30.106]> X-ClientProxiedBy: LO4P123CA0447.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::20) To PH0PR03MB6368.namprd03.prod.outlook.com (2603:10b6:510:aa::21) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR03MB6368:EE_|MN2PR03MB5103:EE_ X-MS-Office365-Filtering-Correlation-Id: 6fdd2259-c611-4e89-eefa-08dab10d2dd9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g4ztZB25UfAl0mRO1QmW/xn2Q9H25dAtr9qpBgwCFIIDR7pLZbuCkIT0X6LlSBmHHz20k0fGmigK90BTBZqmqNfPHZZGT0B5hva4pvgSknoPkC///WIUO4fsKqlpsxYb/NMWM02Qgpy8YaKHk7z+X0NptJu9f20GvEtRIytxl3sVXC9CDn0gjiib20hARrPc/JosZK2dX7kRpmIRin/X31doqf+wIf68vCsBMCDm11XO2tQtF2rD8KoMWxg7g3w7eCxisWEMDB8ZUI+uRe3NUnELigax4H7cEIRmPOJ0LU2jsI9Dkti5j5y6X2wDBcRetqEjPzCLAh/SJ6BLvBsZsOneLNxkqnUDAtsw3WJ4LY0JHDcYb1bwuSVD6uftF32e2Dv2wX/hmTks3KmLynxNPliGWLdMska5lMBQ0bC+unVsRYXD8KmCbrgpHaSyoGT7H6IK9vLChiVdx19YWzEOllSKRvwSLybSTKm46lHH1PAym9dQSoqOdiFycJNeP0rLnoho5RdFJYmF8K5Xb7/jx8Osa6jxEqjTVVcg6H4SUFcOBCCq9lgBge7yMFiPfpf0hrdkie+wOMr+rdGNGvWmq4fo8arcA107K/110qPYkFeKGSamtDRhcPLZ2DKN6S0/0AxQoaYUthYPTc/nqTVzR1Z+YHa0hePCVcMGIPd9aUGZTe1dth6UINKSAQjOPcoLp8whS/0n5w13Dhv70zLGo2WD1UdgviSDFBofPZKhZIM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR03MB6368.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(7916004)(136003)(39860400002)(366004)(376002)(346002)(396003)(451199015)(8936002)(5660300002)(2906002)(8676002)(4326008)(66476007)(66556008)(38100700002)(41300700001)(85182001)(86362001)(82960400001)(6512007)(9686003)(26005)(6506007)(6916009)(186003)(6486002)(966005)(6666004)(478600001)(66946007)(33716001)(316002)(83380400001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Sy9CcllVMGZhNnhISU84TlZyNlNzWmc4bG9iR29ZZlk3RVo2Zjhoc3g0T0po?= =?utf-8?B?T3k4MmpENnBNN0VWN29LRjZPY2M1KzRCQldSZW9SSldKRjRFaXJBcjVxOWRz?= =?utf-8?B?NkNhN0NCVitpTEk0YlV5S1VPZlNFNDcydE42Y1FGejlURVZFNnIxWHJCTXR5?= =?utf-8?B?OXgzQmtRSnBiL09mbkd1d29TdTRnR2QwQ1dQSldjd1B1RGhmTVZKOGNMRlIv?= =?utf-8?B?WXFDMnhkMVdXc3loS0RzNUhFSnNzbDZhZThjdGNZRlE5UHpSRG5kbHRpQVJP?= =?utf-8?B?M2kzSnRLQ3Z2T0tVMk1OK2VNSE15RzhqQUlyOHhmNjlubmJGRThWKzB0NG4y?= =?utf-8?B?SlJQU3I1T0RXam92MFFCV2RHRkpCSlhMbTN0bXk1LzJlbSt1UXlIVmRqMGk1?= =?utf-8?B?MkExaXc4Tk11SUFiTlh2VjhoK2NPTlU5VlpWcW5KaDVhZ0Q2cUxCOVZBUXQ3?= =?utf-8?B?ME41Q2FDeUdkc200TXVHUHJoVHhsOEV5aFBjK050YllzQ0NwZEJxRXlTV01w?= =?utf-8?B?M3dqUU9oaHJTaEZtNnJyMUNsQ29NUUJrVnNKSWxZaE5RbkhBUFYyWmJBNThh?= =?utf-8?B?ZWg5QlVxUDc2ZHM1Q2V3SHR2bGRvajlYSU9GY2UremFnY2VpdHRqQ3djb3VB?= =?utf-8?B?NmU2UWhHV0FKdnpDZUJ2TnozNVdIVVJFVGNvR3dUa1Iwcjc5K2RrcDJLK3Jo?= =?utf-8?B?aXdOTUgyc2JKRVBaM2JqSVhSd24rSkRDd2ZEa21vM0lpWUpKTDBzZHNjR2Q5?= =?utf-8?B?cFE5WXRaT0JhOUVIR1pxWkFYZ1BsSW93eGl2am01OEljdXgvSU9odjFUNkpm?= =?utf-8?B?elQ1aXdobFZXZWpGOUdnZ3lqdWg5K3VKbFdGT3BBdzY5dHZqVDdicDVkNkY5?= =?utf-8?B?SkF1LzdUdHlycmM0UDdWbFI2dG5ia2FQYWVxVWpTMzNFa1ZqV0ZRMzJZSWp1?= =?utf-8?B?WUtvdittTEtMZ081MGxTTHFjMEI1THp5ZXlmZ29QeEgvUmI2SnFITnVZNDcy?= =?utf-8?B?NFBnSmVPSEkwaUdha2p6NW5IKzRRSGFocmx6a1dGdmF3SnRuSEpTZTNFUzI4?= =?utf-8?B?NEZFaXJER0xMVHJsTUNFZUlsTEtnbTJINElDU2xIaVFIWFNxUFI5KzhIZG5L?= =?utf-8?B?OThEMXJGOTBYWVY4QnNkMW10cmRzTTlQUWNlZGZhRVZlT1J6UDVMMWkvb1Va?= =?utf-8?B?VjduRUtWd2N3ZG1lT0NIUmsvWDhTMkdZUDBVUTE3U3diK2NLd0tMRkQxMzBr?= =?utf-8?B?MU5aVitacDBJdHZPc3ZUSTBLNndvNDk4dnphNk44L29OeW9IczFPRUNFQW9C?= =?utf-8?B?eWJ1MGFEU29QZ29HaFlleVFOcUo0ZVA0Njg3UlN5d1ZONzQvQURKL1ZjcEY3?= =?utf-8?B?UlU4djMrNmozUExKWHdBeTlJVWFER1ArdzZLQU9SRmdqOFFSQy9vZ2tNbUxI?= =?utf-8?B?cHQ2K3FCaVlUUkdySHVoRXdla24wT0t0bmlCbFBnWjlyZzNTcU5EbnhhaG1X?= =?utf-8?B?ZDRsbk9jOUN1ZElOY3A3RmF6OEU4TVNUSzFmV1FQbGFUNHdUUlBtSWdGVEg4?= =?utf-8?B?Ym9wb2FXc1FTZWI3M0c1RXBoTS81VkJ4b2gyenh0TXdQUWRDMFdZcFFDM01v?= =?utf-8?B?eE9MMmNuQjdZQWtkVFJlaUhXUHk1KzYxSmdKQTR4VHRCbkgzRWZCK2ZSekU1?= =?utf-8?B?MldhcWdFWlNsM1pCdHZETHRlbUlSZGRESEEvNFVxdm1HWWhlb3d0YUFkT2Np?= =?utf-8?B?ako3Q2hHUWhHKytQaVlTRXl6RTZWMWloenlXNUVCMSt0TDEvRHE4R1FoOEJj?= =?utf-8?B?QWxyYURSMXNzaFpYNElJQ0xZSXpwRDVRV2NoWjdsTzB1TnU4bHk2TUtYYVd4?= =?utf-8?B?VlZDcGU5cjFUU21tY0tzTUFvTmtPckwzZVMvVFVjZi9rNEl3SFIrbGFpZFFj?= =?utf-8?B?V3RqRzQ5eDgyWUQyUm81SU5PV0VuS3loZmFDUk9HYzhMd0ZPVlZRcDhTdHpy?= =?utf-8?B?bFdOZCtPQkk2MUpndlJ0TCtaZnczbHhoNThHbEsvdkY3SnZFbnlwK29ORmRR?= =?utf-8?B?emVNMit4TGJrTC9XOHpZM3lGVGprZmJrSmVXdUptUGRsQUVqOW9OcUdXb1Y0?= =?utf-8?Q?C2Y+FU8ZJiTOvvAumI6TqlLlu?= X-OriginatorOrg: citrix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6fdd2259-c611-4e89-eefa-08dab10d2dd9 X-MS-Exchange-CrossTenant-AuthSource: PH0PR03MB6368.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2022 13:32:19.1263 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 335836de-42ef-43a2-b145-348c2ee9ca5b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hwLADfaJSh9xgfUo3gH/Z2LTAS9dnqDDTjOKhCRKQnsaqdGwfM/ijmVemY7F+FatxuzwKQ9LqyBUzIp4mj90SA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR03MB5103 X-Rspamd-Queue-Id: 4MsFCx0fk7z3Cxs X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=securemail header.b=eVMQhbUh; dkim=pass header.d=citrix.onmicrosoft.com header.s=selector2-citrix-onmicrosoft-com header.b=H3QlnHQo; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of "prvs=283b39a2d=roger.pau@citrix.com" designates 216.71.145.155 as permitted sender) smtp.mailfrom="prvs=283b39a2d=roger.pau@citrix.com" X-Spamd-Result: default: False [-5.16 / 15.00]; DWL_DNSWL_LOW(-1.00)[citrix.com:dkim]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.96)[-0.958]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[citrix.com:s=securemail,citrix.onmicrosoft.com:s=selector2-citrix-onmicrosoft-com]; RCVD_IN_DNSWL_MED(-0.20)[216.71.145.155:from]; R_SPF_ALLOW(-0.20)[+exists:216.71.145.155.spf.hc3370-68.iphmx.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FORGED_SENDER_VERP_SRS(0.00)[]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=283b39a2d=roger.pau@citrix.com]; ASN(0.00)[asn:16417, ipnet:216.71.145.0/24, country:US]; RCPT_COUNT_TWO(0.00)[2]; RCVD_COUNT_THREE(0.00)[4]; MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org]; DKIM_TRACE(0.00)[citrix.com:+,citrix.onmicrosoft.com:+]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Tue, Oct 18, 2022 at 12:15:28PM +0100, Karl Pielorz wrote: > > > --On 14 October 2022 16:40 +0200 Roger Pau Monné > wrote: > > > Hello, > > > > Sorry, been very busy this week and forgot to reply earlier. > > > > Could you try to setup a watchdog in FreeBSD and see if that > > triggers? So that we can get an idea of where the guest locks up. > > Hi - no problem / thanks for the reply... > > I'll give the above ago - part of the problem is not knowing which VM is > going to die (there are quite a few) - the second part, is the waiting > game... > > > Is also the 100% load on all CPUs, or just one? > > > From memory of the graphs - I think it was probably just one (I think > > the > last VM that locked was a two core VM). > > > If the watchdog doesn't work we can try other methods. > > Well, I'll get back to you when it happens again (personally - I hope it > doesn't happen again) - but at least I know it's not quite as much of a dead > end as I feared debug wise. > > If this does happen again - and I'm able, is there any point in doing a > snapshot + memory of the VM? (which is about the only thing I could think of > - not knowing about the watchdog stuff). You could try to get a snapshot, albeit I'm not sure if that will work correctly if the VM is wedged. Since it might not be feasible to setup the watchdog on all VMs my recommendation would be to try sending an NMI to the stuck processor, and see if we can get a trace that way, see: https://lists.freebsd.org/archives/freebsd-xen/2022-October/000125.html That seems to get me a trace when used on a non-locked up VM, so it's worth a try. Thanks, Roger.