From nobody Sat Mar 26 04:03:46 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4D8F11A4279F for ; Sat, 26 Mar 2022 04:03:55 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-YT3-obe.outbound.protection.outlook.com (mail-yt3can01on2042.outbound.protection.outlook.com [40.107.115.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KQQLd519Wz4pHn; Sat, 26 Mar 2022 04:03:53 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YdHYRscEt/GfTAxJUFSl9eI9jY4CjCUbc6OPP66tMzfe8e95Zt5yNVbm1ykOUpF4k+6m3dYnkOMzIINa9tnL59xWn+M3dTpk9lkAnjUMpStKtGqkY5eDVZgW+Sr33N2R0ZCnjbPR+Z8aJxuezAi0h9Kvcm9bg1dwHpy/H6ZcTV4uM5pXP3z2nwjQt1ZY9GkvgOM+1ASilnvCuppYnGszMUOVgE7Hi6xwmqDWoHuag4DSH1pft0wtua4LC6Yev3FNliq0eGiFQxEWw3GuJDXSh/qu4haVX9PXBegKjK7b1VKNqr8e7M/xDnnishRP6g/p/wPe1VbJdmvaC0/sRqomIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hYnXzY1QIOwEXK7iDCMH/LRF2NcXBu1jPF4ffPexmx4=; b=YiTrATKRRsr5ztRzU+oVCTwBCiXrgciz67wCK7MkHYifKgGo8kzq30o83wZju8pjol/3JCYhv9M+ig+tcaDNiteXlDLq0TzLUuix9vZBKODEvzY1RFTUlVv/1wHgHxzxVL2iu0+1NLHzjIZeJ/s5V2vqfqk+Vd6YOD8fEasi76KbnKeaHl5qi0lskUyX53ZM26Z6zHLUNqJj6sbwM6G3Gwcmd2fHbBDbiQsiF7n81PL9hS3RbgbSc13nsYAfxjYIqQi9QYXW5ueQPk2Y1D8bmlkBoFhTvLEo6m8d9qMg7RhoWt0kW42JIUFdjkKrOiJQB9SwiD7b/IM1G7O6gC8WVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hYnXzY1QIOwEXK7iDCMH/LRF2NcXBu1jPF4ffPexmx4=; b=PSHKTRQlsAm1DEbAVK+dqY01j3FJ5I7L/pXPBDoiZZx2Ts6DcEXx2OOa9vy3WzscxGhY5o4I+RPsO3fneL0I+ariwcVRZNC96kl7Vn676uraUerwpLz8iwNF5kw0MzKxCXgC7F/acz8q0uGiRmwaJfTgv+54PWX2cyEuK7SK6ieoB4OAe+CjxUNa+8gCpMzXIWHiofbq/+0C/lWeiSqjSLLSKF0Buhx18Mm2PfRhfXrSS2ECslXuiRiVXEPwMqdeMqZvn4WqRjLZSa7UiTrD5SMvc0OaYxjhC0+/KmcD0LUhoiYsdbDsYJRvlEBDUyLjdm8rTpG/1AM8V8j+68Jjjw== Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:de::14) by YTBPR01MB3023.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:1c::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5102.16; Sat, 26 Mar 2022 04:03:47 +0000 Received: from YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::39d7:e98e:fd00:2591]) by YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM ([fe80::39d7:e98e:fd00:2591%6]) with mapi id 15.20.5102.021; Sat, 26 Mar 2022 04:03:46 +0000 From: Rick Macklem To: John Baldwin , "freebsd-current@freebsd.org" Subject: Re: ktrace on NFSroot failing? Thread-Topic: ktrace on NFSroot failing? Thread-Index: AQHYQI/JkEo0PYUUsU+bkJKIOYzP06zQ6ftQgAAf8BE= Date: Sat, 26 Mar 2022 04:03:46 +0000 Message-ID: References: <70135754-84c2-b642-5641-0721e0b4690d@FreeBSD.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: suggested_attachment_session_id: d01c8e14-9a1a-4986-187f-da997a3c125c x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 23442d84-59cc-437a-347c-08da0edda07c x-ms-traffictypediagnostic: YTBPR01MB3023:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: sxNjUImi2Axy4EDoLV0Kb31E8IW8G28xcmvYXsOlhy4cE2+6vFyibtTs3pe1M6B+0oFS6580Rk0viqoOrfPN6oHIbfK4tSg3BZMbJ+kQVt6sHkJl/YL5eXYXMSOZq7IHvPF4unYHxsSISFfdCJh2ttNF6x6+lMxaCP2PazjqKDyuQDwddkxV/touVYWTc9fUbH+pOeptM9qloT75h3P4OhcpCQu1Vdgtu2OGxmtIs3Z9bi6TJwHoV/nSyBjrj1uCPzPqJ+aEmLE253JkGaTrsEeJMjSHuS+6DaxlyS/AZm4vaFfHPz78RwpcVG6Dk+cuoAldy+U63C42RFnuA6XvoZc38i4XwStRGLWCGBr3PtMntMHdEFNnUHzVuXZNq/nE/lqc5DP73rVXiRWlD88XxE1aToVFGKP1owgEIV8Rf9Q7AaF2qB6pQh05QDza5EyOtmY8G2Ju2q5o2SLgMOlw8pUGYbWtNlrai35vwx8NLltoP201MCN3w+xZgqOn0jfH/4ulQ9ROYJtoQmyATG2dny8ZTtFGoFz3rYIovBxAqya3Q22rfE7ygXR4hBcrvD0j6AQcx4q1bzLKrvM4kzizYGXszIVKGYTdVZIfaJ/NKUe7IRlFgcPwBXImdu59fW/VLMz0VCqr/2N/TjzLA1zWgiglizgtA5NJ0MX0C+Ph2nJf6vPsSAZPc5ZNVi1IfO9OVoZdy6Fx3kHsBjyRrohXqQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(366004)(91956017)(508600001)(450100002)(83380400001)(76116006)(5660300002)(53546011)(110136005)(186003)(52536014)(99936003)(66556008)(122000001)(8676002)(64756008)(66946007)(66476007)(66446008)(8936002)(55016003)(3480700007)(38100700002)(33656002)(86362001)(2906002)(9686003)(71200400001)(316002)(6506007)(7696005)(786003)(38070700005)(2940100002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 2 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?N7t20DfADCVdIxgp54BOB7twF7Ipx23QF9itUrgLgJnhvQKjNoShWkFny9?= =?iso-8859-1?Q?0FosheHttLx4V611mZEUFzvDUS3MY+6QXCjauhghuuidnA1oKJuw8pzXQH?= =?iso-8859-1?Q?/z/gxH0H44H7RlRCNHlfNJN5aNPaLDJFHHJYhPSuT/qe8R65Jt4+ZSsA1z?= =?iso-8859-1?Q?yEE6JeV3v1EL4Jv3ZXrChtg4dljp1dmB+Cw/HCt5cgEb4GojEPn/1SKS0B?= =?iso-8859-1?Q?1rVfysqQBxrWzxFc8wGj4CBOU+x3IoMhiuXEmdNRgLO+cwciyDbgxZjzmp?= =?iso-8859-1?Q?RM/5Jgq/kG76SbW59mW3hP2l9XPEGBhwmm68VsQPJG6htmuLM3oeYT5nt5?= =?iso-8859-1?Q?StYouZwkfzkKSPego/aT01mO72/zdCvODIfgl0CDVrOJ1zNQDa8qGg9zuj?= =?iso-8859-1?Q?ArYW3ri+IJow8tNo+yZ2uu/n1antBbdnyhYGuGCkb4eVkyxo3t3CfxCN18?= =?iso-8859-1?Q?4aDaPOEETjQLA6ErbmZAyoUeZDYbFUrmRYlx4etPZLIkcDJ4QIalcOA4ue?= =?iso-8859-1?Q?yTtV2m5OG0bf8RTuFn6bpIq73WfRS7v71uTAgFmJV+yZWDRXM15irgtFEq?= =?iso-8859-1?Q?RtWj9YqXYPewoaBvByZsh3eAa7mT/5N3zziuLawrC7m6h0miU9ZJYbsH9G?= =?iso-8859-1?Q?XdhO/vYE2z4LU86sh7Mk2GyoQnnFzudoXynluGf6de/tFAD/BHmcSGopLa?= =?iso-8859-1?Q?iNis1H5gF0Od03dPw58GMdN/RkNEcSfsuoL6PdfU0gpzzsGyyKFfgVXkI1?= =?iso-8859-1?Q?F934hiNBU8Y70qubjDc6J53a0PRy/qaXdaoVoh8Qii513AjdK0yre0cwIq?= =?iso-8859-1?Q?54iO+7DsqodDbbEBp8BKFGvcvLw6CsXIYiO1WM/e8hi4CzC56RS26BGbr0?= =?iso-8859-1?Q?FZngyyBPvAyB1NswlmQ3ab863Cp33RpL/ihLUtcsw+UNbBoISMOS4LS9gl?= =?iso-8859-1?Q?Wdawl1ZVwjcMLgGUIIpA1OZk/mbddL8LCicEK128S/+zULzyvHvxiEGSbH?= =?iso-8859-1?Q?Xv5r3waQdEuOfGDsuUYHdC4bFddin18+nksbKaCQQccpFdxRuAw1iX4l1/?= =?iso-8859-1?Q?kM21DunotfWS3RkxlLaQCIHcLw8YPMzxK65P3tN9XCcgLV0yWYRj74NxAj?= =?iso-8859-1?Q?wpQ2bUfyt/9UeBNaJ2Gf2WXg7Sj/4yKclDqy6dtbnyWgJaYDZWqd7a1hn1?= =?iso-8859-1?Q?b9jd2RBWEqkLEgg6F1NgD2LZo0r4p2xwFTqeWEEGe+RdOFZq6pZH22dwGy?= =?iso-8859-1?Q?c0yeeH74I11id8gCB88smd1Q9GQwy3UxUq2a6tBU8vHjsO0OVMNkvOBy2J?= =?iso-8859-1?Q?uCKsShugeRg61RpbKUWJV2nDi2zrtJ2P6Be9aMwv44Ufki3cZsh/TM+7YU?= =?iso-8859-1?Q?JZ2z4zadHyBA08W5GGGHFYA/erOVE4yy/k92ZT6UvntGARMDjhqJokxHjt?= =?iso-8859-1?Q?O/aUCaBnw0v1+dN8GfVluRhNXZzI6Jag6eW0XE3NWe2HV3txwbuRLfy5VP?= =?iso-8859-1?Q?pD7aK1SoqhPeRUQZCGTy8e2RyJdt6dNlsi+YRs4dV/IEooXEUjj2Dov5Mo?= =?iso-8859-1?Q?eETB3PIV7/gOUUt+hDKK3ryaiVy7S1o+sl8avTGOAkGAdDJ/Df0gVeyQ9s?= =?iso-8859-1?Q?Y6WlKPLVDrmnCOH0Ro0ioJbrfqhZCdv29fgSAGyxv7Sqr6MV6G8LMQDDUi?= =?iso-8859-1?Q?j4LCM36DTqPAk+5n1zpXTZ1TVEvgFz7bpVlEfqn5aPC+27NVrVmTy0Wr57?= =?iso-8859-1?Q?I5AdmWyAXJcY/HXN/ntK8+AcRxzfuBPxc27Qx63fiN24SS4mfirRZdiGja?= =?iso-8859-1?Q?64N3uYfVPo/O/7qYIc4ZzQXOiyBZMGlv2KYFHD1GqSg/N+s3iBr9nwvG3c?= =?iso-8859-1?Q?Jt?= x-ms-exchange-antispam-messagedata-1: zSefCdEeVmnZzA== Content-Type: multipart/mixed; boundary="_002_YT2PR01MB973039BE3072ED3F94CFE141DD1B9YT2PR01MB9730CANP_" List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YT2PR01MB9730.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 23442d84-59cc-437a-347c-08da0edda07c X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Mar 2022 04:03:46.6110 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: YumUgNNLVr/UYNk/p9wjzucclgCw/ccY+n8F6mBtDVd7x9/kztp+D68CpsvBtpvfDUiCpIdnglDgyOzMvl3aHQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTBPR01MB3023 X-Rspamd-Queue-Id: 4KQQLd519Wz4pHn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector2 header.b=PSHKTRQl; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.115.42 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.98 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector2]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; HAS_ATTACHMENT(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; NEURAL_HAM_SHORT(-0.98)[-0.980]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.115.42:from]; MLMMJ_DEST(0.00)[freebsd-current]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; RCVD_TLS_LAST(0.00)[]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1] X-ThisMailContainsUnwantedMimeParts: N --_002_YT2PR01MB973039BE3072ED3F94CFE141DD1B9YT2PR01MB9730CANP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Rick Macklem wrote:=0A= > John Baldwin wrote:=0A= > > On 3/10/22 8:14 AM, Mateusz Guzik wrote:=0A= > > > On 3/10/22, Bjoern A. Zeeb wrote:=0A= > > >> Hi,=0A= > > >>=0A= > > >> I am having a weird issue with ktrace on an nfsroot machine:=0A= > > >>=0A= > > >> root:/tmp # ktrace sleep 1=0A= > > >> root:/tmp # kdump=0A= > > >> -559038242 Events dropped.=0A= > > >> kdump: bogus length 0xdeadc0de=0A= > > >>=0A= > > >> Anyone seen something like this before?=0A= > > >>=0A= > > >=0A= > > > I just did a quick check and it definitely fails on nfs mounts:=0A= > > > # ktrace pwd=0A= > > > /root/mjg=0A= > > > # kdump=0A= > > > -559038242 Events dropped.=0A= > > > kdump: bogus length 0xdeadc0de=0A= > > >=0A= > > > I don't have time to look into it this week though.=0A= > >=0A= > > Possibly related: core dumps are no longer working for me on NFS=0A= > > mounts. I get a 0 byte foo.core instead of a valid core dump.=0A= > I just tried a core dump for a kernel built from main sources as of=0A= > to-day and it worked ok.=0A= > =0A= > However my userland is several months old and I can't easily upgrade=0A= > it for now.=0A= >=0A= > There was a recent ZFS problem that found its way into 13.0-p8 that=0A= > I'm pretty sure is fixed now. If you had a fairly recent server exporting= =0A= > ZFS, that *might* explain it?=0A= > =0A= > In particular, an NFSroot uses NFSv3 and nothing has changed for=0A= > NFSv3 in a looonnggg time.=0A= Oops, I did make a change that affected NFSv3 as well as NFSv4.=0A= Last December I committed a path to head that made IO_APPEND=0A= writes use nfs_directio_write() and avoid the buffer cache.=0A= =0A= Turns out nfs_directio_write() only worked for UIO_USERSPACE.=0A= The attached trivial patch fixes ktrace for me.=0A= I have no idea if core dumping ever does IO_APPEND VOP_WRITE()s?=0A= =0A= Please test the attached patch.=0A= =0A= Thanks, rick=0A= =0A= rick=0A= =0A= --=0A= John Baldwin=0A= =0A= --_002_YT2PR01MB973039BE3072ED3F94CFE141DD1B9YT2PR01MB9730CANP_ Content-Type: application/octet-stream; name="ktrace.patch" Content-Description: ktrace.patch Content-Disposition: attachment; filename="ktrace.patch"; size=416; creation-date="Sat, 26 Mar 2022 04:03:40 GMT"; modification-date="Sat, 26 Mar 2022 04:03:41 GMT" Content-Transfer-Encoding: base64 LS0tIHN5cy9mcy9uZnNjbGllbnQvbmZzX2NsYmlvLmMuc2F2CTIwMjItMDMtMjUgMTk6MzY6NTgu MzgxNTU4MDAwIC0wNzAwCisrKyBzeXMvZnMvbmZzY2xpZW50L25mc19jbGJpby5jCTIwMjItMDMt MjUgMTk6NDQ6MTUuMzg2NDY5MDAwIC0wNzAwCkBAIC03ODAsNyArNzgwLDcgQEAgZG9fc3luYzoK IAkJCXVpby51aW9faW92Y250ID0gMTsKIAkJCXVpby51aW9fb2Zmc2V0ID0gdWlvcC0+dWlvX29m ZnNldDsKIAkJCXVpby51aW9fcmVzaWQgPSBzaXplOwotCQkJdWlvLnVpb19zZWdmbGcgPSBVSU9f VVNFUlNQQUNFOworCQkJdWlvLnVpb19zZWdmbGcgPSB1aW9wLT51aW9fc2VnZmxnOwogCQkJdWlv LnVpb19ydyA9IFVJT19XUklURTsKIAkJCXVpby51aW9fdGQgPSB0ZDsKIAkJCWlvbW9kZSA9IE5G U1dSSVRFX0ZJTEVTWU5DOwo= --_002_YT2PR01MB973039BE3072ED3F94CFE141DD1B9YT2PR01MB9730CANP_--