From nobody Sun Jan 01 17:30:25 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NlQxh6S7mz2lK56 for ; Sun, 1 Jan 2023 17:30:32 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-7.mit.edu (outgoing-exchange-7.mit.edu [18.9.28.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NlQxh17mkz3Nl8 for ; Sun, 1 Jan 2023 17:30:31 +0000 (UTC) (envelope-from jfc@mit.edu) Authentication-Results: mx1.freebsd.org; none Received: from oc11exedge1.exchange.mit.edu (OC11EXEDGE1.EXCHANGE.MIT.EDU [18.9.3.17]) by outgoing-exchange-7.mit.edu (8.14.7/8.12.4) with ESMTP id 301HUSN6020354; Sun, 1 Jan 2023 12:30:29 -0500 Received: from oc11expo22.exchange.mit.edu (18.9.4.84) by oc11exedge1.exchange.mit.edu (18.9.3.17) with Microsoft SMTP Server (TLS) id 15.0.1497.38; Sun, 1 Jan 2023 12:29:29 -0500 Received: from oc11exhyb5.exchange.mit.edu (18.9.1.110) by oc11expo22.exchange.mit.edu (18.9.4.84) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Sun, 1 Jan 2023 12:30:28 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.174) by oc11exhyb5.exchange.mit.edu (18.9.1.110) with Microsoft SMTP Server (TLS) id 15.0.1497.42 via Frontend Transport; Sun, 1 Jan 2023 12:30:28 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Dc9Rjs9NzGpzB5RMzBPWBHwywGVEUTuVsg++zcDn68sI2KcEUbW7MBJ46/Mw+0OAjLrxgce63dXBIg4PykeeOxsFPDgsxZWXYB4wf2DKA60Y4JXuoIDUxUYXCaOKnRhEfAhEw2KqUlpkrcBcWMOJ/bBRBIOMHshFXnOu7PXFKV+3sHIfUZKShZT1ICoNgqwBPER/3QQGGqXtIzDonmgM6QAsMGQnTcpVJAIFKFWiIFUzW1hIgMxUdAsYdY353lnrs/OgE3aY79m1FI2+P1RcEfgtOS4j9fGyLX9wE30sEommwRtyCzdeE4S56+KewlGfaVJRRqaFzaRn/83UoqRVoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FgFbfxFQWuHfmAklxwFk5JYy/tnwfcSLsGMQXNOGwE4=; b=OYSvHkbIJ8aJB+sDJxMRyY0XP/hKZ4eGMSrtHug0I/aJQX5MgmXumsobI9Yc9ovdMvtkwLl5MhX2VjatHRVMcd+V+HCGlWXi5CsyxDkzvwHLPZgGwQGNLw8tiNwfM3dsStoGm9B/6HOqUedNm1n6PD+L/5NdwBsmnXWFIOXeyX9FDQshaQWfaAK7VqoQDekCmS+0nSU5Ht811NcXLHS4TwLKXO16cPL9O/nxMgvOIHuFYvGPk5ohOrnwcL++dRRmS0iDzVBQILXH+imHLbY27qlW08WN4F6Iecf0HvEiNIbr/CX/dnnYxF2XsdeS/PG1TTOUB93qkw+/lAxRta2YIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mit.edu; dmarc=pass action=none header.from=mit.edu; dkim=pass header.d=mit.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mit.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FgFbfxFQWuHfmAklxwFk5JYy/tnwfcSLsGMQXNOGwE4=; b=NTy0aAl7wthCVsuvBiiPIdTl+rdrITJnti9aOuDKE9GKXGlIic17IYxfvJ7LdHrI5+xx0aU6yBfkSt/VHHmPWgtJ0VOLfs8SPyJFQNfWmjMbH9WZEnYiGmKeNPLiHCXx9mBazt28tlr5N35C+xGUIFuKr87oDoZ6N2F86DJMeg4= Received: from DS7PR01MB7712.prod.exchangelabs.com (2603:10b6:8:7b::17) by CY1PR01MB2201.prod.exchangelabs.com (2a01:111:e400:c616::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.16; Sun, 1 Jan 2023 17:30:26 +0000 Received: from DS7PR01MB7712.prod.exchangelabs.com ([fe80::b8dd:4b3b:3d80:500b]) by DS7PR01MB7712.prod.exchangelabs.com ([fe80::b8dd:4b3b:3d80:500b%8]) with mapi id 15.20.5944.019; Sun, 1 Jan 2023 17:30:26 +0000 From: John F Carr To: Ronald Klop CC: Mark Millard , "freebsd-arm@freebsd.org" , Andrew Turner Subject: Re: lsof crashes in Arm Optimized Routines Thread-Topic: lsof crashes in Arm Optimized Routines Thread-Index: AQHZHd+B9l8QgVYiMEOVgUWRe9Xnf66J0ayA Date: Sun, 1 Jan 2023 17:30:25 +0000 Message-ID: References: <1331707040.259440.1668459233836@localhost> <490902644.115954.1668511998644@localhost> <690511F4-25A6-41E8-A75A-FFE80C352DFA@yahoo.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DS7PR01MB7712:EE_|CY1PR01MB2201:EE_ x-ms-office365-filtering-correlation-id: c76c162a-1338-433a-d764-08daec1dde86 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: PhwziLDvGOpJrz4m9Zg2vP76TswitaZoQnIsiR8glTlVMWoxE0TIu95ojg3U42N41ktExo12EcDsY5cMWEUHb8EDGajd7fnMWxaBbZNuQhR+3yV/rgO2xnqZHjv4kSuDsK+6WEAlmJplQLAtOtONOz2Jl8AiHl7JvFyMvjGO3+jKPGyk+xHcFdBtwn6+upSfZOwJpsSOs+9DPQCzHTsL5wDzDXZlhhhEkQNOKnh4/aTytLs9YOzRcfKPYPO5B27VhNlSjYvPlgadOTHvOEeVIYM93TPv/ibObydBMzZqLFWvf6+VKsxbHoN9remlYk1NttteU7cs25gSXbrufPiLYjmBJhoNHmrHh2ystnESalYCmTcyE4DDvEAoZTXvqJ2uHHqJnj9aArBk+Y+CfXk/gjBi728SG/0RxPJO736j7t2gszTBn43RD4tpo1ImZ7dhhWqa+WzOVt7S42UT50TbUXaJESPUyxcONZkeK/Xcqk5jR7JvBh6b3lZXaOEY/+AGHFtp/fIG/O3YzVkTj6E1L3mIFeSD4A/+yLiIO51GdPBJnCIfLleQ5IfIf9LII8M5PzDaIIJRoyrHvOvnsWF3HgfixmSQS3cj3cM2SN07MDLaihh560qtOK6BQnyuDUuqCQLaVZWT1M5R9Q99ZvKDmiozS2V2OGlV2LkoMxEvGU/RF8r2BJnu2ldXvACu/kcsWIlKbQFZRiq6WeNR+iKcaDbCIqj+1nX4MfurI+OdCtespWFz/5yUGzUyAu/HObT+ x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR01MB7712.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(376002)(396003)(366004)(39860400002)(136003)(346002)(451199015)(38070700005)(478600001)(6506007)(6916009)(6486002)(966005)(33656002)(54906003)(36756003)(2616005)(75432002)(53546011)(86362001)(41300700001)(2906002)(26005)(38100700002)(71200400001)(786003)(186003)(6512007)(316002)(83380400001)(5660300002)(8936002)(91956017)(122000001)(66446008)(8676002)(66946007)(64756008)(66556008)(66476007)(76116006)(4326008)(22166003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?KG52Cb5UCfiwGoWv37hfwPwLdAu255daN+8MKQ0AXB+UT6ZvDV7upaEoh3F/?= =?us-ascii?Q?S4Ts/H3hv1Lng6KRFnqZ9wHgxCaV1/MNAow+6ZeU1572WSCvN5jVL2emwFOq?= =?us-ascii?Q?/O9sFInjvEPwal7LsCYxJroixG8dO3JDuZhHolWPIoDlwMNEBcvzWKUI3ehz?= =?us-ascii?Q?PkHARjCiFCduS6an7kK0nsmDoPFeXKlGLjiYnTSe5GXLvmVdoT/lqXlar6bD?= =?us-ascii?Q?7mMEbquRgAyFrhK32eipEcBke7D0s29W86JPny4r7aqgb75Uy74WfwB5bxYX?= =?us-ascii?Q?swKdE0xswdX0tTZ7QUca96s4RnClqFPPx5uR7SE/Q+zLhxPE82mJQrQPQscU?= =?us-ascii?Q?JzPsGBa5iMcWSDVMJ2+fMMZzeWWMjjZ/fBqgX9M3y867Tr0joH7nYEQU0PXE?= =?us-ascii?Q?Kc97qunS/+RHosHWV4mL6QKRS3SK/538BGgLxHbUlToq9bYp3uxRlbu57BYj?= =?us-ascii?Q?evuOzWT0U9DBHeiJ8gWO35YmlqRmqajJ7xJopOyJAnXGOygMi1awCak5qCvk?= =?us-ascii?Q?lFmCW1poIsK9j6VBAwlB+2MmIQLrzohE6KupJxmhl0RWhHLHCOdKC20Hj6mk?= =?us-ascii?Q?25Z2oiYrNJo3b6JNmJA/D4ihcnWneEDs/TVNvt4ZUD8v7tTDZWOBkBHECtNv?= =?us-ascii?Q?13tlAX6oxHHE59mSU5gNqrs/Um4vahdAZu8er4JRsx6yx5ECakzjPVLIemJk?= =?us-ascii?Q?I4745Utg25Kds7JfEGa8jGjGDPOAE0QU9vEXSyhQCKi6Y5mMUVrpj3X2kJB1?= =?us-ascii?Q?jpZ5OtFLwmkJW3bX95d5epI7kZYxrks07xMwY5obqDh0zYjvVKeQxlonuOZs?= =?us-ascii?Q?7/fq/oWgm1pGQ8B5dF6OIcMWrzKWU4L8SX4S8sHb0SkfflGFLTsl/cXSlvSY?= =?us-ascii?Q?Bd0wppbt+XQlv+Em9l7rZnCNqxneixsQ8LuSSYqdtopPVM8x3XFpSpeuOyNn?= =?us-ascii?Q?iuMb5wgMm/Ea7YvcZpPMfbDy0b5uVAEOHiiNSWAVJirWcrqOtcb2RClU2WhA?= =?us-ascii?Q?mg1PXQOK3Ukm8FQ1t+cf41Ok32FcJFisfmlxrq8+IA2t1HkWN0gh6RjCHWRH?= =?us-ascii?Q?aoBKrqKaSPcsuGsLTerIM6L2v7hKzdWuQ9i5zUbQXXBCPwR6DbFo5MB9bm2y?= =?us-ascii?Q?rvdNaISc9SQxZYPsY5vJZBM1pC474ZKYRxrfRI5Pnbpj+0linxWDu8ytP9oY?= =?us-ascii?Q?n88eRk0D+YyQeh2k7VKmq90YU/ZzrOW0JxC3k/PsoJ6blpSc/0o0TiTwBar6?= =?us-ascii?Q?/xwpcs9HnKjanWOrZ5hSpR7a1SU8BmirGVTLgwkDWk4JeNtPGQgkyoVaJrUJ?= =?us-ascii?Q?K299NoNqLY3TNMSY9HS9znHvOP/H99v14im5OXepYVrGUixmLlm7jR45mjT1?= =?us-ascii?Q?CKvyXcrQwB0/5KevnG9LotBjzNrtcd4ALWFa61pfum0X79nQ1LzziDx4gLCO?= =?us-ascii?Q?2vyuXiTSW+8578Qtu1lPUEP3/Y9+xw5AcBtYBx6F1jEDA7zQllSG3uPXcWRL?= =?us-ascii?Q?PjIa0uf1TLYbGiBeK/7pIAkPtjDscCUmJU+aRcm8bAzRp7tzyOjnp4fBF1wm?= =?us-ascii?Q?r5cT25byZMhT9LMj9xs=3D?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DS7PR01MB7712.prod.exchangelabs.com X-MS-Exchange-CrossTenant-Network-Message-Id: c76c162a-1338-433a-d764-08daec1dde86 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jan 2023 17:30:25.7776 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 64afd9ba-0ecf-4acf-bc36-935f6235ba8b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 0IYJfg2Ie3LiJKe9WP4vDiBQs05XwAsv5SXxDpcD/Fb56gd8nLDBzqdaS5T1VCYY X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR01MB2201 X-OriginatorOrg: mit.edu X-Rspamd-Queue-Id: 4NlQxh17mkz3Nl8 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On Jan 1, 2023, at 07:49, Ronald Klop wrote: >=20 > On 11/18/22 01:57, Mark Millard wrote: >>> On Nov 15, 2022, at 03:33, Ronald Klop wrote: >>>=20 >>> Sorry for the noise. >>>=20 >>> But I cannot reproduce this today. I can scroll back in my terminal and= see the command and error from yesterday, but running the same again just = works. >> FYI: >> I do not have specifics any more, but I'll note that I've seen >> such lsof behavior of failing at one time and later working >> without any installed updates to it or the system between. I >> rarely use lsof and, so, this was not recently. >> I've no clue how to cause the failure(s) to show up. I've no >> clue how common the issue is. But, over time, it is not just >> you. >>>=20 >>> Van: Ronald Klop >>> Datum: maandag, 14 november 2022 21:53 >>> Aan: freebsd-arm@FreeBSD.org, Andrew Turner >>> Onderwerp: lsof crashes in Arm Optimized Routines >>> Hi, >>>=20 >>> See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D267760 : Segmen= tation fault in lsof. Program received signal SIGSEGV, Segmentation fault. >>> Invalid permissions for mapped object. >>> memcpy () at /home/ronald/dev/freebsd/src/contrib/arm-optimized-routine= s/string/aarch64/memcpy.S:175 >>> 175 stp D_l, D_h, [dst, 64]! >>>=20 >>> I also remembered this change: https://cgit.freebsd.org/src/log/contrib= /arm-optimized-routines?showmsg=3D1 about Arm Optimized Routines. >>>=20 >>> Could this be related? What can I do to help debug this? >>>=20 >> =3D=3D=3D >> Mark Millard >> marklmi at yahoo.com >=20 >=20 > I'm having this issue again. >=20 > No debugging symbols found in lsof) > (gdb) run > Starting program: /usr/local/sbin/lsof >=20 > Program received signal SIGSEGV, Segmentation fault. > Invalid permissions for mapped object. > memcpy () at /home/ronald/dev/freebsd/src/contrib/arm-optimized-routines/= string/aarch64/memcpy.S:171 > bt > 171 stp B_l, B_h, [dst, 32] > (gdb) bt > #0 memcpy () at /home/ronald/dev/freebsd/src/contrib/arm-optimized-routi= nes/string/aarch64/memcpy.S:171 > #1 0x0000000000218be4 in ?? () > #2 0x0000000400000000 in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?= ) > (gdb) >=20 >=20 > Some output of "truss -o /tmp/lsof.txt lsof": >=20 > __sysctl("kern.proc.filedesc.1",4,0x0,0x80ba06f0,0x0,0) =3D 0 (0x0) > __sysctl("kern.proc.filedesc.1",4,0x851d6000,0x80ba06f0,0x0,0) =3D 0 (0x0= ) > __sysctl("kern.proc.filedesc.385",4,0x0,0x80ba06f0,0x0,0) =3D 0 (0x0) > __sysctl("kern.proc.filedesc.385",4,0x8516ec00,0x80ba06f0,0x0,0) =3D 0 (0= x0) > __sysctl("kern.proc.filedesc.97537",4,0x0,0x80ba06f0,0x0,0) =3D 0 (0x0) > __sysctl("kern.proc.filedesc.97537",4,0x8516ec00,0x80ba06f0,0x0,0) =3D 0 = (0x0) > statfs("/data/jails/jail13/_root/home/root/dev/workspace/FreeBSD-Ports-13= /_root/usr/local/poudriere/data/.m/freebsd13-custom/04/bin/sh",{ fstypename= =3Dnullfs,mntonname=3D/data/jails/jail13/_root/home,mntfromname=3D/data/jai= ls/_home,fsid=3D3cff022929000000 }) =3D 0 (0x0) > statfs("/data/jails/jail13/_root",{ fstypename=3Dnullfs,mntonname=3D/data= /jails/jail13/_root,mntfromname=3D/data/jails/freebsd13,fsid=3D37ff02292900= 0000 }) =3D 0 (0x0) > statfs("/data/jails/_home3root/dev/workspace/FreeBSD-Ports-13/_root/usr/l= ocal/poudriere/data/.m/freebsd13-custom/04/bin/sh",0x80b9ef40) ERR#2 'No su= ch file or directory' > statfs("/data/jails/_home3root/dev/workspace/FreeBSD-Ports-13/_root/usr/l= ocal/poudriere/data/.m/freebsd13-custom/04/wrkdirs/usr/ports/devel/cmake-co= re/work/cmake-3.24.3/Source",0x80b9ef40) ERR#2 'No such file or directory' > statfs("/data/jails/_home3root/dev/workspace/FreeBSD-Ports-13/_root/usr/l= ocal/poudriere/data/.m/freebsd13-custom/04",0x80b9ef40) ERR#2 'No such file= or directory' > statfs("/data/jails/freebsd13ovt",0x80b9ef40) ERR#2 'No such file or d= irectory' > SIGNAL 11 (SIGSEGV) code=3DSEGV_MAPERR trapno=3D36 addr=3D0x80ba1000 > process killed, signal =3D 11 (core dumped) >=20 >=20 > I'm surprised that the path names in the truss output are corrupted: _hom= e3root should be _home/root. >=20 > NB: I'm using lsof while running poudriere in a jail in a Jenkins agent. >=20 > Regards, > Ronald. >=20 >=20 I think this is a bug in lsof and the optimized memcpy routine is doing wha= t it is asked to do, copy into a block of memory that the caller does not h= ave write access to. The faulting data address 0x80ba1000 is at the start = of a page. The faulting instruction address is in the middle of a block of= code that writes to successively increasing addresses. The destination po= inter passed to memcpy must be valid or the function would have crashed ear= lier. But the end address is out of bounds, meaning the size is wrong. If= you can get the program in a debugger again, or you can find a core file, = check the value of register x2 ("count" in assembly code). If that is huge= then you have an uninitialized or otherwise invalid third argument to memc= py. In a jail system calls to determine the current filesystem behave different= ly. The odd path names may be symptoms of jail-induced confusion.