From nobody Sat Sep 30 23:00:03 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RyjNN28Dkz4vwk0 for ; Sat, 30 Sep 2023 23:00:04 +0000 (UTC) (envelope-from jwd@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RyjNN1Dr0z3cHH; Sat, 30 Sep 2023 23:00:04 +0000 (UTC) (envelope-from jwd@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696114804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bL8LyWGzZB92URYqSXNfVZVaWFigvZ8OEEhhG8XL31o=; b=FUHrMNP1YLXJpWNm4yGRRNnXdMH1KnquB3RCrtYbCRtI9Xf1ZfFSBsLFHPDlwGN/8XouyO GqPQJWedoTzBvSVXVRexTsKt33OrmntAO2vak5L+u5BqoqkpA49YHFJXSiH/NrGSTSwB8J WZzXxGlobxmHGwQe9tOCa4IWNf1xLJ+vcrkcWNc/UaylFhPlJvGH+TPp+68gIAXZNR4Rzt RZh2/ZMzxjAl/ZJ2LTtwnBmK6qE1YK9wsSG3DCt6rHWByV7mpAdRYY86X50Vau1Mj00mo+ 7cW0/RCgXAnpm1lIJjYAMKEY7hPiq/yfn86nEatZarkfvd1G1os5uCRbKvuSjw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696114804; a=rsa-sha256; cv=none; b=aDm2fahNUVnkkBDjymT8r7pY19XdAph+ec3FmwdPG1iRRXl19Lczbr8xUjfFdCMlTyoFfD mmdvMv/EXmVn+ZgS6ZBkn4LsIDlkrAUoBKXeZViJ168ferfxOY93sXYWEdqK5wcbKvK1J9 ccSAW27llHoHR3ixX4ITlYj5Ha8N1qJtuPCpG2tOPBMTEFzMEM6A3xLnkaC7gtf7xwBCT6 ccpvDAHyz5rh2X1xMUOOQROVV7FnzsNcXKsGEAiaNTcrzIaUGi4bYCDGDR5X8bcbJUckU3 bd1/WAEStBjThCsFgnlaJQzo2NgYdgViiyAYIHZeWkPqiSdcvDnvSfSXfx8Iag== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696114804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bL8LyWGzZB92URYqSXNfVZVaWFigvZ8OEEhhG8XL31o=; b=QzPhH8i9T9BIramounS8iU0gCd9adpQz5kFhGC1mHHOBPQM13uQw25tiW2V1uZUqjPbY47 WLjbj1P+OHESyHnWZTb67RuTOJCt4Tf2BcuQny0bxaZZjoNqApEQ4CQu+ynOreYJ3kjj+m i13BfXH6LUfesqQPXSOCmiK9mCmgN+kmtHVCrDJseMCSD8jKjX3zZJViNulQt9+VCT7Vzu OutM4xx0yJk/z+/POIgN6CeZRSe5g0ElM3SaAsPEjbp/yPSyaPILrRR852z32Va9hMDeNE N8ImhD1ktqeGxY+045h8oB+YWuiA1aXtcH25/AfGo5vhnsJm2usbkR6c7rm7NQ== Received: by freefall.freebsd.org (Postfix, from userid 821) id ED8FF9338; Sat, 30 Sep 2023 23:00:03 +0000 (UTC) Date: Sat, 30 Sep 2023 23:00:03 +0000 From: John To: Konstantin Belousov Cc: "Freebsd-Net@FreeBSD.org" Subject: Re: kernel crash: devctl set driver -f mlx5_core6 ppt Message-ID: References: List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ----- Konstantin Belousov's Original Message ----- > On Fri, Sep 29, 2023 at 04:32:30AM +0000, John wrote: > > Hi Folks, > > > > Working against 13.2-STABLE. > > > > I have a chance to get some bhyve VMs running on new hardware > > with Mellanox 100Gb/s cards. After creating VF entries with iovctl > > at boottime, a devctl command to detach the mlx5 driver and attach > > the ppt driver causes the kernel to crash: > > > > # devctl set driver -f mlx5_core6 ppt > > > > backtrace here: https://people.freebsd.org/~jwd/mlx5.dump.txt > What is the line number for pci_iov_detach_method+0x5e? > Better, load vmcore into debugger and get the backtrace from kgdb. Took a bit to get a netdump off the system. Results: #12 0xffffffff810e0a89 in trap_fatal (frame=0xfffffe278b8e9860, eva=2016) at /usr/src/sys/amd64/amd64/trap.c:940 #13 0xffffffff810e0adf in trap_pfault (frame=0xfffffe278b8e9860, usermode=false, signo=, ucode=) at /usr/src/sys/amd64/amd64/trap.c:759 #14 #15 0xffffffff80860a0d in PCI_IOV_UNINIT (dev=0xfffffa0085704400) at ./pci_iov_if.h:44 #16 pci_iov_delete_iov_children (dinfo=0xfffffa0086e3b300) at /usr/src/sys/dev/pci/pci_iov.c:873 #17 0xffffffff808607ce in pci_iov_detach_method (bus=, dev=0xfffffa0085704400) at /usr/src/sys/dev/pci/pci_iov.c:208 #18 0xffffffff80ea64a1 in PCI_IOV_DETACH (dev=0xfffff80110ffb400, child=0xfffffa0085704400) at ./pci_if.h:510 #19 pci_iov_detach (dev=0xfffffa0085704400) at /usr/src/sys/dev/pci/pci_iov.h:47 #20 remove_one (pdev=0xfffff8011000c180) at /usr/src/sys/dev/mlx5/mlx5_core/mlx5_main.c:1739 #21 0xffffffff80e82666 in linux_pci_detach_device (pdev=pdev@entry=0xfffff8011000c180) at /usr/src/sys/compat/linuxkpi/common/src/linux_pci.c:524 #22 0xffffffff80e84e74 in linux_pci_detach (dev=0xfffffa0085704400) at /usr/src/sys/compat/linuxkpi/common/src/linux_pci.c:514 #23 0xffffffff80c4d9f6 in DEVICE_DETACH (dev=0xfffffa0085704400) at ./device_if.h:234 #24 device_detach (dev=dev@entry=0xfffffa0085704400) at /usr/src/sys/kern/subr_bus.c:3093 #25 0xffffffff80c54f7b in devctl2_ioctl (cdev=, cmd=, data=0xfffff8014bcc8500 "mlx5_core6", fflag=, td=) at /usr/src/sys/kern/subr_bus.c:5949 #26 0xffffffff80aa376c in devfs_ioctl (ap=0xfffffe278b8e9ba8) at /usr/src/sys/fs/devfs/devfs_vnops.c:942 #27 0xffffffff80d0c7b8 in vn_ioctl (fp=0xfffff80150025730, com=18446744071589646072, data=0xfffff8014bcc8500, active_cred=0xfffff80166137300, td=0x0) at /usr/src/sys/kern/vfs_vnops.c:1701 #28 0xffffffff80aa3e3e in devfs_ioctl_f (fp=0x6, com=18446744071589646072, data=0xfc, cred=0x0, td=0x0) at /usr/src/sys/fs/devfs/devfs_vnops.c:873 Moving up the stack frames: 41 static __inline void PCI_IOV_UNINIT(device_t dev) 42 { 43 kobjop_t _m; 44 KOBJOPLOOKUP(((kobj_t)dev)->ops,pci_iov_uninit); 45 ((pci_iov_uninit_t *) _m)(dev); 46 } p ((kobj_t)dev)->ops $8 = (kobj_ops_t) 0x0 #define KOBJOPLOOKUP(OPS,OP) do { \ kobjop_desc_t _desc = &OP##_##desc; \ kobj_method_t **_cep = \ &OPS->cache[_desc->id & (KOBJ_CACHE_SIZE-1)]; \ Leading to OPS == NULL > > If I create the VF in ppt mode I can correctly detach > > the ppt driver and attach the mlx5 driver. > > > > Also of note, if multiple VFs are created and a single > > VF is targeted for the detach operation, all VFs are operated > > on. It seems the VFs are not seen as individual entities > > but a group of children in detach_method().