From nobody Thu Apr 07 19:50:21 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1AC0D1A8D2B4 for ; Thu, 7 Apr 2022 19:50:31 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KZBnL2RVcz3DG9; Thu, 7 Apr 2022 19:50:30 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qt1-x833.google.com with SMTP id t7so8929932qta.10; Thu, 07 Apr 2022 12:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=Bpg+5c66kpbDvCyGSsGPM1bGWJ8Hf7XlSTsq18btt0o=; b=E255mwU64tOis156NZGSJKGCEDd2KGyTFDkrmR1Oo/wsU6exfQ8vDypaIIL7lo7g0a URpVk8A+sUNOO7Z+RMS4JTf7yBDSAm8jjjqPjNbd+hunKnABSfpzzuw1XAPjwiuT4HUg rQXJujDDf7VfcYA6EVSSvHOyO9vAc5+fWQ5jXgeDttR/9mA2pjPq06jLkB+mTrtNx/bb ELlEo+KzNOPqJQDN7V2p9+vs45lmezx/6oE5s5mkTqh0DnaBcqaB1PX0Cx49Qr4f8fik gyUFfxajRwODNpjgK6pGT/QAVbTGF8vaafpvARAEEeDPwiXErPdV65fue1qob5ESel19 vsfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=Bpg+5c66kpbDvCyGSsGPM1bGWJ8Hf7XlSTsq18btt0o=; b=f6oM8nfCIe0VmR6YaVSsIGkf6CGGU4qnkqjCOcj8qTKN61oH+bUSnpElB2xXgzTF/G mG9RrIA45NXmjS/jUpCk8Fr6CSGHkfWuWlh6D5Ak5zhcljGeshwLvndwalzCa2/P0O4A G9i01olBSSeykz9w+RoGdJ3OxD+R4GDX+9Hn1ZWKGjSomjZjKik6oQek5yv7HZULd3pc ymouQJeRj3mpVR6p+HmyDvclh9oeC1CcubJ84Td5TLsOgL1XifGvvOII5b7wwBz4Kbis g8PdKMAfhUJi/1GYSNG4BuaFjG4ADEq3gE851ZExYv7p19OH+OGEECbQ+N/k0ceAgNQM k/qg== X-Gm-Message-State: AOAM532KGJOyoPfyNOCmjrX6tV9dcIeGhxmJRQhID4DA8EJcS9hdg/YV UdJ1n2+dlVHkjoAyssgxEaY8qFLc1qE= X-Google-Smtp-Source: ABdhPJw3piX3ASgMH1nfZkBz5MnqpgyVk8sOX+6zCihqWfrE4E73UAKIjySiQRHij4wg06fKVnLUWA== X-Received: by 2002:ac8:5f95:0:b0:2e1:dc8b:f851 with SMTP id j21-20020ac85f95000000b002e1dc8bf851mr13104224qta.36.1649361024391; Thu, 07 Apr 2022 12:50:24 -0700 (PDT) Received: from nuc (198-84-189-58.cpe.teksavvy.com. [198.84.189.58]) by smtp.gmail.com with ESMTPSA id b82-20020ae9eb55000000b0067b314c0fedsm12518785qkg.42.2022.04.07.12.50.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Apr 2022 12:50:23 -0700 (PDT) Date: Thu, 7 Apr 2022 15:50:21 -0400 From: Mark Johnston To: "Bjoern A. Zeeb" Cc: Peter Holm , freebsd-net Subject: Re: epoch callback panic Message-ID: References: <23673db5-5798-5c06-c04c-7c680ebe41d4@selasky.org> <2E5707B9-63C6-4F8C-AAD5-692B14FF5DE1@lists.zabbadoz.net> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2E5707B9-63C6-4F8C-AAD5-692B14FF5DE1@lists.zabbadoz.net> X-Rspamd-Queue-Id: 4KZBnL2RVcz3DG9 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=E255mwU6; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::833 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.45 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; NEURAL_HAM_MEDIUM(-0.81)[-0.810]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; NEURAL_HAM_LONG(-0.95)[-0.954]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.99)[-0.988]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::833:from]; MLMMJ_DEST(0.00)[freebsd-net]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On Fri, Apr 01, 2022 at 09:15:39PM +0000, Bjoern A. Zeeb wrote: > On 1 Apr 2022, at 20:51, Peter Holm wrote: > > > On Fri, Apr 01, 2022 at 10:33:15PM +0200, Hans Petter Selasky wrote: > >> On 4/1/22 19:07, Peter Holm wrote: > >>> markj@ asked me to post this one: > >>> > >>> panic: rw lock 0xfffff801bccb1410 not unlocked > >>> cpuid = 4 > >>> time = 1648770125 > >>> KDB: stack backtrace: > >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > >>> 0xfffffe00e48a3d10 > >>> vpanic() at vpanic+0x17f/frame 0xfffffe00e48a3d60 > >>> panic() at panic+0x43/frame 0xfffffe00e48a3dc0 > >>> _rw_destroy() at _rw_destroy+0x35/frame 0xfffffe00e48a3dd0 > >>> in_lltable_destroy_lle_unlocked() at > >>> in_lltable_destroy_lle_unlocked+0x1a/frame 0xfffffe00e48a3df0 > >>> epoch_call_task() at epoch_call_task+0x13a/frame 0xfffffe00e48a3e40 > >>> gtaskqueue_run_locked() at gtaskqueue_run_locked+0xa7/frame > >>> 0xfffffe00e48a3ec0 > >>> gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame > >>> 0xfffffe00e48a3ef0 > >>> fork_exit() at fork_exit+0x80/frame 0xfffffe00e48a3f30 > >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e48a3f30 > >>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > >>> > >>> Details @ https://people.freebsd.org/~pho/stress/log/log0275.txt > >>> > >> > >> Hi, > >> > >> Maybe you need to grab the lock before destroying it? > >> > > > > This was on a pristine main-n254137-31190aa02eef0. > > If there was no other emory corruption and my memory serves me right, > la_flags = 0x1 was LLE_DELETED which gives a good hint on the call path. > > If I had to bet it’s coming out of the 2nd condition in > in_scrubprefixlle ... I’d check lltable_delete_addr .. and so on .. I believe this is fixed by https://reviews.freebsd.org/D34831 .