From nobody Thu Mar 06 14:34:32 2025 X-Original-To: wireless@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Z7sPm1X4lz5pgw3 for ; Thu, 06 Mar 2025 14:34:36 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Z7sPm14nqz3DdG; Thu, 06 Mar 2025 14:34:36 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1741271676; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=QxY00vgDtetdQXRf84I6+luMoTUsXNFeypAbRFEcCdo=; b=LOJci1gkl9Ph4vx5ur8Dk7/nbIMaJMqIid7c+fcH7AvCm8bcoZ27l8dSQy+zojlffK7t1z ql3Ytn/clDpStFW/YF1c+DPx1I1RaiU6eHjhLy83AfgpPYTgIQXfwtBUB3qSQ5wH9q8FKJ NkF85DmFWOGIcGeoWsg7UQxvEdUHiOhUWM9EtufK4mNuyCo+3tc1PL0GWMDztduq1u/y0R YHrne/YXZ24mo0jVeIfwHrsF2MZLy4Aakx/6XQTLhoV/zOFhJmxKyyYO74f8Wjz53Exqtw en5H81wOQISoNEp9nd503cIF2Y0wVx9lXFryIBV4aDuGYuBOcSVPTadiRrZXYQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1741271676; a=rsa-sha256; cv=none; b=xFJLL9HYTAm154GjE9o8awAtEeq/wz2HvDdZPwE0hY77kGteiylsY0MYUDklzEgAzYq2e7 M3w2Wd43KzRflWc8wUKmqsMlEMz7CSLnVXCeGJe9A0stUwWgkdZl/JKgyYt7Hpq0ZuE8SF BDDxhiKra239UeF1Ox+MzIYKygcB5EWvlxr8iT2lolmpAssJnowL0O0vaA3yux2h+abWQO MfY6/SCu13E6Uv2fykcaH0hzOPfbzo+wvy6F6BeiagNqCbIVyPeBm7T60Py/d0lCLoUb+B H2Kprj+jr1nTc6EC6H37FUL7stSvicg3HxkfrvllPEMqVrkM+oD0onl4Y+n0IQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1741271676; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=QxY00vgDtetdQXRf84I6+luMoTUsXNFeypAbRFEcCdo=; b=Pu+9MlynUoVADiHcCF4imwsC6Z7B/1aS5hjbS1UdjHcJ7/tR9rq2NYi/sdNQWcwv+Tmju4 n0uG10idvOgLEdD1bAdawGIKASx67vTmgQhfz9Yw1ZQYV8szxfAaTM2enOsTWBfRCmISTV IFsbMDk1A2AIIEkwW9e9iloKlqzCeyzGNHgs4ZaBLYoxUGhx6SKykpAJGXlU9MR3B1zFru 4e/FkOJPqw1z+tCGLp/CzGuf5wyR/KjNUMHA1ixCg5HBSVL8IkbnWulClIi0FQVgCRx9dn iSNdlnt6ncNkS1J/NNyBWoagWs9TjzKdqtUL5QjTk0cl+yi0/jIx2gtnI1oYQg== Received: from mx-01.divo.sbone.de (mx-01.divo.sbone.de [IPv6:2003:a:140a:2200:6:594:fffe:19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (prime256v1) client-digest SHA256) (Client CN "mx-01.divo.sbone.de", Issuer "E5" (verified OK)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Z7sPl5gqZzx9Z; Thu, 06 Mar 2025 14:34:35 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by mx-01.divo.sbone.de (Postfix) with ESMTPS id 567C3A64808; Thu, 06 Mar 2025 14:34:32 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 23CD52D029D8; Thu, 6 Mar 2025 14:34:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id 7k0yShd0Q6xO; Thu, 6 Mar 2025 14:34:32 +0000 (UTC) Received: from strong-rtwn0.sbone.de (strong-rtwn0.sbone.de [IPv6:fde9:577b:c1a9:4902:da44:89ff:fedd:d5ab]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id BFB3F2D029E0; Thu, 6 Mar 2025 14:34:32 +0000 (UTC) Date: Thu, 6 Mar 2025 14:34:32 +0000 (UTC) From: "Bjoern A. Zeeb" To: Adrian Chadd cc: Baptiste Daroussin , FreeBSD wireless mailing list Subject: Re: HEADS UP! Do not update on main currently (panic - on boot) In-Reply-To: Message-ID: <6257760p-p535-8p6q-8096-9489q6on4837@serrofq.bet> References: <38523r8n-46pr-9r68-8049-47pqqr774711@SerrOFQ.bet> <63xz4b6wrnhr4zwune5sbdflumjk5nlqwkq35cm2yh55sskkm7@nebya5fcoknu> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-wireless List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-wireless@freebsd.org Sender: owner-freebsd-wireless@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed On Thu, 6 Mar 2025, Adrian Chadd wrote: > On Thu, 6 Mar 2025 at 01:44, Bjoern A. Zeeb wrote: > >> On Thu, 6 Mar 2025, Baptiste Daroussin wrote: >> >>> On Wed 05 Mar 23:39, Bjoern A. Zeeb wrote: >>>> Hi, >>>> >>>> the fix from >> https://cgit.FreeBSD.org/src/commit/?id=27bf5c405bf2eb69392e45c06605defc78882612 >>>> makes wireless panic for me in my development branch; that said I am >>>> working on TKIP currently so I have code in progress. >>>> >>>> In case you hit the panic (on boot) because you have updated, please >>>> disable the HW_CRYPTO tunable from loader and live without HT/VHT for a >>>> few hours again. >>>> >>>> I am looking where the problem comes from now and for a fix then. >>>> >>>> /bz >>>> >>> >>> Thanks for the heads up! >>> >>> The weekly snapshots for pkgbase are generated the weekend, if you >> haven't been >>> able to figure out a fix by then, if would be nice for pkgbase consumers >> if you >>> could revert the change before the next snapshot building. >>> >>> Many people took the habbit to run pkgbase from weekly snapshots and >> reboot >>> every monday on a fresh new current machine. >> >> I have a very crude workaround I am testing. I still hope to be able to >> fix it more properly if net80211 locking allows us. >> >> The problem isn't so much of reverting. >> 11db70b6057e41b259dc2245cd893d5b19179fcc >> taked about the possible panic on key delete I could not reproduce >> anymore. As I had guessed the mentioned commit was the reason. The >> problem was that that commit had logic inverted and we corrected that >> yesterday so now net80211 and ath AP mode and others are fine again but >> the panic on key delete is back depending on timing. >> >> The problem is that net80211 already has incosistent locking there and >> we have to conditionally unlock for the downcall to driver/firmware >> which can sleep. >> Both together opens a possible race which is there in theory with or >> without LinuxKPI as it is two threads (wpa and net80211) racing. >> I am just able to trigger it again (note that some drivers simply omitted >> these bits; that's why you don't see ic_cryptocaps set in iwm or iwn) >> but that no longer works as firmware needs to be able to handle crypto >> for doing A-MPDU offloading. >> > > Ah crap, can you email me the panic and stack traces? > > It seems fine on rtwn which does HW crypto, but its also very possible that > rtwn > already has the workarounds. Or multiple: - unlocked register writes - rtwn_cmd_sleepable() which defers it to a task which doesn't fly but simply opens more problems, especially for the "SET" operation; otherwise you'll have to hold all packets until you get a callback (I presume) [and make sure crypto state doesn't change anymore]. Even more fun, you have to make sure a del key is done before doing another set; which either will have to error or sleep again if the del is sitll pending so simply "reduces chances", or use the taskq for serialization as well (going back to holding packets while any crypto operation is in progress). Seems either way it's not a clean win. Given LinuxKPI drivers return flags (still an open TODO in my TKIP land, silly TKIPMIC 'cipher') on setting the key we definitively need to wait before any further packets or touching the crypto state. Also you do not normally have sleepable locks under you in native drivers like LinuxKPI drivers do. mwl seems to work that way. I hadn't looked what wpi, ath, rsu, rum, run, and mtw but this made me: Okay rsu uses a task. rum uses rum_cmd_sleepable() (assume also task) run uses ieee80211_runtask mtw uses ieee80211_runtask wpi seems to use native locking and deal with the async somehow? ath seems to use native locking around the keycache sync (you probably know how that simply just works)? So the problem has long been known and everyone ported around it rather than fixing it (that they may need to sleep); and no one noticed the theoricaly possible race in net80211 working around the issue (or ignored it). That said my solution is also to simply swap locks for now and use iv_key_update_begin/end as a start, as a proper solution and testing STA+AP + various drivers will take days and given bapt asked for something working before the weekend I decided to go that way for now. Still better than detecting the second thread and spinning there... /bz -- Bjoern A. Zeeb r15:7