From nobody Mon Feb 27 19:35:23 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PQW1V2DV1z3vWnP for ; Mon, 27 Feb 2023 19:35:26 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PQW1T6zW7z4CLn; Mon, 27 Feb 2023 19:35:25 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677526526; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+ARd3fee6ATh0q/3o4wUAfU6Zx5KzRYKt7TKclDXzCQ=; b=GcpKiFYF5r/ltLIVC8fwkwYhis1zBVh0b0U0vxlyX7B045Ft1BD6ev1M9iqCMrcTOXvBRF QaMlC7msd5ci6Gza84xMKWVjNoNnDLsaLA7gs8PCbyF2j05ZlclyxWnM32mB0dOWYjYPrF ZToIsgig0mvf0lc8kYAeoPcIXLJsBdM62ur5h0suyQSGy8FPWUOgLdeDcs1KI/i+Zivqz0 UfRgmNCE8EwIYM/oUOFDxDcwUUEmm31GQeHChBO6QgJjfKQGuUGB1GlzA7jpp8R7vpC4tK tWR5extCMXM1vLFZVTEh5JF2ew/o1P1LPXQCpZ3mhL8n569097ORG2TQ67FmeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677526526; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+ARd3fee6ATh0q/3o4wUAfU6Zx5KzRYKt7TKclDXzCQ=; b=yAWqlpVinCHV1UtEwomQAanFdq2vjQIV7LsQ0l85EVY4r8K2aI8YW63Q+o7ObDajwHHzRd lRmdIagWFEslUeTJ3ZMonYfBJIlIsvlK57uurVV0miADiR5PU2ba25127p/knCQ2UPsLK+ O3bVRzRVNg6qxVNYu99R8CSX+79xWDmz0//dmnl7plwuu61tkCwIsfUuZaUGuqA82FFoga HINoe6ZGcvLZ2uCsPBJosWuOCnyUJbCe3ciAzSn7sFAPMRiF6rdWeKpbpvGi50pxUOdjH4 TRbk0qluEjweupb0WsetfwLpOZ0BpJ4L/sZ0mzh3doWOMyoab3hjvwE+pu/dRw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677526526; a=rsa-sha256; cv=none; b=XmCWEaT2FZkd9AxIwKIsq7+ggG4H/v1/HgpkxoHxesr89kJ36qqd9uykZ9MraJJEIoImud RQzhx3s9m4QOjIvjdcDdyWJp0Z6Wh9zooJzCzV+x1TBpKUPda1B8CW6qrgt329Y7N3fc5k GpcnYhrpAGhuZMBocS4rGZobw4dJ35PuqWRwt02MhSLzI4+sLznzYp8WwJey9SQtAwgWfp xs0iRs9MppeMx2IG84cYewXhaWWnPF/cHtIMNMZvbb/TI/f8rxPrjqlgMx6kB2MKP0BOAj Uo55nFDzTEe2/JYMTwXiO5vzZ2gbq+QLbgOaWAveNiQ6wKCJg29keEy29z/1nw== Received: from [IPV6:2601:648:8680:16b0:a016:f345:edc0:be49] (unknown [IPv6:2601:648:8680:16b0:a016:f345:edc0:be49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PQW1T39sCz1C7K; Mon, 27 Feb 2023 19:35:25 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: <945be88e-b6f2-d616-7fa6-a1c6405f3bab@FreeBSD.org> Date: Mon, 27 Feb 2023 11:35:23 -0800 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 To: Zhenlei Huang , Rick Macklem References: <16C3AC44-4D19-4B6E-B277-03A3BF2C3E10@FreeBSD.org> Content-Language: en-US From: John Baldwin Cc: "freebsd-hackers@freebsd.org" Subject: Re: Confused about the kernel stack backtrace In-Reply-To: <16C3AC44-4D19-4B6E-B277-03A3BF2C3E10@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N On 2/25/23 10:26 PM, Zhenlei Huang wrote: > >> On Feb 24, 2023, at 11:43 PM, Rick Macklem wrote: >> >> Btw, thanks to markj@'s quick review, D38750 is now in main. >> I'll keep an eye on the ci test results, but I suspect this is now >> fixed. >> >> Sorry about the breakage, rick > > No worry. I was not blaming but think this might be an issue of DDB / KDB (for the falsely reported stack trace). Likely what happened is that the compiler moved the call to panic to the end of the function in a "cold" section, and since panic is marked "noreturn" there wasn't an instruction after the call to panic. The panic stackframe still saves a return address, it just points to the instruction after the call/branch. However, that instruction is no longer in the mtrash_ctor function, and if there wasn't a padding gap, it instead points to the first instruction of the next function, in this case mtrash_dtor. Some unwinders do try to correct for this (e.g. I think I've patched at least one somewhere in FreeBSD) by subtracting 1 from the return address when resolving the function symbol. However, you'd have to undo the subtraction to manually fixup the offset. Most of the time it really isn't worth dealing with as the other parts of the stack trace are sufficient to determine what's going on. >> >> On Fri, Feb 24, 2023 at 5:26 AM Zhenlei Huang wrote: >>> >>> CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca >>> >>> >>> Hi, >>> >>> The job FreeBSD-main-amd64-test on ci is failing, and some kernel stack backtrace [1] >>> looks weird. >>> >>>> Memory modified after free 0xfffffe00ccc29000(8184) val=0 @ 0xfffffe00ccc29698 >>>> panic: Most recently used by temp >>> >>>> cpuid = 0 >>>> time = 1677239728 >>>> KDB: stack backtrace: >>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0084e3eaa0 >>>> vpanic() at vpanic+0x152/frame 0xfffffe0084e3eaf0 >>>> panic() at panic+0x43/frame 0xfffffe0084e3eb50 >>>> mtrash_dtor() at mtrash_dtor/frame 0xfffffe0084e3eb70 >>>> item_ctor() at item_ctor+0x11f/frame 0xfffffe0084e3ebc0 >>>> malloc() at malloc+0x7f/frame 0xfffffe0084e3ec00 >>>> g_read_data() at g_read_data+0x82/frame 0xfffffe0084e3ec40 >>>> g_use_g_read_data() at g_use_g_read_data+0x46/frame 0xfffffe0084e3ec60 >>>> readsuper() at readsuper+0x29/frame 0xfffffe0084e3ecf0 >>>> ffs_sbget() at ffs_sbget+0x84/frame 0xfffffe0084e3ed70 >>>> g_label_ufs_taste_common() at g_label_ufs_taste_common+0x8b/frame 0xfffffe0084e3edc0 >>>> g_label_taste() at g_label_taste+0x1d0/frame 0xfffffe0084e3eea0 >>>> g_new_provider_event() at g_new_provider_event+0x9a/frame 0xfffffe0084e3eec0 >>>> g_run_events() at g_run_events+0x104/frame 0xfffffe0084e3eef0 >>>> fork_exit() at fork_exit+0x80/frame 0xfffffe0084e3ef30 >>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084e3ef30 >>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >>>> KDB: enter: panic >>> >>> The source code sys/vm/uma_dbg.c shows clearly that the panic comes from `mtrash_ctor()`. >>> >>> Why KDB shows that the panic is from `mtrash_dtor()` ? >>> >>> [1] https://lists.freebsd.org/archives/dev-ci/2023-February/003055.html >>> >>> Best regards, >>> Zhenlei >>> >>> > > > -- John Baldwin