From nobody Mon Feb 27 16:36:58 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PQR3h6tnGz3vKc9 for ; Mon, 27 Feb 2023 16:37:04 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PQR3h1xCQz3mPm; Mon, 27 Feb 2023 16:37:04 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=bA4MLi5o; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::836 as permitted sender) smtp.mailfrom=markjdb@gmail.com; dmarc=none Received: by mail-qt1-x836.google.com with SMTP id c19so7229988qtn.13; Mon, 27 Feb 2023 08:37:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=NPhS+PRHH4Q4zu2GiAPg/mfeVaIn1t6NJAXCmhLDeQA=; b=bA4MLi5oKUB6JRSBm1hu9PMvn+w+nRyeiQ8seYY8tKOUvtixJFoDcz0hl4SzbciViU 2GINyJjhR0N5h/fYaIQ8PoHG5ObWWXsUxdYEq6M2miQCf4oLpDGquvJi+egscFzwQ2lw Im/ZtgTD73xAXe4sWfUhYvmNNjZQjUKhdazmkX/SrjKbV2JFPqbJ0qSExtb/+A75/OMn epIrZqadStylTPSMMpCEemOQrVztFdnkmnkZReaBFJeKF5acOomObX/IuaKWCERXwsJc p+SblAl8ecBqC9a+fct7H6/s8qrtx084zt7yR7yfedpyThlilC8IDbzg0w5DtuNXSAjS MBBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NPhS+PRHH4Q4zu2GiAPg/mfeVaIn1t6NJAXCmhLDeQA=; b=aap5DOSc6FFty/SNdMXkKK4RvAttrb17FNWscboQcC/go17WJDa0/kdpSM2DxwIJGV J5hypfJJ+lIUxbB6s1xs4P9DmsCZ37b8/3B+700D9jT2Mj++J/FmZO+bs+jlcWVR4eQr d7XLErr09PsODvDRYvAiYdoy4knuSfc5491mOg89yPWJxVJZPHxDBjRGfMNvxEvpJ+Fh ZZ/1p2xijzFQERQbZ8WaBuXXY3FXPnzcIM1ZUFsLK5I5YAsOCfRuEdbLCcWsM8YT6/v5 IYKzcHUTM9jXRKAHFqmnlTb9hxRgAIP9TZXvxqNh9AXmUwGBVRSBRhcMMdxYVtJDX55t s/7A== X-Gm-Message-State: AO0yUKWYhubnw3UnnQ7Sxx2ybE3oNFd4++iEj+98R7OTP8y9dMjKTgBJ xgSA8FTsMsz0m8N1W7WCgntaR+BkxBQ= X-Google-Smtp-Source: AK7set8btRS2/vDvD+cvJ13A6WgKHdwIsvVD/2yHkmpgjxqWpJeRKtUlYjtK7ntYNoSX3DBVpjlu8w== X-Received: by 2002:a05:622a:343:b0:3ba:151a:d300 with SMTP id r3-20020a05622a034300b003ba151ad300mr32488780qtw.60.1677515820828; Mon, 27 Feb 2023 08:37:00 -0800 (PST) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id o9-20020ac841c9000000b003b6382f66b1sm4876853qtm.29.2023.02.27.08.37.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Feb 2023 08:37:00 -0800 (PST) Date: Mon, 27 Feb 2023 11:36:58 -0500 From: Mark Johnston To: Zhenlei Huang Cc: freebsd-hackers@freebsd.org Subject: Re: Confused about the kernel stack backtrace Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-2.68 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.976]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::836:from]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_COUNT_THREE(0.00)[3]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PQR3h1xCQz3mPm X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Sun, Feb 26, 2023 at 02:22:08PM +0800, Zhenlei Huang wrote: > > On Feb 24, 2023, at 11:34 PM, Mark Johnston wrote: > >>> Memory modified after free 0xfffffe00ccc29000(8184) val=0 @ 0xfffffe00ccc29698 > >>> panic: Most recently used by temp > >> > >>> cpuid = 0 > >>> time = 1677239728 > >>> KDB: stack backtrace: > >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0084e3eaa0 > >>> vpanic() at vpanic+0x152/frame 0xfffffe0084e3eaf0 > >>> panic() at panic+0x43/frame 0xfffffe0084e3eb50 > >>> mtrash_dtor() at mtrash_dtor/frame 0xfffffe0084e3eb70 > >>> item_ctor() at item_ctor+0x11f/frame 0xfffffe0084e3ebc0 > >>> malloc() at malloc+0x7f/frame 0xfffffe0084e3ec00 > >>> g_read_data() at g_read_data+0x82/frame 0xfffffe0084e3ec40 > >>> g_use_g_read_data() at g_use_g_read_data+0x46/frame 0xfffffe0084e3ec60 > >>> readsuper() at readsuper+0x29/frame 0xfffffe0084e3ecf0 > >>> ffs_sbget() at ffs_sbget+0x84/frame 0xfffffe0084e3ed70 > >>> g_label_ufs_taste_common() at g_label_ufs_taste_common+0x8b/frame 0xfffffe0084e3edc0 > >>> g_label_taste() at g_label_taste+0x1d0/frame 0xfffffe0084e3eea0 > >>> g_new_provider_event() at g_new_provider_event+0x9a/frame 0xfffffe0084e3eec0 > >>> g_run_events() at g_run_events+0x104/frame 0xfffffe0084e3eef0 > >>> fork_exit() at fork_exit+0x80/frame 0xfffffe0084e3ef30 > >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084e3ef30 > >>> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > >>> KDB: enter: panic > >> > >> The source code sys/vm/uma_dbg.c shows clearly that the panic comes from `mtrash_ctor()`. > >> > >> Why KDB shows that the panic is from `mtrash_dtor()` ? > > > > I couldn't reproduce this locally (i.e., the stack trace looks correct > > when the UAF is triggered), but the problem is a bit clearer after > > grabbing a kernel from artifact.ci.freebsd.org . > > Maybe a hand-crafted kernel module which modify after free intensionally can reproduce this easily. > > > > > In mtrash_ctor(), the final instruction is a call to panic(): > > > > (kgdb) disas mtrash_ctor > > ... > > 0xffffffff80f766be <+110>: mov 0x10(%rax),%rsi > > 0xffffffff80f766c2 <+114>: mov $0xffffffff81200154,%rdi > > 0xffffffff80f766c9 <+121>: xor %eax,%eax > > 0xffffffff80f766cb <+123>: call 0xffffffff80bed350 > > (kgdb) > > > > This works because the compiler knows that panic() never returns. > > > > However, the return address saved on the stack will still point to the > > "next" instruction, which is now outside of the bounds of the > > mtrash_ctor symbol, and it happens to be the first instruction of > > mtrash_dtor(): > > > > (kgdb) x/2i 0xffffffff80f766cb > > > > 0xffffffff80f766cb : call 0xffffffff80bed350 > > 0xffffffff80f766d0 : push %rbp > > > > So DDB's stack unwinder reports the call as coming from mtrash_dtor() > > instead of mtrash_ctor(). > > Thanks for the detailed analyzation ! > > > > > I'm not sure how to fix this. Instead of resolving the symbol > > containing the return address, it could maybe resolve the symbol > > containing the previous instruction, but variable-length instructions > > make that tricky. > > I'd like to look at this issue when I have time. It was pointed out in private mail that there's no need to use an instruction boundary when resolving a text address. In fact, db_nextframe() already knows about this problem, and I believe the patch below is sufficient to fix the stack trace. But we need some way to test it. diff --git a/sys/amd64/amd64/db_trace.c b/sys/amd64/amd64/db_trace.c index c38f4f6a4860..70aa6c3acdd1 100644 --- a/sys/amd64/amd64/db_trace.c +++ b/sys/amd64/amd64/db_trace.c @@ -200,7 +200,10 @@ db_nextframe(struct amd64_frame **fp, db_addr_t *ip, struct thread *td) return; } - db_print_stack_entry(name, rip, &(*fp)->f_frame); + /* + * See the comment above the db_search_symbol() call. + */ + db_print_stack_entry(name, rip - 1, &(*fp)->f_frame); /* * Point to base of trapframe which is just above the