From nobody Sun Feb 26 06:22:08 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PPYSn6l2Qz3rxX4 for ; Sun, 26 Feb 2023 06:22:17 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PPYSn6T3Zz44QL; Sun, 26 Feb 2023 06:22:17 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677392537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=UvamiAbmoARLuCByXGbRV3akV/BMDK4xDkZ0MNFhXt4=; b=J8klzxsiGRv9InGsQJmqGPwvWW71K6TVEKMNRCwtfVdjy9BWrg6C7rhZms/rX+2F289SlX 0B3nGl2itCAumjIvoB0qc9CCacyWVgdhs1LbJyrIuToTn+LRv8O52Lj/cVidA4IS8f0ekr j24AnCpaT0Gcq2GqBX6k9VGEAeSj36RlhdTgEkYg3Uw1rZCdZ5lsuDkVxd2ET6nVtkSRfW 2KsZo01geG4Bbo0K2Z/Z+kW9MzUL78VrqoSW23NPoYuzoJc+lnzGFuP2fFImxyVX6xvXhk lFL9n5jvYB3jR1Q7g9axi3a6C+AHUgYMDCejCkqSk4jUTuMTZ03ZJQg6DIvoWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1677392537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=UvamiAbmoARLuCByXGbRV3akV/BMDK4xDkZ0MNFhXt4=; b=mRj/JxwCXJ1DQRm2JSySrhcRsLbdc3jE0Qa62d0zmcGU2WJEfMk3v7hcj8Zwzl91duFgCB qBurO+n9sI1MQ29qPxnxN/yVyQEjx1dDamBt+91AwlksH2Bznh6SZeEueKg/J/YbK/+cF8 a+iPW0ZbCUpz8LD6pHmB1GPmZUMXOQ6q5U6rhydE1Q8FCNv3UtRT/uQcBMdtZV6GkFL+Te i8VSM8ah5ZIxFtm3JD8G5+L00GBF5WO4zC5pnan+ayqK9YkZG6pNtBlFWhmEM+qdmWwT9V jRFjuAl7jqsz1bCd7AMwXFEVgrglRFTFtPM/GkWy0E+v1KiWKCVV3jsuylbozQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1677392537; a=rsa-sha256; cv=none; b=uhnHWklpk3sq5s8HgglCcemicL4M+uBqIbM2OVAd1j36QUJCJVZXO5d8TWV1Wk/fd1xlug 1tZgHom+OZ/k9u92sQPLAZ76PT/v78HLAd0WbfmZW+DAkZoyoUFnCJCj+rXvCFMS1CJUwR JpBs+ATeR0u2re+nFoIRzBqYuv3pllbez53xLbpsQCR6Ib41xehpPiMvRAdNY664WErr2x RkhNCZLxsLbbea83NtnGz9QgRoj+oT+cC9qS/GOrqXbRUt2SwtXOJfeIJFFRBAyHq3DraV DL6HSnR5GC+HoiY4YfIa4OjpqYxsxfQrlSY/4ghlGIckca+qAy3P62e8mFB02A== Received: from smtpclient.apple (unknown [IPv6:2001:19f0:6001:9db:98f0:9fe0:3545:10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PPYSm3CtZzTZD; Sun, 26 Feb 2023 06:22:16 +0000 (UTC) (envelope-from zlei@FreeBSD.org) From: Zhenlei Huang Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_F3C94DDD-62DF-4493-B69A-70E8A990BE3F" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\)) Subject: Re: Confused about the kernel stack backtrace Date: Sun, 26 Feb 2023 14:22:08 +0800 In-Reply-To: Cc: FBSD Developers , freebsd-hackers@freebsd.org To: Mark Johnston References: X-Mailer: Apple Mail (2.3696.120.41.1.2) X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_F3C94DDD-62DF-4493-B69A-70E8A990BE3F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Feb 24, 2023, at 11:34 PM, Mark Johnston wrote: >=20 > On Fri, Feb 24, 2023 at 09:26:28PM +0800, Zhenlei Huang wrote: >> Hi, >>=20 >> The job FreeBSD-main-amd64-test on ci is failing, and some kernel = stack backtrace [1] >> looks weird. >=20 > BTW, it is preferred to use public mailing lists for this kind of > question. Also CC freebsd-hackers . >=20 >>> Memory modified after free 0xfffffe00ccc29000(8184) val=3D0 @ = 0xfffffe00ccc29698 >>> panic: Most recently used by temp >>=20 >>> cpuid =3D 0 >>> time =3D 1677239728 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame = 0xfffffe0084e3eaa0 >>> vpanic() at vpanic+0x152/frame 0xfffffe0084e3eaf0 >>> panic() at panic+0x43/frame 0xfffffe0084e3eb50 >>> mtrash_dtor() at mtrash_dtor/frame 0xfffffe0084e3eb70 >>> item_ctor() at item_ctor+0x11f/frame 0xfffffe0084e3ebc0 >>> malloc() at malloc+0x7f/frame 0xfffffe0084e3ec00 >>> g_read_data() at g_read_data+0x82/frame 0xfffffe0084e3ec40 >>> g_use_g_read_data() at g_use_g_read_data+0x46/frame = 0xfffffe0084e3ec60 >>> readsuper() at readsuper+0x29/frame 0xfffffe0084e3ecf0 >>> ffs_sbget() at ffs_sbget+0x84/frame 0xfffffe0084e3ed70 >>> g_label_ufs_taste_common() at g_label_ufs_taste_common+0x8b/frame = 0xfffffe0084e3edc0 >>> g_label_taste() at g_label_taste+0x1d0/frame 0xfffffe0084e3eea0 >>> g_new_provider_event() at g_new_provider_event+0x9a/frame = 0xfffffe0084e3eec0 >>> g_run_events() at g_run_events+0x104/frame 0xfffffe0084e3eef0 >>> fork_exit() at fork_exit+0x80/frame 0xfffffe0084e3ef30 >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084e3ef30 >>> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- >>> KDB: enter: panic >>=20 >> The source code sys/vm/uma_dbg.c shows clearly that the panic comes = from `mtrash_ctor()`. >>=20 >> Why KDB shows that the panic is from `mtrash_dtor()` ? >=20 > I couldn't reproduce this locally (i.e., the stack trace looks correct > when the UAF is triggered), but the problem is a bit clearer after > grabbing a kernel from artifact.ci.freebsd.org = . Maybe a hand-crafted kernel module which modify after free intensionally = can reproduce this easily. >=20 > In mtrash_ctor(), the final instruction is a call to panic(): >=20 > (kgdb) disas mtrash_ctor > ... > 0xffffffff80f766be <+110>: mov 0x10(%rax),%rsi > 0xffffffff80f766c2 <+114>: mov $0xffffffff81200154,%rdi > 0xffffffff80f766c9 <+121>: xor %eax,%eax > 0xffffffff80f766cb <+123>: call 0xffffffff80bed350 > (kgdb) >=20 > This works because the compiler knows that panic() never returns. >=20 > However, the return address saved on the stack will still point to the > "next" instruction, which is now outside of the bounds of the > mtrash_ctor symbol, and it happens to be the first instruction of > mtrash_dtor(): >=20 > (kgdb) x/2i 0xffffffff80f766cb >=20 > 0xffffffff80f766cb : call = 0xffffffff80bed350 > 0xffffffff80f766d0 : push %rbp >=20 > So DDB's stack unwinder reports the call as coming from mtrash_dtor() > instead of mtrash_ctor(). Thanks for the detailed analyzation ! >=20 > I'm not sure how to fix this. Instead of resolving the symbol > containing the return address, it could maybe resolve the symbol > containing the previous instruction, but variable-length instructions > make that tricky. I'd like to look at this issue when I have time. Best regards, Zhenlei --Apple-Mail=_F3C94DDD-62DF-4493-B69A-70E8A990BE3F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Feb 24, 2023, at 11:34 PM, Mark Johnston <markj@freebsd.org> = wrote:

On Fri, Feb 24, 2023 at 09:26:28PM +0800, Zhenlei Huang = wrote:
Hi,

The job FreeBSD-main-amd64-test on ci is = failing, and some kernel stack backtrace [1]
looks = weird.

BTW, it is preferred to use public mailing lists for this = kind of
question.

Also CC freebsd-hackers .


Memory modified after = free 0xfffffe00ccc29000(8184) val=3D0 @ 0xfffffe00ccc29698
panic: Most recently used by temp

cpuid =3D 0
time =3D 1677239728
KDB: stack backtrace:
db_trace_self_wrapper() = at db_trace_self_wrapper+0x2b/frame 0xfffffe0084e3eaa0
vpanic() at vpanic+0x152/frame 0xfffffe0084e3eaf0
panic() at panic+0x43/frame 0xfffffe0084e3eb50
mtrash_dtor() at mtrash_dtor/frame 0xfffffe0084e3eb70
item_ctor() at item_ctor+0x11f/frame 0xfffffe0084e3ebc0
malloc() at malloc+0x7f/frame 0xfffffe0084e3ec00
g_read_data() at g_read_data+0x82/frame 0xfffffe0084e3ec40
g_use_g_read_data() at g_use_g_read_data+0x46/frame = 0xfffffe0084e3ec60
readsuper() at readsuper+0x29/frame = 0xfffffe0084e3ecf0
ffs_sbget() at ffs_sbget+0x84/frame = 0xfffffe0084e3ed70
g_label_ufs_taste_common() at = g_label_ufs_taste_common+0x8b/frame 0xfffffe0084e3edc0
g_label_taste() at g_label_taste+0x1d0/frame = 0xfffffe0084e3eea0
g_new_provider_event() at = g_new_provider_event+0x9a/frame 0xfffffe0084e3eec0
g_run_events() at g_run_events+0x104/frame = 0xfffffe0084e3eef0
fork_exit() at fork_exit+0x80/frame = 0xfffffe0084e3ef30
fork_trampoline() at = fork_trampoline+0xe/frame 0xfffffe0084e3ef30
--- trap 0, = rip =3D 0, rsp =3D 0, rbp =3D 0 ---
KDB: enter: panic

The source code sys/vm/uma_dbg.c = shows clearly that the panic comes from `mtrash_ctor()`.
Why KDB shows that the panic is from `mtrash_dtor()` ?

I couldn't reproduce this locally (i.e., the stack trace = looks correct
when the UAF = is triggered), but the problem is a bit clearer after
grabbing a = kernel from artifact.ci.freebsd.org.

Maybe a = hand-crafted kernel module which modify after free intensionally can = reproduce  this easily.


In mtrash_ctor(), the final instruction is a call to = panic():

(kgdb) disas = mtrash_ctor
  ...
  0xffffffff80f766be <+110>:   mov =    0x10(%rax),%rsi
  0xffffffff80f766c2 <+114>:   mov =    $0xffffffff81200154,%rdi
  0xffffffff80f766c9 <+121>:   xor =    %eax,%eax
  0xffffffff80f766cb <+123>:   call =   0xffffffff80bed350 <panic>
(kgdb)

This works because the compiler knows that panic() never = returns.

However, the = return address saved on the stack will still point to the
"next" = instruction, which is now outside of the bounds of the
mtrash_ctor = symbol, and it happens to be the first instruction of
mtrash_dtor():

(kgdb) x/2i 0xffffffff80f766cb

  0xffffffff80f766cb <mtrash_ctor+123>: =        call =   0xffffffff80bed350 <panic>
  0xffffffff80f766d0 <mtrash_dtor>: =    push   %rbp

So DDB's stack unwinder reports the call as coming from = mtrash_dtor()
instead of = mtrash_ctor().

Thanks for the detailed analyzation !


I'm not sure = how to fix this.  Instead of resolving the symbol
containing = the return address, it could maybe resolve the symbol
containing = the previous instruction, but variable-length instructions
make that = tricky.

I'd like to = look at this issue when I have time.

Best regards,
Zhenlei

= --Apple-Mail=_F3C94DDD-62DF-4493-B69A-70E8A990BE3F--