From nobody Mon Nov 22 15:26:34 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 182C8188C622 for ; Mon, 22 Nov 2021 15:27:08 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HyWNC1Zfhz4p1R; Mon, 22 Nov 2021 15:27:07 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-lf1-f42.google.com with SMTP id k37so82750819lfv.3; Mon, 22 Nov 2021 07:27:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=JM1vVQx6u1nrTF8N8gLce8FkOhqaGyoKvAE1u0iPZc4=; b=h00rzpavxiwzFY6nsJvwSFMiolMnzlLXC14h5019zHD0LB0yKvh8F3NOzbFDl+Htws SkGPehMZeKar88zbS5gDQaJrPQI+sP8VfbobTuZS98d+KUeK1S1ylzuEvQS+IiN8brer RJZE1+RmFrzTWoS9PdTzPP6pJbhC95ORVCnIEs5ibxnEdzgSxWi2Nhz1vbz4ibWDJH85 sDdOt9kD/xEEC7ZF5Waff1Lm4DCktK13y08xI7npkR4/utNTath/RTVWs6+NPR75W8iH jmJliYLyZhVKLVssljX6p6R7x5Uu+SG2TQ+fkb13KqAyaSj8sPEfppblhptEjNPE+jf7 iEIw== X-Gm-Message-State: AOAM531lZNPqVtypZ37NRfkuJp89pvICtBnGmDZ7AefDLIccoJAzz/CO F8AgrMYHP8ZVy9tkIZt5lKUycpeAjNNQinCGcCQxBadlqZw= X-Google-Smtp-Source: ABdhPJzgCgYCC5R4l+b7opGyBOFZthhy8rJ0lKk3Er6MUr0NpQUSA59ureKilf8ikCpnfGLdFE+4i37WX91qQPPvhas= X-Received: by 2002:a05:651c:a12:: with SMTP id k18mr54979026ljq.251.1637594819890; Mon, 22 Nov 2021 07:26:59 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <305082d9f216bb8382773c074eaf7a5c3101cc13.camel@moritz.systems> In-Reply-To: <305082d9f216bb8382773c074eaf7a5c3101cc13.camel@moritz.systems> From: Ed Maste Date: Mon, 22 Nov 2021 10:26:34 -0500 Message-ID: Subject: Re: Looking for rationale for the minidump format To: =?UTF-8?B?TWljaGHFgiBHw7Nybnk=?= Cc: FreeBSD Hackers , John Baldwin , Peter Wemm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4HyWNC1Zfhz4p1R X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.167.42 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [0.03 / 15.00]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEFALL_USER(0.00)[carpeddiem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; ARC_NA(0.00)[]; NEURAL_SPAM_MEDIUM(0.03)[0.033]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_SPAM_LONG(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[209.85.167.42:from]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.167.42:from]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com] X-ThisMailContainsUnwantedMimeParts: N On Sun, 21 Nov 2021 at 09:42, Micha=C5=82 G=C3=B3rny wrote: > > Hi, everyone. > > As part of the work contracted by the FreeBSD Foundation, I'm working > on adding explicit minidump support to LLDB. When discussing > the options with upstream, I've been asked why FreeBSD created their own > minidump format. > > I did a bit of digging but TBH all the rationale I could get was to > create partial memory dumps. However, unless I'm mistaken the ELF > format is perfectly capable of that -- e.g. via creating an explicit > segment for every continuous active region. ELF can, indeed. One of my previous co-op students wrote a little tool to convert a minidump to ELF, see https://reviews.freebsd.org/D19253. That's what it does - creates a segment for each contiguous region. If we ended up going this way we could integrate D19253 into savecore. One issue is that originally ELF supported only a 16-bit segment count, the extension to support greater came later on. I think minidumps could also store userland pages, but without any tooling to access them it's not that useful IMO. Oh, I just found an old (2014!) thread where I discussed some of this with Greg Clayton: On Thu, 22 May 2014 at 13:43, Ed Maste wrote: > > Thanks for the insight Greg - a few comments: > > On 22 May 2014 12:39, Greg Clayton wrote: > > Core file kernel debugging: > > - You shouldn't have to do too much to the current ProcessELFCore excep= t help it to locate hint address for the list of shared libraries and the m= ain kernel file in memory. This is done in: > > Note that kernel core files on FreeBSD aren't (currently) ELF files, > and may be a different format on different architectures, so there's > some extra work here. Dumps are written in a contiguous blob > (typically to swap) at the time of a crash, and then copied to the > filesystem after the next reboot. > > The dump function for amd64/x86_64 is minidumpsys in > minidump_machdep.c[1], and consists of: > > 1. Dump header (stripped during copy to filesystem) > 2. Minidump header > 3. Copy of kernel message buffer > 4. Bitmap of pages present in the dump > 5. Page directory pages > 6. Pages themselves > 7. Trailer > > All of this is abstracted by the libkvm library, which provide > kvm_read and kvm_write functions which a VA address parameter. I > think this is the right way to introduce support for FreeBSD kernel > cores at first, as I'll explain below. > > > Live kernel debugging: > > - You will make a new process subclass like you were saying unless the = GDB remote protocol is used to talk to the kernel? > > We currently have separate ways of interacting with the kernel for > remote and local debugging. For remote debugging the kernel has a > built-in GDB protocol stub which is available over a serial or > firewire interface, and this should mostly work with the existing LLDB > support. We do local debugging (examining threads, global objects, > etc.) via /dev/mem, abstracted by the same kvm_read and kvm_write > functions in libkvm. Since local FreeBSD debugging will be done on a > FreeBSD host (by definition), using libkvm makes sense. And since > we'll have that, using it also for cores makes sense to get started. > > Of course we'll want to allow debugging a FreeBSD kernel core on OS X > or Linux too. I think the way forward for us there is to create a > utility in the FreeBSD base system that converts the minidump format > to ELF, rather than teaching LLDB about our core formats in a > cross-platform way. This utility could be built-in to savecore(8), > the tool which reads the dump out of a raw partition and writes to the > filesystem. There is also ongoing discussion about bringing a KDP > target to the kernel for live debugging. > > > Extra credit: > > - Get threads that are context switched in memory to show up. In the Ma= cOSX kernel debugger we have the notion of 1 core we can talk to through th= e KDP interface which is the code that runs the debugger. > > We have code[2] for this already in kgdb, our GDB 6.1 port with kernel > support, and we should be able to re-use most of it as is. The kernel > threads just appear as regular threads in kgdb: > > (kgdb) info threads > 827 Thread 102126 (PID=3D28738: kgdb) sched_switch > (td=3D0xfffffe021882b000, newtd=3D0xfffffe0008392920, flags=3D optimized out>) > at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904 > 826 Thread 102334 (PID=3D28737: sudo) sched_switch > (td=3D0xfffffe02d2669920, newtd=3D0xfffffe0008392920, flags=3D optimized out>) > at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904 > 825 Thread 104699 (PID=3D28715: more) sched_switch > (td=3D0xfffffe049cec7920, newtd=3D0xfffffe0008390000, flags=3D optimized out>) > at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904 > ... > > -Ed > > [1] http://svnweb.freebsd.org/base/head/sys/amd64/amd64/minidump_machdep.= c?annotate=3D257216#l219 > [2] http://svnweb.freebsd.org/base/head/gnu/usr.bin/gdb/kgdb/kthr.c?annot= ate=3D246893