Re: Looking for rationale for the minidump format

On Sun, 21 Nov 2021 at 09:42, Michał Górny <mgorny@moritz.systems> wrote:
>
> Hi, everyone.
>
> As part of the work contracted by the FreeBSD Foundation, I'm working
> on adding explicit minidump support to LLDB.  When discussing
> the options with upstream, I've been asked why FreeBSD created their own
> minidump format.
>
> I did a bit of digging but TBH all the rationale I could get was to
> create partial memory dumps.  However, unless I'm mistaken the ELF
> format is perfectly capable of that -- e.g. via creating an explicit
> segment for every continuous active region.

ELF can, indeed. One of my previous co-op students wrote a little tool
to convert a minidump to ELF, see https://reviews.freebsd.org/D19253.
That's what it does - creates a segment for each contiguous region. If
we ended up going this way we could integrate D19253 into savecore.

One issue is that originally ELF supported only a 16-bit segment
count, the extension to support greater came later on.

I think minidumps could also store userland pages, but without any
tooling to access them it's not that useful IMO.

Oh, I just found an old (2014!) thread where I discussed some of this
with Greg Clayton:

On Thu, 22 May 2014 at 13:43, Ed Maste <emaste@freebsd.org> wrote:
>
> Thanks for the insight Greg - a few comments:
>
> On 22 May 2014 12:39, Greg Clayton <gclayton@apple.com> wrote:
> > Core file kernel debugging:
> > - You shouldn't have to do too much to the current ProcessELFCore except help it to locate hint address for the list of shared libraries and the main kernel file in memory. This is done in:
>
> Note that kernel core files on FreeBSD aren't (currently) ELF files,
> and may be a different format on different architectures, so there's
> some extra work here.  Dumps are written in a contiguous blob
> (typically to swap) at the time of a crash, and then copied to the
> filesystem after the next reboot.
>
> The dump function for amd64/x86_64 is minidumpsys in
> minidump_machdep.c[1], and consists of:
>
> 1. Dump header (stripped during copy to filesystem)
> 2. Minidump header
> 3. Copy of kernel message buffer
> 4. Bitmap of pages present in the dump
> 5. Page directory pages
> 6. Pages themselves
> 7. Trailer
>
> All of this is abstracted by the libkvm library, which provide
> kvm_read and kvm_write functions which a VA address parameter.  I
> think this is the right way to introduce support for FreeBSD kernel
> cores at first, as I'll explain below.
>
> > Live kernel debugging:
> > - You will make a new process subclass like you were saying unless the GDB remote protocol is used to talk to the kernel?
>
> We currently have separate ways of interacting with the kernel for
> remote and local debugging.  For remote debugging the kernel has a
> built-in GDB protocol stub which is available over a serial or
> firewire interface, and this should mostly work with the existing LLDB
> support.  We do local debugging (examining threads, global objects,
> etc.) via /dev/mem, abstracted by the same kvm_read and kvm_write
> functions in libkvm.  Since local FreeBSD debugging will be done on a
> FreeBSD host (by definition), using libkvm makes sense.  And since
> we'll have that, using it also for cores makes sense to get started.
>
> Of course we'll want to allow debugging a FreeBSD kernel core on OS X
> or Linux too.  I think the way forward for us there is to create a
> utility in the FreeBSD base system that converts the minidump format
> to ELF, rather than teaching LLDB about our core formats in a
> cross-platform way.  This utility could be built-in to savecore(8),
> the tool which reads the dump out of a raw partition and writes to the
> filesystem.  There is also ongoing discussion about bringing a KDP
> target to the kernel for live debugging.
>
> > Extra credit:
> > - Get threads that are context switched in memory to show up. In the MacOSX kernel debugger we have the notion of 1 core we can talk to through the KDP interface which is the code that runs the debugger.
>
> We have code[2] for this already in kgdb, our GDB 6.1 port with kernel
> support, and we should be able to re-use most of it as is.  The kernel
> threads just appear as regular threads in kgdb:
>
> (kgdb) info threads
>   827 Thread 102126 (PID=28738: kgdb)  sched_switch
> (td=0xfffffe021882b000, newtd=0xfffffe0008392920, flags=<value
> optimized out>)
>     at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
>   826 Thread 102334 (PID=28737: sudo)  sched_switch
> (td=0xfffffe02d2669920, newtd=0xfffffe0008392920, flags=<value
> optimized out>)
>     at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
>   825 Thread 104699 (PID=28715: more)  sched_switch
> (td=0xfffffe049cec7920, newtd=0xfffffe0008390000, flags=<value
> optimized out>)
>     at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
> ...
>
> -Ed
>
> [1] http://svnweb.freebsd.org/base/head/sys/amd64/amd64/minidump_machdep.c?annotate=257216#l219
> [2] http://svnweb.freebsd.org/base/head/gnu/usr.bin/gdb/kgdb/kthr.c?annotate=246893