Supporting cross-debugging vmcores in libkvm

[John-Mark Gurney]

John Baldwin wrote this message on Tue, Aug 04, 2015 at 10:56 -0700:
> Many debuggers (recent gdb and lldb) support cross-architecture debugging 
> just fine.  My current WIP port of kgdb to gdb7 supports cross-debugging for
> remote targets already, but I wanted it to also support cross-debugging for
> vmcores.
> 
> The existing libkvm/kgdb code in the tree has some limited support for
> cross-debugging.  It requires building a custom libkvm (e.g. libkvm-i386.a)
> and custom kgdb for each target platform.  However, gdb (and lldb) both
> support multiple targets in a single binary, so I'd like to have a single
> kgdb binary that can cross-debug anything.
> 
> I started hacking on libkvm last weekend and have a prototype that I've used
> (along with some patches to my kgdb port) to debug an amd64 vmcore on an
> i386 machine and vice versa.
> 
> To do this I've made some additions to the libkvm API:
> 
> 1) A new 'kvaddr_t' type represents a kernel virtual address.  This is
>    similar to the psaddr_t type used for MI process addresses in userland
>    debugging.  I almost reused psaddr_t directly, but that would have made
>    <libkvm.h> depend on <sys/procfs.h>.  Instead, I opted for a separate
>    type.  It is currently a uint64_t.

I like this..  W/ the work Lovasko has been working on, having this
type is good and makes the most sense...

> 2) A new 'struct kvm_nlist'.  This is a stripped-down version of
>    'struct nlist' that uses kvadd_t for n_value instead of an unsigned
>    long.
> 
> 3) kvm_native() returns true if an open kvm descriptor is for a native
>    kernel and memory image.
> 
> 4) kvm_nlist2() is like kvm_nlist() but it uses 'struct kvm_nlist'
>    instead of 'struct nlist'.  Internally symbol names are always
>    resolved to kvaddr_t addresses rather than u_long addresses.
>    Native kernels still use _fdnlist() from libc to resolve symbols.
>    Cross kernels use a caller supplied function to resolve symbols
>    (the older cross code for libkvm required the caller to provide
>    a global ps_pglobal_lookup symbol typically provided for
>    <proc_service.h>).
> 
> 5) kvm_open2() is like kvm_openfiles() except that it drops the unused
>    'swapfile' argument and adds a new function pointer argument to a
>    symbol resolving function.  The function pointer can be NULL in
>    which case only native kernels can be opened.  Kernels used with
>    /dev/mem or /dev/kmem must be native.
> 
> 6) kvm_read2() is like kvm_read() except that it uses kvaddr_t
>    instead of unsigned long for the kernel virtual address.

All the above looks good...

How will we prevent native only aware apps from getting confused when
accessing non-native cores?  Will kvm_openfiles fail for non-native
cores? or will kvm_read fail for non-native cores?

> Adding new symbols (specifically kvm_nlist2 and kvm_read2) preserves
> ABI and API compatibility.  Note that most libkvm functions such as
> kvm_getprocs(), etc. only work with native kernels.  I have not yet
> done a full sweep to force them to fail for non-native kernels.

For things like getprocs, we could move to using Lovasko's project
to use ctf data to read the proc structures for non-native cores...
Then the core parts of getprocs would not have to change, and it'd
just work in all places..

> Also, the vnet and dpcpu stuff only works for native kernels currently
> though that can be fixed at some point in the future.
> 
> For the MD backends, I've added a new kvm_arch switch:
> 
> struct kvm_arch {
> 	int	(*ka_probe)(kvm_t *);
> 	int	(*ka_initvtop)(kvm_t *);
> 	void	(*ka_freevtop)(kvm_t *);
> 	int	(*ka_kvatop)(kvm_t *, kvaddr_t, off_t *);
> 	int	(*ka_uvatop)(kvm_t *, const struct proc *, kvaddr_t, off_t *);
> 	int	ka_native;
> };
> 
> Each backend implements the necessary callbacks (uvatop is optional)
> and is added to a global linker set that kvm_open2() walks to find the
> appropriate kvm_arch for a given kernel + vmcore.  On x86 I've used
> separate kvm_arch structures for "plain" vs minidumps.
> 
> The backends now have to avoid using native headers.  For ELF handling
> this means using libelf instead of <machine/elf.h> and raw mmap().  For
> the x86 backends it meant defining some duplicate constants for certain
> page table fields since <machine/pmap.h> can't be relied on (e.g.
> I386_PG_V instead of PG_V).  I added static assertions in the "native"
> case (e.g. building kvm_i386.c on i386) to ensure the duplicate constants
> match the originals.
> 
> You can see the current WIP patches here:
> 
> https://github.com/freebsd/freebsd/compare/master...bsdjhb:kgdb_enhancements
> 
> What I'm mostly after is comments on the API, etc.  Once that is settled I
> will move forward on converting and/or stubbing the other backends (the
> stub route would be to only support other backends on native systems for
> now).

You're API looks good...  I can't see anything wrong w/ it...

> Oh, and I do hope to have a 'KGDB' option for the devel/gdb port in the
> near future.

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."