[Development report #4] Improve the kinst DTrace provider

From: Christos Margiolis <christos_at_freebsd.org>
Date: Mon, 06 Mar 2023 09:11:41 UTC
The past week I submitted a PR for the inline function tracing code I've
been working on [1], which fixes the `return` probe bug I mentioned in
my last report.

The approach for implementing return probes is roughly the following:

We figure out whether the DIE of the inline copy has the DW_AT_low_pc
and DW_AT_high_pc attributes, in which case the return address is
[DW_AT_lowpc + DW_AT_highpc], or if it has the DW_AT_ranges attribute
set, in which case we make return probes for the higher boundary of each
range.

But that's not exactly true because DWARF considers the return address
to be that *after* the last instruction:

	$ ./inlinecall cam_strvis_flag /usr/lib/debug/boot/kernel/kernel.debug
	/usr/src/sys/cam/cam.c:131
		[0xffffffff80353119 - 0xffffffff80353173]       /usr/src/sys/cam/cam.c:126      cam_strvis()
		[0xffffffff8035317e - 0xffffffff803531e1]       /usr/src/sys/cam/cam.c:126      cam_strvis()

If we look at what instructions 0xffffffff80353173 and
0xffffffff803531e1 correspond:

	(kgdb) disas /r cam_strvis
	   ...
	   0xffffffff8035316e <+110>:   call   0xffffffff80a8e2c0 <sbuf_finish>
	   0xffffffff80353173 <+115>:   add    $0x48,%rsp
	   ...
	   0xffffffff803531df <+223>:   jmp    0xffffffff8035319b <cam_strvis+155>
	   0xffffffff803531e1 <+225>:   data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1)

0xffffffff803531e1 in fact is outside cam_strvis() altogether. So to get
around this I use LibELF and dtrace_disx86() to go back one instruction
and put a probe there. The return offset then is calculated as
`inline_copy_retaddr - caller_func_entryaddr`, so for cam_strvis_flag(),
we would end up with the offsets 110 and 223:

	# dtrace -dn 'kinst::cam_strvis_flag:return'
	dtrace:::ERROR
	{
		((self->%error) = 0x1);
	}

	kinst::cam_strvis:110
	{
	}

	kinst::cam_strvis:223
	{
	}

	dtrace: description 'kinst::cam_strvis_flag:return' matched 3 probes
	...

So far this seems to be working fine, but I'll need to do more extensive
testing, as well as look into how gdb handles inline frames, to make
sure this mechanism is robust enough.

Christos

[1] https://reviews.freebsd.org/D38825