Re: amd64 syscall ABI (vs. Darwin)

On Mon, Jan 17, 2022 at 12:41:59PM +0000, Damian Malarczyk wrote:
> Hello,
> 
> I'm hacking on a toy project to run Darwin (MachO) binaries on FreeBSD.
> Currently I'm at a stage of syscalls support, and I've noticed a difference in the amd64 ABI that I didn't expect.
> 
> FreeBSD is changing values of some registers that aren't used as the syscall output. e.g., r8-r11 are changed, while r12-r15 don't seem to be affected.
> That's not the case on Darwin, from what I've seen onlyrax, rdx used as syscall results are changed.
> It looks like FreeBSD's syscalls calling convention is more like standard function calling, and r8-r11 should be always caller saved.
It is not 'more like'. FreeBSD follows C ABI for amd64 for syscall
registers handling. An additional twist is that the registers which are
declared as calleee-clobered are zeroed to avoid kernel data leakage to
userspace.

> 
> At a first glance Darwin approach seems more optimal, as less registers get clobbered. Is there any specific reason why this isn't also the case on FreeBSD?
> I'm also wondering where exactly the register values are changed. When I look at thetrapframe contents in the sv_set_syscall_retvalsystem vector callback the r8 register value is same as on the input, so it must be changed somewhere later. Does anyone know where exactly this happens?

Look at the sys/amd64/amd64/exceptions.S.  The fast_syscall entry point
is where we receive control after the syscall instruction.

> 
> Thanks in advance for any tips.
> 
> Here're the programs I used to test this behaviour:
> - [FreeBSD](https://gist.github.com/dmcyk/11c29b2d5e5d3e04e5b954e43e12d384)
> - [macOS](https://gist.github.com/dmcyk/ed1c6fcced78844c8e2e4a0fb3d18391)
> 
> When you run the macOS version it wil write twice the number of arguments to stdout, FreeBSD will write the number only once followed by a 0, because r8 got overwritten.
> 
> P.S. I'm relatively new to FreeBSD, and first time writing here on the mailing list so hello everyone :).
> 
> - Damian