amd64 syscall ABI (vs. Darwin)

From: Damian Malarczyk <damian_at_dmcyk.xyz>
Date: Mon, 17 Jan 2022 12:41:59 UTC
Hello,

I'm hacking on a toy project to run Darwin (MachO) binaries on FreeBSD.
Currently I'm at a stage of syscalls support, and I've noticed a difference in the amd64 ABI that I didn't expect.

FreeBSD is changing values of some registers that aren't used as the syscall output. e.g., r8-r11 are changed, while r12-r15 don't seem to be affected.
That's not the case on Darwin, from what I've seen onlyrax, rdx used as syscall results are changed.
It looks like FreeBSD's syscalls calling convention is more like standard function calling, and r8-r11 should be always caller saved.

At a first glance Darwin approach seems more optimal, as less registers get clobbered. Is there any specific reason why this isn't also the case on FreeBSD?
I'm also wondering where exactly the register values are changed. When I look at thetrapframe contents in the sv_set_syscall_retvalsystem vector callback the r8 register value is same as on the input, so it must be changed somewhere later. Does anyone know where exactly this happens?

Thanks in advance for any tips.

Here're the programs I used to test this behaviour:
- [FreeBSD](https://gist.github.com/dmcyk/11c29b2d5e5d3e04e5b954e43e12d384)
- [macOS](https://gist.github.com/dmcyk/ed1c6fcced78844c8e2e4a0fb3d18391)

When you run the macOS version it wil write twice the number of arguments to stdout, FreeBSD will write the number only once followed by a 0, because r8 got overwritten.

P.S. I'm relatively new to FreeBSD, and first time writing here on the mailing list so hello everyone :).

- Damian