Re: amd64 syscall ABI (vs. Darwin)

On Mon, 17 Jan 2022 12:41:59 +0000
Damian Malarczyk <damian@dmcyk.xyz> wrote:

> Hello,
> 
> I'm hacking on a toy project to run Darwin (MachO) binaries on FreeBSD.
> Currently I'm at a stage of syscalls support, and I've noticed a difference in the amd64 ABI that I didn't expect.
> 
> FreeBSD is changing values of some registers that aren't used as the syscall output. e.g., r8-r11 are changed, while r12-r15 don't seem to be affected.
> That's not the case on Darwin, from what I've seen onlyrax, rdx used as syscall results are changed.
> It looks like FreeBSD's syscalls calling convention is more like standard function calling, and r8-r11 should be always caller saved.
> 
> At a first glance Darwin approach seems more optimal, as less registers get clobbered. Is there any specific reason why this isn't also the case on FreeBSD?
> I'm also wondering where exactly the register values are changed. When I look at thetrapframe contents in the sv_set_syscall_retvalsystem vector callback the r8 register value is same as on the input, so it must be changed somewhere later. Does anyone know where exactly this happens?
> 
> Thanks in advance for any tips.
> 

This happens in the tree in various assembler files.  The primary use
of these registers appears to be for holding temporary data.

Try running this in /usr/src:

find -type f -name "*.S" -print0 | xargs -0 grep -e r8 -e r9 -e r10 -e r11 \
--mmap -l $1

> Here're the programs I used to test this behaviour:
> - [FreeBSD](https://gist.github.com/dmcyk/11c29b2d5e5d3e04e5b954e43e12d384)
> - [macOS](https://gist.github.com/dmcyk/ed1c6fcced78844c8e2e4a0fb3d18391)
> 
> When you run the macOS version it wil write twice the number of arguments to stdout, FreeBSD will write the number only once followed by a 0, because r8 got overwritten.
> 
> P.S. I'm relatively new to FreeBSD, and first time writing here on the mailing list so hello everyone :).
> 
> - Damian

-- 
Gary Jennejohn