two proposed linuxulator fixes + ptrace

[Yanko Yankulov]

 Hi all,

I stumbled on two problems trying to run a proprietary java/native Linux
app on FreeBSD. I managed to get it working with two small changes to the
Linux compatibility code. Also in order to pinpoint the issues I got the
ptrace system working for Linux apps, enabling truss & gdb.

This is all on recent CURRENT.

The patches work for me, but I am absolutely sure there is better way to do
this, especially for the ptrace part.

The first issue was 100% CPU usage on start with no progress. Turned out
that the app was expecting TracerPid field in its /proc/pid/status, so just
adding  "sbuf_printf(sb, "TracerPid:\t%d\n",     0);" in
linprocfs_doprocstatus solved it.

The second was random segfaults in the Java code. Traced it to mangled r10
in the sigreturn path. Just preserving the r10 when returning from
sigreturn resolves it.

--- a/sys/amd64/linux/linux_sysvec.c
+++ b/sys/amd64/linux/linux_sysvec.c
@@ -228,8 +228,9 @@ linux_set_syscall_retval(struct thread *td, int error)
         * the syscall.  So, do not clobber %rdx and %r10.
         */
        td->td_retval[1] = frame->tf_rdx;
-       frame->tf_r10 = frame->tf_rcx;
-
+       if( td->td_sa.code != LINUX_SYS_linux_rt_sigreturn )
+               frame->tf_r10 = frame->tf_rcx;
+
        cpu_set_syscall_retval(td, error);

         /* Restore all registers. */


So this two fixes solved my issues and was able to run/use the problematic
application.

The ptrace code is lot more messy, and I am really not happy with it, but I
didn't have the time to figure out a cleaner  solution. Attaching it
though, as it at least might help someone to devise a better fix.

Best,
Yanko
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-linprocfs-return-0-TracerPid.patch
Type: text/x-patch
Size: 835 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-emulation/attachments/20180614/7dba09c2/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-linuxlator-enable-ptrace.patch
Type: text/x-patch
Size: 3343 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-emulation/attachments/20180614/7dba09c2/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-linuxlator-amd64-don-t-mangle-r10-on-sigreturn.patch
Type: text/x-patch
Size: 894 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-emulation/attachments/20180614/7dba09c2/attachment-0002.bin>