Linux compat ioctl return values

[Pieter de Boer]

Roman Divacky wrote:

>> I've been working on a kernel driver that creates a device. This device
>> in turn is opened and ioctl'd from a Linux executable. I've registered a
>> handler for these ioctl's and my ioctl handler is succesfully executed.
>>
>> My ioctl-handler returns a large positive value, but the userland
>> application retrieves the value 1, EPERM. If I return 42, the userland
>> application retrieves 42, but 260 is retrieved as 1. It appears there's a
>> threshold somewhere above which the return value is set to 1, but I
>> haven't been able to find out where in the code this is done. The Linux
>> executable actually expects the value I return, and doesn't work when
>> EPERM is found instead.
>>
>> So, the question is: does anyone know where such a threshold may
>> reside and how to work around it?
> 
> this is done in (for i386) sys/i386/i386/trap.c around line 1050.
> 
> in short, we define in the sysvec structure sv_errtbl and if returned
> error > the size of the table we just return -1. error table for
> linux is roughly to 70. thats why you are getting -1 (1 after translation)
> 
> you might extend the errno table (i386/linux/linux_sysvec.c for i386, line 126)

The issue appears to be a bit more involved. It seems that in Linux, 
when the ioctl() syscall returns a negative value 'error', 'errno' is 
set to '-error' and the return value of the ioctl() library call is -1. 
All positive values are simply passed through: when the ioctl() syscall 
returns 35235, the ioctl() library call also returns 35235.

This seems to be a difference in semantics between FreeBSD and Linux; 
FreeBSD is a bit more conservative. As the trap code in 
sys/i386/i386/trap.c is used for both FreeBSD and Linux executables, I 
wonder how to differentiate between both in trap.c.

To see if I can at least make my Linux executable work for now, I'm 
going to test the following patch (to trap.c):
-  				error = -1;	/* XXX */
+  				/* Do nothing */

I suppose a patch that differentiates between Linux and FreeBSD syscalls 
is needed here, but how this could be done, dunno.

-- 
Pieter