Linux compat ioctl return values

[Roman Divacky]

On Thu, May 01, 2008 at 11:57:42AM +0200, Pieter de Boer wrote:
> Roman Divacky wrote:
> 
> >>I've been working on a kernel driver that creates a device. This device
> >>in turn is opened and ioctl'd from a Linux executable. I've registered a
> >>handler for these ioctl's and my ioctl handler is succesfully executed.
> >>
> >>My ioctl-handler returns a large positive value, but the userland
> >>application retrieves the value 1, EPERM. If I return 42, the userland
> >>application retrieves 42, but 260 is retrieved as 1. It appears there's a
> >>threshold somewhere above which the return value is set to 1, but I
> >>haven't been able to find out where in the code this is done. The Linux
> >>executable actually expects the value I return, and doesn't work when
> >>EPERM is found instead.
> >>
> >>So, the question is: does anyone know where such a threshold may
> >>reside and how to work around it?
> >
> >this is done in (for i386) sys/i386/i386/trap.c around line 1050.
> >
> >in short, we define in the sysvec structure sv_errtbl and if returned
> >error > the size of the table we just return -1. error table for
> >linux is roughly to 70. thats why you are getting -1 (1 after translation)
> >
> >you might extend the errno table (i386/linux/linux_sysvec.c for i386, line 
> >126)
> 
> The issue appears to be a bit more involved. It seems that in Linux, 
> when the ioctl() syscall returns a negative value 'error', 'errno' is 
> set to '-error' and the return value of the ioctl() library call is -1. 
> All positive values are simply passed through: when the ioctl() syscall 
> returns 35235, the ioctl() library call also returns 35235.
> 
> This seems to be a difference in semantics between FreeBSD and Linux; 
> FreeBSD is a bit more conservative. As the trap code in 
> sys/i386/i386/trap.c is used for both FreeBSD and Linux executables, I 
> wonder how to differentiate between both in trap.c.
> 
> To see if I can at least make my Linux executable work for now, I'm 
> going to test the following patch (to trap.c):
> -  				error = -1;	/* XXX */
> +  				/* Do nothing */
> 
> I suppose a patch that differentiates between Linux and FreeBSD syscalls 
> is needed here, but how this could be done, dunno.

I dont think so.... native freebsd does NOT have errno translation table.

you patch makes sense I'd say