Linux compat ioctl return values

[Roman Divacky]

On Wed, Apr 30, 2008 at 06:00:43PM +0200, Pieter de Boer wrote:
> Hi,
> 
> I've been working on a kernel driver that creates a device. This device
> in turn is opened and ioctl'd from a Linux executable. I've registered a
> handler for these ioctl's and my ioctl handler is succesfully executed.
> 
> My ioctl-handler returns a large positive value, but the userland
> application retrieves the value 1, EPERM. If I return 42, the userland
> application retrieves 42, but 260 is retrieved as 1. It appears there's a
> threshold somewhere above which the return value is set to 1, but I
> haven't been able to find out where in the code this is done. The Linux
> executable actually expects the value I return, and doesn't work when
> EPERM is found instead.
> 
> So, the question is: does anyone know where such a threshold may
> reside and how to work around it?

this is done in (for i386) sys/i386/i386/trap.c around line 1050.

in short, we define in the sysvec structure sv_errtbl and if returned
error > the size of the table we just return -1. error table for
linux is roughly to 70. thats why you are getting -1 (1 after translation)

you might extend the errno table (i386/linux/linux_sysvec.c for i386, line 126)

if you provide (tested!) patch for i386/amd64, I am sure it will get commited

roman