Multi-threading access to device drivers.

[Matthew Dillon]

:As was quoted POSIX, the update should be atomic..  so my reading of
:that is that you lock the fd, when you read the offset from the fd,
:you need to also update the offset with the number of bytes read, so
:that a racing read both doesn't drop any data, nor duplicate any data..
:
:i.e.  single file with fixed records, it should be perfectly safe to
:throw five threads looping doing a read processing of the records w/o
:fear of duplicate records, or dropped records...
:
:And if we handle it in this manner, then you can allow multiple reads
:into the driver...
:
:-- 
:  John-Mark Gurney				Voice: +1 415 225 5579

    If it's a regular file then the only requirements are for data atomicy 
    against writes, and no duplication of the data space (no missing pieces,
    no overlapping pieces).  This does not preclude the ability for N threads
    to all issue a read() on the same file descriptor simultaniously.  A
    simple seek space range lock (internal to the kernel) is all that is
    required to guarentee atomicy.  Serialization is not required.  e.g.

    read(fd, buf, bytes)
    {
	off_t offset;

	serializing_lock(fd);	/* temporarily protect fd->offset */
	offset = fd->offset;
	fd->offset += bytes;	(ignoring EOF issues for the moment)
	serializing_unlock(fd);
	range_lock_shared(fd, offset, bytes);
	[ ACTUAL READ INTO USER BUFFER ]
	range_unlock(fd, offset, bytes);
    }

    Or, alternatively, if you want I/O errors to also be atomic and the
    data fits in the cache, it would be something like:

    read(fd, buf, bytes)
    {
	off_t offset;
	int done = 0;

	while (!done) {
	    offset = fd->offset;
	    [ BRING DATA INTO THE CACHE ] (note: not locked at this point)
	    serializing_lock(fd);	/* temporarily protect fd->offset */
	    if (fd->offset == offset) {	/* check for race */
		fd->offset += bytes;
		done = 1;
	    }
	    serializing_unlock(fd);
	}
	range_lock_shared(fd, offset, bytes);
	[ ACTUAL READ INTO USER BUFFER ]
	range_unlock(fd, offset, bytes);
    }

    This pseudo code ignores overlapping atomicy issues if the user
    buffer happens to be mmap()'d from the same file, and ignores deadlock
    issues for same from a different file (A,B) <-> (B,A).  Generally,
    however, that sort of deadlock detection is really easy to implement,
    and would not interfere with the basic algorithm.

    Dataspace range locking is something we will be doing in DFly though the
    purpose is primarily to allow one process to block on I/O while another
    successfully reads or writes data on the same file via the VM cache
    without both being stalled.  It is not an SMP issue per-say, even though
    it can also be used to fix SMP based parallel access situations.  It's
    really a concurrency issue between blocked I/O and cached I/O that is
    one of the biggest reasons why our FS performance sucks compared to
    linux when reading and writing a single file (e.g. database).

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>