Sharing memory between ithread and userland

[Peter Jeremy]

I have a device driver that uses a mmap'd region to share status
information between the kernel driver and the userland application
and am finding that some updates made by the interrupt handler
don't seem to be visible to the application.  I can't see anything
I am doing wrong and hope that someone here can tell me what I've
done wrong.  I've tried disassembling both functions in question
and as far as I can tell the machine code matches my expectations.

The system is a single-CPU Athlon-XP running 5.3-RELEASEp4.

Looking at the following code, my application is receiving SIGUSR1
signals with ev1ctr incrementing as expected but status, event1 and
ev1sts all report no event has occurred.  The signals are occurring at
the expected rate and the actual data (which arrives via another
mmap'd region) also looks correct (if I ignore event1 and just rely on
ev1ctr, everything seems to work correctly).  I'd suspect that
something else was corrupting memory but is seems odd that the
corruption affects non-adjacent words and always sets them to zero.

Can anyone offer any suggestions?

The shared area looks like:

struct dev_status {
	uint32_t	event1;
	uint32_t	event2;
	uint32_t	event3;
...
	uint32_t	ev1ctr;
...
	uint32_t	status;
...
	uint32_t	ev1sts;
...
};

My kernel interrupt handler looks like:

static void dev_intr(void *arg)
{
	struct dev_softc *sc = (struct dev_softc *)arg;
	struct dev_status *sts = sc->sc_sts;
	uint32_t irep, r;
	struct proc *p;

	irep = DEVICE_READ(INTERRUPT_REPORT);
	sts->ssts_irep = irep;

	sts->event1 = 0;
	sts->event2 = 0;
	sts->event3 = 0;

	if (irep & EVENT3)
		sts->event3 = 1;
	if (irep & EVENT2) {
		sts->event2 = 1;
		....
	}
	if (irep & EVENT1) {
		r = DEVICE_READ(EVENT1_STATUS);
		sts->ev1sts = r;
		if (r & EVENT1_READY) {
			sts->event1 = 1;
			....
			sts->ev1ctr++;
		}
	}

	if (sts->event1 || sts->event2 || sts->event3) {
		p = pfind(sc->sc_sigpid)) != NULL) {
		psignal(p, SIGUSR1);
		PROC_UNLOCK(p);
	}
}

The mmap handler looks like:

dev_mmap(DPTYPE dev, vm_offset_t offset, vm_paddr_t *paddr, int nprot)
{
        struct dev_softc *sc;

        sc = devclass_get_softc(dev_devclass, minor(dev));

        if (offset == MAGIC_OFFSET) {
                *paddr = vtophys(sc->sc_sts);
                return 0;
        }
	...
}

And my userland code looks like:

volatile struct dev_status	*sts;

	sts = mmap(NULL, STATUS_SIZE, PROT_READ, MAP_SHARED, device_fd, MAGIC_OFFSET);

void handle_sigusr1(int sig)
{
	static uint32_t		buf1ctr;

	if (sts->event1) {
		while (bufctr < sts->ev1ctr) {
			...
			bufctr++;
		}
	}
	if (sts->event2) {
		...
	}
	if (sts->event3) {
		...
	}
}

The following is a typical output from debugging code at the beginning
of handle_sigusr1() - though the output is not identical from run to run:
      buf1ctr     ev1ctr      event1    status
vptr:0x00000000 0x00000001 0x00000000 0x00000000
vptr:0x00000000 0x00000002 0x00000000 0x00000000
vptr:0x00000000 0x00000003 0x00000000 0x00000000
vptr:0x00000000 0x00000004 0x00000000 0x00000000
vptr:0x00000000 0x00000005 0x00000000 0x00000000
vptr:0x00000000 0x00000006 0x00000000 0x00000000
vptr:0x00000000 0x00000007 0x00000000 0x00000000
vptr:0x00000000 0x00000008 0x00000000 0x00000000
vptr:0x00000000 0x00000009 0x00000000 0x00000000
vptr:0x00000000 0x0000000A 0x00000001 0x00000001
vptr:0x0000000B 0x0000000B 0x00000000 0x00000000
vptr:0x0000000B 0x0000000C 0x00000000 0x00000000
vptr:0x0000000B 0x0000000D 0x00000001 0x00000001
vptr:0x0000000D 0x0000000E 0x00000000 0x00000000
vptr:0x0000000D 0x0000000F 0x00000001 0x00000001
vptr:0x0000000F 0x00000010 0x00000001 0x00000001
vptr:0x00000010 0x00000011 0x00000000 0x00000000
vptr:0x00000010 0x00000012 0x00000000 0x00000000
vptr:0x00000010 0x00000013 0x00000000 0x00000000
vptr:0x00000010 0x00000014 0x00000000 0x00000000
vptr:0x00000010 0x00000015 0x00000000 0x00000000
vptr:0x00000010 0x00000016 0x00000001 0x00000001
vptr:0x00000016 0x00000017 0x00000000 0x00000000
vptr:0x00000016 0x00000018 0x00000000 0x00000000
vptr:0x00000016 0x00000019 0x00000001 0x00000001
vptr:0x00000019 0x0000001A 0x00000001 0x00000001
vptr:0x0000001A 0x0000001B 0x00000001 0x00000001
vptr:0x0000001B 0x0000001C 0x00000001 0x00000001
vptr:0x0000001C 0x0000001D 0x00000001 0x00000001
vptr:0x0000001D 0x0000001E 0x00000001 0x00000001
vptr:0x0000001E 0x0000001F 0x00000001 0x00000001
vptr:0x0000001F 0x00000020 0x00000001 0x00000001
vptr:0x00000020 0x00000021 0x00000001 0x00000001
vptr:0x00000021 0x00000022 0x00000001 0x00000001

-- 
Peter Jeremy