vnet: acessing module's virtualized global variables from another module

[Mikolaj Golub]

Hi,

Trying ipfw_nat under VIMAGE kernel I got this panic on the module load:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x4
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc09f098e
stack pointer           = 0x28:0xf563b944
frame pointer           = 0x28:0xf563b998
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 4264 (kldload)

witness_checkorder(c6d5e91c,9,ca0ac2e3,223,0,...) at witness_checkorder+0x6e
_rw_wlock(c6d5e91c,ca0ac2e3,223,0,c0e8f795,...) at _rw_wlock+0x82
ipfw_nat_modevent(c98a48c0,0,0,75,0,...) at ipfw_nat_modevent+0x41
module_register_init(ca0ad508,0,c0e8d834,e6,0,...) at module_register_init+0xa7
linker_load_module(0,f563bc18,c0e8d834,3fc,f563bc28,...) at linker_load_module+0xa05
kern_kldload(c86835c0,c72d3400,f563bc40,0,c8d0d000,...) at kern_kldload+0x133
kldload(c86835c0,f563bcec,c09e8940,c86835c0,0,...) at kldload+0x74
syscallenter(c86835c0,f563bce4,c0ce05dd,c1022150,0,...) at syscallenter+0x263
syscall(f563bd28) at syscall+0x34
Xint0x80_syscall() at Xint0x80_syscall+0x21
--- syscall (304, FreeBSD ELF32, kldload), eip = 0x280da00b, esp = 0xbfbfe79c, ebp = 0xbfbfec88 -

It crashed on acessing data from virtualized global variable V_layer3_chain in
ipfw_nat_modevent(). V_layer3_chain is defined in ipfw module and it turns out
that &V_layer3_chain returns wrong location from anywhere but ipfw.ko.

May be this is a known issue, but I have not found info about this, so below
are details of investigation why this happens.

Virtualized global variables are defined using the VNET_DEFINE() macro, which
places them in the 'set_vnet' linker set (in the base kernel or in
module). This is used to

1) copy these "default" values to each virtual network stack instance when created;

2) act as unique global names by which the variable can be referred to. The
location of a per-virtual instance variable is calculated at run-time like in
the example below for layer3_chain variable in the default vnet (vnet0):

vnet0->vnet_data_base + (uintptr_t) & vnet_entry_layer3_chain		(1)

For modules the thing is more complicated. When a module is loaded its global
variables from 'set_vnet' linker set are copied to the kernel 'set_vnet', and
for module to be able to access them the linker reallocates all references
accordingly (kern/link_elf.c:elf_relocaddr()):

	if (x >= ef->vnet_start && x < ef->vnet_stop)
		return ((x - ef->vnet_start) + ef->vnet_base);

So from inside the module the access to its virtualized variables works, but
from the outside we get wrong location using calculation like above (1),
because &vnet_entry_layer3_chain returns address of the variable in the
module's 'set_vnet'.

The workaround is to compile such modules into the kernel or use a hack I have
done for ipfw_nat -- add the function to ipfw module which returns the
location of virtualized layer3_chain variable and use this location instead of
V_layer3_chain macro (see the attached patch).

But I suppose the problem is not a new and there might be better approach
already invented to deal with this?

-- 
Mikolaj Golub
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipfw_nat.patch
Type: text/x-patch
Size: 3451 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20110509/eede117f/ipfw_nat-0001.bin