FIX: accessing module's virtualized global variables from another module

Mikolaj Golub trociny at freebsd.org
Sun Jul 24 08:31:14 UTC 2011


Hi,

Some time ago I reported about this problem (the thread "vnet: acessing
module's virtualized global variables from another module"). I got a reply
only from Marko but failed to convinced him that it was not issue with curvnet
context being not set properly.

I will try to explain the issue again, as this time I come with a patch that
might be a solution. Below are some things that may be obvious for people who
are involved in the subject. Please skip to the last paragraphs then :-).

So, kernel and modules have 'set_vnet' linker set and the virtualized global
variables (defined via VNET_DEFINE() macro) are placed in this set. This
serves two purposes:

1) When a new virtual network stack instance is allocated it is initialized
copying the globals from kernel 'set_vnet' linker set.

2) The variable name acts as unique global name by which the variable can be
referred to. The location of a per-virtual instance variable is calculated at
run-time using accessor macros (VNET_VNET, VNET, ...), like in the example
below for ifnet variable in the vnet0:

	vnet0->vnet_data_base + (uintptr_t) & vnet_entry_ifnet

When a vnet is created its vnet_data_base is set accordingly to make this
work.

This works for virtualized variables from kernel 'set_vnet' linker set, but
modules need additional work. When a module is loaded its global variables
from 'set_vnet' linker set are copied to the kernel 'set_vnet', which has
additional space ('modspace') reserved for this purpose. This makes (1) work.
And to make the formula (2) valid for the module's virtual global the linker
reallocates all its references according to this formula:

	(x - ef->vnet_start) + ef->vnet_base

(see kern/link_elf.c:elf_relocaddr()).

Here is an example. In the ipfw module layer3_chain virtual global is defined:

kopusha:~% nm /boot/kernel/ipfw.ko |grep -i 'set_vnet\|layer3_chain'
00010820 A __start_set_vnet
00010b0c A __stop_set_vnet
00010840 d vnet_entry_layer3_chain

So it has relative address 0x00010840. The module is loaded at 0x9337f000:

kopusha:~% kldstat|grep ipfw.ko
22    2 0x9337f000 11000    ipfw.ko

and vnet_entry_layer3_chain is at 0x9337f000 + 0x00010840 = 0x9338f840:

(kgdb) p &vnet_entry_layer3_chain
$1 = (struct ip_fw_chain *) 0x9338f840

But inside ipfw.ko, the linker relocated the 0x9338f840 references to
0x86dac580, otherwise VNET macros will access wrong location:

(kgdb) p *(struct ip_fw_chain *)(vnet0->vnet_data_base + (uintptr_t) &vnet_entry_layer3_chain)
Error accessing memory address 0x9908b2c0: Bad address.
(kgdb) p *(struct ip_fw_chain *)(vnet0->vnet_data_base + (uintptr_t) 0x86dac580)
$2 = {rules = 0x0, reap = 0x872d296c, default_rule = 0x937bdbd0, n_rules = -1823040548, 
  ...

So far so good. But what if the global defined in one module is accessed from
another module (as it is in the case of layer3_chain, which is accessed from
ipfw_nat)? The linker relocates references only for "local" globals (defined
in its own module), so inside ipfw_nat the global is accessed using unmodified
vnet_entry_layer3_chain location, thus by wrong address.

The fix could be to make the linker on a module load recognize "external"
virtual globals defined in previously loaded modules and relocate them
accordingly. For this we need set_vnet list, where the addresses of modules
'set_vnet' linker sets are stored in. The attach patch implements this and
works for me.

Also, could someone explain what is kern/link_elf_obj.c for? The patch does
not deal with this file, but I see the reloactions here similar to those in
link_elf.c. So I might need to do similar modifiactions in link_elf_obj.c too,
but I don't know where this code is used (it looks like it is not executed on
my box, so I don't know how to test it).

-- 
Mikolaj Golub

-------------- next part --------------
A non-text attachment was scrubbed...
Name: vnet.vnet_set.patch
Type: text/x-patch
Size: 5399 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20110724/0b2893df/vnet.vnet_set.bin


More information about the freebsd-virtualization mailing list