Kernel linker and undefined references in KLD

[Dmitry Krivenok]

Unfortunately, one can easily miss such problems during build of the module.
I'm working on a system which consists of lots of user-space programs
and kernel modules and
uses it's own complicated build system.
gcc option -Werror is not used by the build system (I believe it
should), that's why we just missed
a compiler warning when called undeclared function (actually declared
as static deep in kernel source).

Then we got kernel panic and started investigating it.
I disassembled seltdinit function and found that it takes pointer to
thread from %eax register and global kernel
functions (e.g. kern_select) calling seltdinit copy the pointer to
%eax just before "call" instruction.

Then I disassembled my own function calling seltdinit and found that
it passes the pointer via stack and
doesn't do anything with %eax register.

I'm not OS/compilers/ASM expert, but I guess that gcc is free to
optimize passing of parameters to static functions
because it has all the information about callers of these static
functions (all stuff is inside one translation unit).
So kernel functions call static functions in a right way, but modules
compiled separately may use wrong calling
convention.

On Thu, Jul 15, 2010 at 6:32 PM, Kostik Belousov <kostikbel at gmail.com> wrote:
> On Thu, Jul 15, 2010 at 06:07:36PM +0400, Dmitry Krivenok wrote:
>> Hello Hackers,
>>
>> I have a question about kernel linker.
>> Please take a look at an example of simple module:
>>
>> ############################################################################
>> #include <sys/param.h>
>> #include <sys/module.h>
>> #include <sys/kernel.h>
>> #include <sys/systm.h>
>> #include <sys/proc.h>
>> #include <sys/sched.h>
>> #include <sys/pcpu.h>
>>
>> /* DECLARING A FUNCTION JUST TO AVOID WARNING WHICH IS TREAT AS ERROR
>> BY DEFAULT */
>> void seltdinit(struct thread* t);
>>
>> static int event_handler(struct module *module, int event, void *arg)
>> {
>>   int e = 0;
>>   switch (event)
>>     {
>>       case MOD_LOAD:
>>         /* CALLING A FUNCTION DECLARED AS STATIC IN kern/sys_generic.c */
>>         seltdinit(curthread);
>>         break;
>>       case MOD_QUIESCE:
>>         break;
>>       case MOD_UNLOAD:
>>         break;
>>       case MOD_SHUTDOWN:
>>         break;
>>       default:
>>         e = EOPNOTSUPP;
>>         break;
>>     }
>>
>>   return(e);
>> };
>> ############################################################################
>>
>> As you can see, this module calls seltdinit function declared as
>> _static_ in kern/sys_generic.c file.
>> I added a declaration of seltdinit function w/o static just to avoid
>> compilation error since -Werror is used
>> by default in FreeBSD.
>>
>> Then I successfully built the module:
>>
>> $ make
>> Warning: Object directory not changed from original
>> /usr/home/krived/work/freebsd/trouble
>> cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE
>> -nostdinc   -I. -I@ -I@/contrib/altq -finline-limit=8000 --param
>> inline-unit-growth=100 --param large-function-growth=1000 -fno-common
>> -fno-omit-frame-pointer  -mcmodel=kernel -mno-red-zone  -mfpmath=387
>> -mno-sse -mno-sse2 -mno-sse3 -mno-mmx -mno-3dnow  -msoft-float
>> -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector
>> -std=iso9899:1999 -fstack-protector -Wall -Wredundant-decls
>> -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes
>> -Wpointer-arith -Winline -Wcast-qual  -Wundef -Wno-pointer-sign
>> -fformat-extensions -c trouble.c
>> ld  -d -warn-common -r -d -o trouble.ko trouble.o
>> :> export_syms
>> awk -f /sys/conf/kmod_syms.awk trouble.ko  export_syms | xargs -J%
>> objcopy % trouble.ko
>> objcopy --strip-debug trouble.ko
>> $
>>
>> As expected, seltdinit symbol is undefined in trouble.ko file:
>> $ nm trouble.ko | grep seltdinit
>>                  U seltdinit
>> $
>> and is local in kernel binary file:
>> $ nm /boot/kernel/kernel | grep seltdinit
>> ffffffff805fda50 t seltdinit
>> $
>>
>> I expected to get something like "undefined reference to seltdinit"
>> error, but the module was loaded w/o any errors:
>>
>> $ sudo make load
>> /sbin/kldload -v /usr/home/krived/work/freebsd/trouble/trouble.ko
>> Loaded /usr/home/krived/work/freebsd/trouble/trouble.ko, id=2
>> $ kldstat | grep trouble
>>  2    1 0xffffffff81212000 126      trouble.ko
>> $ sudo make unload
>> /sbin/kldunload -v trouble.ko
>> Unloading trouble.ko, id=2
>> $
>>
>> Could you please explain why the linker (or whatever is loading
>> modules in FreeBSD) doesn't complain?
>>
>> For example, ld tool complains for the similar user-space example:
>>
>> ############################################################################
>> $ cat seltdinit.c
>> static void seltdinit(void* td)
>> {
>>   return;
>> }
>> $ cat main.c
>> void seltdinit(void* td);
>>
>> int main()
>> {
>>   seltdinit(0);
>>   return 0;
>> }
>> $ gcc -Wall -c seltdinit.c
>> seltdinit.c:2: warning: 'seltdinit' defined but not used
>> $ gcc -Wall -c main.c
>> $ gcc seltdinit.o main.o -o sel
>> main.o: In function `main':
>> main.c:(.text+0xa): undefined reference to `seltdinit'
>> collect2: ld returned 1 exit status
>> $
>> ############################################################################
>>
>> Thanks in advance!
> The kernel linker ignores weak attribute of the symbol, as you see.
> There is more bugs in this department, in regard of the list of
> exported symbols from the modules.
>
> I have a patch that fixes the issues, but I am leery to commit it, since
> the fix effectively breaks significant set of the modules.
>



-- 
Sincerely yours, Dmitry V. Krivenok
e-mail: krivenok.dmitry at gmail.com
skype: krivenok_dmitry
jabber: krivenok_dmitry at jabber.ru
icq: 242-526-443