Porting KSE to a new architecture.. kernel part

[Julian Elischer]

David, Dan, Marcel, peter

Here is my first hack at a small doc to help 
$ARCH developers enable KSE threading on their architecture.


comments? additions?


There are several small functions that need to be rewritten
to allow KSE threading to run on a new architecture.

Firstly it is important to understand the extra interractions that
KSE threading has between the userland and the kernel.

When a thread blocks in the kernel, the (virtual) cpu is reallocated to
another newly created context (thread) and it is allowed to proceed back
to userland and runs a known routine with a known pointer as its
argument.

This is called an "upcall" and there need to be machine specific
functions to set it up.

The blocking thread is allowed to continue at a later time but since
the userland has already been returned to, It must not return there.
Instead, when it reaches the user boundary, its context is stored back
into the userland thread storage for that thread, in exactly the same
way that it would look as if it had returned from the kernel, and then
called 'yield'. The format of the  saved context is "struct mcontext"
and it should already be defined for each architecture.

**************** Implementation ******************
## The UPCALL ##

There are two Machine independent functions that do the upcall,
(It's done in two stages) and each calls a machine dependent
part to do such things as may need to be done for each architecture.

The machine INdependent functions are:
 thread_schedule_upcall()  and thread_userret()

Thread_schedule_upcall() calls cpu_set_upcall()
and 
thread_userret() calls cpu_set_upcall_kse()

Thread_schedule_upcall() sets up a new 'unused' Thread and 
requires that cpu_set_upcall() set up its context so that that thread,
when run will return towards the user boundary. It does this in almost 
the same manner that fork() sets up the child thread to do the same
thing.
 As a result, cpu_set_upcall() is almost the same in logic
as cpu_fork(). It even sets up the context so that
the next code run by the new thread is in fact fork_trampoline()
and the (new) thread proceeds towards the userland boundary 
pretty much believing that it has just done a fork().
The main difference is that the flag TDP_UPCALLING  is set in the
thread's flags.

In i386 this is about 14 lines of C and in ia64 it is about 18
lines of C.

When the thread reaches the user boundary it runs userret()
which in turn calls thread_userret() as mentionned above.
Thread_userret() notices that TDP_UPCALLING is set and take special
actions before allowing it to proceed to userland.
One of these actions is to load the userland context with a PC
(instruction pointer) for the upcall function for that
particular (virtual) cpu, as well as setting the stack pointer 
to the stack associated with it. It also fiddles the arguments on the
stack that the upcall function will see, setting the (single) argument
to the address of that Virtual CPU's mailbox.
This is all done by the function cpu_set_upcall_kse().

In i386 it is 4 lines of C and in ia64 it is about 14 lines of C.

## The context export to userland **

The blocking thread needs to be able to store its context back to
userland when it has completed its kernel task, and an Machine dependent 
function is required for that. The basic userret() calls
thread_userret() and that in turn calls thread_export_context().

Thread_export_context() in turn uses the machiine dependent function
get_mcontext to extract the saved context from the thread's stack
(or wherever it is stored) and write it in the proscribed 
format (struct mcontext), which is then written out to the
thread context storage area in userland.

It is possible that this may be rewritten to avoid the intermadiate copy
of the mcontext in kernel space and write it directly 
to userland, however this is not what is done at this time.

get_mcontext() has code that is much like that used to write signals to
userland and can largly be cribbed (in loginc if not in code) from
there (sendsig()).

## Other ##

KSE threads also need some small assistance with signals.
a machine dependent function cpu_thread_siginfo() is needed
to format the information being sent to the userland scheduler
into the correct format for that architecture.