KSE/ia64: a quick update

[Marcel Moolenaar]

On Wed, Aug 06, 2003 at 03:28:52PM -0400, Daniel Eischen wrote:
> On Wed, 6 Aug 2003, Marcel Moolenaar wrote:
> 
> > Gang,
> > 
> > Given the panics that Daniel is having on pluto1 it's probably a
> > good idea to fill people in on the status of KSE/ia64:
> 
> First, thanks for your help on ia64.  As I've found out,
> there still seems to be a lot more that needs to get
> done (in general, not KSE specific), and you seem to
> be the only one working on it.

Yes, unfortunately. The amount of work will not reduce for
quite some time and it will only get worse before it gets
better.

> > The problem is when we preempt an interrupted thread, export the
> 
> You mean when an interrupt thread preempts an SA thread?  And
> this would be while it was in userland, not in the kernel?

I think so. I don't know exactly what the implications are for
SA threads, but in general the problem is that we have a trap
or interrupt while executing some thread and it results in an
upcall.

> > context to the UTS and do an upcall. We end up having an async.
> > context in userland. I'm not sure at this time what we should do
> > with it. We have the following options:
> > 
> > o  Extend _ia64_restore_context() so that libkse can restore async
> >    contexts. The downside is that it will very likely cause a
> >    disabled high FP trap, which results in the process having the
> >    high FP registers enabled. A performance hit. (see also below)
> 
> Having to know about 2 different contexts isn't too new.  Alpha
> may need the same thing since sigframes and trapframes are not
> the same (why?).

But both are async frames on alpha. The only difference is the
layout, not what's put in it. On ia64 the layout is always the
same, but we don't populate everything all the time.

> > o  Have _ia64_restore_context() call setcontext() for async contexts
> >    and do the work in the kernel. Restoring the high FP will not
> >    result in the enablement of the high FP registers, because we
> >    can restore them to the PCB. They will be loaded into the CPU
> >    when there's a need for them (which may be never).
> 
> This can't really work without a special system call because
> you need to atomically set the thread mailbox pointer.

Yes. That too. I had a brainwave: We still support the old
syscall path, which is based on the break instruction. It's
a trap. So, we can use the break instruction to trap into
the kernel, executing the setcontext() syscall and go back
to the interrupted context without any hackery. This at
least resolves the problem of having 2 paths into the
kernel: we take a trap to restore a context created by
a trap. I guess I won't obsolete the old syscalls anymore :-)

Alas, I forgot about the mailbox pointer... We don't have a
syscall for that. I could probably put the info in the context
itself, set a flag and enhance setcontext() to do it atomically
as a possibly undocumented feature/extension to support KSE.
Hmmm...

> If you chose this route, or something similar that needs a
> system call, you might optimize the kernel.  You might
> only need to make an upcall when the KSE's quantum expired
> instead of whenever a thread was interrupted.  Or perhaps
> when there are no unblocked threads in the kernel that
> require an upcall to notify the UTS, there's no need
> to make an upcall.
> 
> In regards to the high FPU thing.  Can't you tell the
> UTS via the exported context whether a high FPU restore
> is necessary?

It's always necessary. Even if the high FP wasn't enabled
at the time of the interrupt, it doesn't mean that the high
FP registers don't have valid values. It's just that they
haven't been accessed in that timeslice yet (we need some
heuristics to enable the high FP by default for that process
until after some timeout -- that avoids a trap every time
slice).
Also, another thread may use the high FP registers and since
user level threading is indistinguishable for the kernel, we
need to either have synchronous context switches or save the
whole lot for async context switches.

I think I'll go with the following (for async contexts):

o  Use ifa and isr in the context to pass the pointer to the
   mailbox and the value we want to write to it. Both are
   only used by the kernel to pass information about the
   fault or interrupt to the kernel, so they are unused
   fields in contexts.
o  Set a flag in the context to indicate that we want the
   mailbox set (if applicable).
o  Use the break-based syscall path to call setcontext(2).
o  Have the kernel switch the context and set the mailbox.

It's a bit of a hack, but I think other solutions are much
more painful. Both implementation-wise and kludge-wise.

-- 
 Marcel Moolenaar	  USPA: A-39004		 marcel at xcllnt.net