Patches for threads/scheduler abstraction.

[Julian Elischer]

On Thu, 17 Apr 2003, Jeff Roberson wrote:

> 
> On Wed, 16 Apr 2003, Julian Elischer wrote:
> 
> >
> >
> > > >
> > > > My plan is:
> > > > eventually, sched_4bsd.c will only do the original BSD
> > > > scheduling (except with threads instead of procsses)
> > > > i.e. unfair, but simple and easy to benchmark on non-threaded apps.
> > > > (or on threaded apps with no competing processes)
> > > >
> > > > sched_4bsd_kse.c will suck in the current contents of kern_switch.c
> > > > and the KSE'd version of sched_4bsd.c.
> > > >
> > > > this will act like the current 4bsd scheduler WITH KSEs
> > > >
> > >
> > > There is a better way to do this.  If you would listen to my tiered
> > > approach you could end up with one copy of sched_4bsd and not two.
> > >  The kse specific logic could just sit in kern_kse which would not
> > > tell sched_4bsd about threads until they owned the KSE.  We could
> > > even do this without the kse structure as a go between.
> >
> > I think that is an overcomplication.
> > the two BSD based files would be very different.
> 
> It's not an overcomplication.  The two tasks are independent.

I'm missing something here.

I think our problem is one of definitions..
in my vocb.

the threading system:
 has an API to allow user space to create/delete threads and specify
 behaviour of threads.. it knows about processes, KSEGRPS
(subprocesses?, sched_groups?) and threads. it doesn't know about KSEs.

the scheduler: Takes requests from the general OS to make threads
runnable, and takes requests to find a new thread to run (Bfrom
switch()). It also has some entrypoints for such things as clock tick
notification, and entrypoints for events such as a fork, exit,
thread_creation, thread destruction. It contains code to recalculate
priorities and to change them. It may, internally use KSEs or it may
not. That is a decision for the writer. It only communicatges with the
higher layers only in terms of threads, ksegrps (sched_groups or
whatever they get called) and processes. It has private storage in all
those structures which may be used to implement all sorts of data
structures (e.g. run queues, or priority lists or whatever it wants).


Are you saying that you want the threading system to know about KSEs?
if so, WHY? 
especially when some schedulers may NOT know about KSEs.

> 
> > For a start the simple one would be queueing threads on the run queues.
> > A system compiled with that scheduelr would have no KSEs anywhere
> > in the entire kernel.
> > The kse one would be queueing KSEs. I don't see how you can do this
> > with a shared file.
> 
> You're missing the point.  The scheduler shouldn't be tied to the
> threading implementation.

It wouldn't be if the threading implementation didn't need to know about
KSEs. Or are you saying that you want KSEs to be what the lower end of
the threading system uses and passes to teh scheduler, even if the
scheduler has no other need for them except to extract a thread out of
it and ignore the KSE?

> 
> The threading implementation can pick a KSE to run, assign that thread to
> the KSE, and then set that thread runnable with the scheduler.  If it
> later decides that the KSE should be running something else it can remove
> the thread currently assigned to it from the scheduler and then reassign
> the KSE and add the new thread to the scheduler, which places it on the
> run queue.

I think I'm seeing the problem here.. We hav different ideas of where
the boundaries of the threading and scheduling modules are. You are
saying that the threading implementation takes part in the scheduling..
I on the other hand don't think that is right.. The SCHEDULER takes part
in the scheduling. the threading system just exists in order to allow
the system to create and destroy threads. The scheduling system is
responsible for taking requests to make a thread runnable (from many
parts of the system) and for deciding what thread is returned to
switch() when it asks for a new thread context to load and run.



> 
> This way you will not duplicate code and you will keep the two tasks
> independant.  Essentially the sched_*.c files decide system scope
> contention while the threading implementation determines the process scope
> contention which may include some concurrency limits imposed by KSE or
> some other structure.


Ok, so you want to schedule a KSE and once you have decided what KSE is
run , you allow the threading system to map that to a thread, or, maybe
a sequence of threads.. is that right?

The difference is that I don't see that as part of the threading system.
I see that as part of the scheduling system..
If it makes things better, I can agree that there is a posibility that
the scheduler could be implemented as two parts.. The basic scheduler
that decides what ksegrp(sched_group, subproc would be better names) is
going to be allowed to get time to run threads, and a thread scheduler
that decides which threads within that larger entity are going to be run
on its behalf. The difference is that I don't think that that lower part
is part of the thread system. I think it is part of the scheduler.






> 
> Do you see?  This way we could get KSEs out of the entire kernel other
> than kern_kse.c and still support them with the sched_4bsd and sched_ule
> scheduler.  Otherwise we're going to have a copy of each scheduler for
> each threading implementation and we wont be able to support two threading
> implementations simultaneously.


I'm afraid we have such a wide terminology difference here that the only
way we are going to be able to truely understand what we both mean is
with pictures and examples.


> 
> > Anyhow, the following hack (totaly unoptimised.... notice the
> > > existance of sched_td_exit, sched_exit_thread, and sched_thr_exit..
> > I just haven't got to cleaning it) is not so advanced that
> > the question of tiered schedulers is relevant yet..
> > this patch just shows that it is possible to get the KSEs
> > out of the rst of the system.

Which you have agreed is not a bad idea.