Re: Current ULE state and plans (was Re: HMP scheduling on FreeBSD)

Hi,

> > Hmm...
> Hoowee. That's some passive aggressive stuff, right there. ;)

Yes.  And?

My skepticism mostly comes from what you're saying, and also not saying (see below), in this thread.  But I won't hide it also comes in part from an interaction we had about two years ago where you showed approximately the same traits as in this thread.

Now reading what you wrote below, in retrospect "Hmm..." was *way* too nice.  Anyway.

> You have gone to great lengths to indicate that you have put alot of time
> and effort into spearheading the task of improving ULE scheduling and that
> your findings will dictate it's outcome.
> Even to the demise of 4BSD:
> 
> > Once these problems are fixed, it's likely that, experiments confirming,
> > there won't be any more reason to keep 4BSD around.  One of my personal
> > goals is to deprecate 4BSD in 16 if possible, as it is more and more
> > irrelevant on modern machines and seems too lightly tested.
> 
> You clearly have an axe to grind.

Not at all.  It's very likely you have problems with reading/understanding.  Do you see the words "likely", "experiments confirming", "if possible", "seems" in your quotation?  Why did I bother to write them, in your opinion?  Contrast that with your use of "dictate".  You are removing every nuance in what I said initially, and ignoring what happened after in this thread.  This is just dishonest.

Some of my meta-goals are to maintain schedulers to the highest quality and lessen the maintenance/testing burden.  Perhaps I should have stated that first, but not sure it would have prevented you from scaring yourself.  Considering deprecating 4BSD is a natural consequence, for reasons that I have already developed in this thread and that I'm not going to repeat here.

> IMHO I find it astounding that you're so dismissive toward 4BSDs relevance.
> You're quick to dismiss anyone's suggestion to the contrary. With the 
> possible exception of Steve.

Not only are you talking past me, but now putting words in my mouth.  Even better.  Confirming my initial skepticism about your real intentions.

I am saying that 4BSD has fundamental limitations that cannot be fixed, and that ULE was intended to fix.  They are mostly evident in circumstances of high load (lots of processes being created/terminating) and in terms of interactivity.  I already posted a couple of links in support of that.

I've been on the contrary asking for feedback on who uses 4BSD, and why.  But, as should have been obvious, responding just "it works better for me" is only a small piece of information, and not enough.  Especially when this statement is based on tests performed years ago.

While we are at it, here's something you did not answer:

> > FWIW we use 4BSD exclusively on some 1,200 servers. As it has proven to 
> > return the best results for our own use case,
> > as well as our clients use cases.
>
> Results measured how?  When was the last time a comparison was made?

It's almost funny that you dare accuse me of dismissing "suggestions" given:
1. I said I would like that we go further than "suggestions", in the form of scenarios, numbers, etc., in other words, any tangible, objective, and ultimately actionable data.  This actually proves that I'm taking into account the feedbacks.
2. I've looked at the only evidence provided in this thread, Steve's, and responded to him after a quick analysis (that I see typical symptoms of problem number 2; that he should test after the fixes, as they are likely to fix his scenario for a large part, if not entirely, but we will need tests to be 100% sure).
3. You don't bother to provide any details when asked to.  Despite which, I've perfectly heard your point of view, contrary to what you're claiming.  But for now, it is just that, an opinion and a very vague report that we are even not sure still applies today.

> You have zero knowledge of anyone else's use case for choosing 4BSD over ULE.

And now personal attacks.  What do you know about what I know?  "zero knowledge"?  Are you sure, especially after what I answered to Steve?

>  This is not taking an objective view on scheduling. Nor does 
> it make a good case for your "improvements" on ULE.

That's your opinion, apparently based on no technical knowledge of the matter, as you said yourself elsewhere.

Planned improvements to ULE are based on multiple analyses (theoretical, and some practical use cases).

> But scheduling is the epitome of objectivity and nothing in this thread suggests 
> objectivity.

You're not really reading, and actually putting on me your very own deficiencies.

> I'd like to suggest that we take a whole other look at this and how we might
> *all* make contributions to improve FreeBSDs scheduler or schedulers.
> These contributions might only be to present actual use cases, and or the
> results of those use cases.

A "whole other look"?  Presenting actual use cases, and results?

Are you really kidding us?

This is what I've been asking from the start, and you didn't bother to answer.

Oh, excuse me, I forgot that you were not really reading.

> A predisposition to nuke 4BSD does not suggest a
> well conceived plan for improving the scheduling/schedulers on FreeBSD.
> We may well discover that one-true-scheduler is not practical and that having
> several is a more realistic choice.

Then I have a scoop for you:  Nobody serious in the scheduling field has ever thought that there is a one-true scheduler.

That said, a lot of OSes, including FreeBSD, have proved that, in practice, a single scheduler can adapt to most workloads (more or less imperfectly), and this is what we are aiming at with ULE.

There are some fundamental traits that must be present for a scheduler to deal with highly concurrent workloads.  Traits that ULE has and that 4BSD does not have.  But scheduling does not stop there, it also relies on a lot of important details, some of which ULE got wrong.

Again, same question as for Steve: Can you please explain what the point of keeping 4BSD is if it's not better than ULE in anything (assuming we can get to that point, which isn't the case right now)?  I have some answers of my own, but I'd like to hear yours.
 
> It is for this reason (I assume) that we discovered that 4BSD achieved better 
> results in "flattening" the graph than ULE. This is the reason I chose to defend 
> 4BSD.

Repeating my previous questions:
1. On which metrics was this assessment based on?
2. When was this assessment performed?

> My concerns are pragmatic.

This is absolutely not what you are showing so far, as far as the community is concerned.

> Let's "reboot" this thread. What do you say? :)

I say it's mostly your attitude that needs a reboot.  What do you think of going through this thread again and read it carefully with an honest mind, and finally becoming constructive by exposing your use case and measurements with greater details, instead of sneak attacks as well as suggestions and insinuations on topics you do not have technical command of?  And committing to testing ULE again once the fixes have been done?

-- 
Olivier Certner