Improving the handling of PR:s

[Kris Kennaway]

韓家標 Bill Hacker wrote:
> Mark Linimon wrote:
>> On Fri, Jan 11, 2008 at 07:35:41PM +0100, Peter Schuller wrote:
>>> But understandable or not, the problem becomes particularly 
>>> frustrating when it affects PR:s that contain patches. Such PR:s 
>>> constitute direct contributions to the project. In cases where such 
>>> patches are correct/good enough, the non-application of those patches 
>>> have what I believe to be significant effects.
>>
>> I think you make some excellent points -- especially with how
>> understandable it is for someone who's submitted a patch which got
>> ignored, to not do the work to submit the next one.
>>
>> With respect to patches, the two things that I've tried -- and they have
>> been insufficient -- are the following:
>>
>>  - weekly email of "PRs containing patches"
>>
>>  - weekly email of "PRs recommended by the bugbuster team"
>>
>> We can make the following observations:
>>
>>  - the "push" paradigm doesn't work particularly well.  Partly this is
>>    due to fatigue if you try to read through all this stuff (see below).
>>
>>  - the "recommended" list has been slightly effective, but it's going
>>    to take some time for it to take off.  For one thing, it has been
>>    underpublicized; for another, we don't have the culture of committers
>>    getting "warm fuzzies" from committing PRs.  (Since no one working
>>    on that kind of stuff gets paid AFAIK, that's the only positive
>>    feedback they're going to get.)
>>
>> Last week someone made the observation that if these things were instead
>> updated daily and posted to a web page, that it would be far more useful.
>> I've taken that up as a task.
> 
> Observation: PR or 'other' - it is presently rather difficult to find 
> [older | most] patches- which reduces the opportunity for testing and 
> feedback, and increases the likelihood of having the situation reported 
> yet-again. And again..
> 
> so ..
> 
> - Can there be a port category (real or 'virtuaized', added - along the 
> lines of:
> 
> 'patches and workarounds'
> 
> With caveats, of course.... but hopefully also with provision for 
> co-located feeback, both positive and negative.
> 
>>
>> Your other point about "latest PRs" is also a good one.  That should be
>> relatively easy to do.  I'll take up that task.
>>
> 
> Question: Can an improved 'data mining' project add value?
> 
> One wherein the tools to look for commonality - sometimes of the less 
> obvious nature - would be easier for all-hands?
> 
> I include in this the ability to build 'reports' - not just do the 
> present ad hoc searches.
> 
> If such could be done 'better', it might also benefit from crossproject 
> (i.e all the *BSD's) scope.
> 
>>> * The committer may not be familiar with the code and, even though 
>>> the patch looks good generally, it is felt that someone familiar with 
>>> that part of the system needs to have a look.
>>
>> In particular, this hurts for areas of the system that are unmaintained.
>>
>>> What I suggest is a system where a group of non-committers 
>>> systematically pre-process PR:s, to provide feedback and additional 
>>> quality assurance to the committer that is to process the PR.
>>>
>>> This group of people could either be a self-proclaimed group of 
>>> volunteers, or perhaps a group of people satisfying the criteria of 
>>> "guy we kinda trust to do testing and provide a useful indication of 
>>> sanity and correctness, but not with a commit bit".
>>
>> So far it hasn't happened.  We've set up the freebsd-bugbusters@ mailing
>> list and the #freebsd-bugbusters IRC channel on EFNet (and please join
>> us!) and the latter is where our last 2 bugathons took place.
>>
>> It's clear that there are several people who want to help process the
>> PRs, and we don't have a good answer for them on "how can I contribute?".
>> The existing tool, and social conventions, don't allow for non-committers
>> to change PR states.  As far as we've done in the past is to grant people
>> "GNATS access" rights but not "commit rights", on an experimental basis.
>> We've done this twice, and although it has worked well, just two people
>> isn't enough.  (One has gone on to become a full committer -- which is
>> great!; the other current does not have as much time for FreeBSD work).
>> Several hundred PRs were dealt with by these two folks, so I consider the
>> experiment a success.
>>
>> What we used as a qualification was "track record of responding to PRs 
>> and
>> questions on mailing lists", fwiw.
>>
>>> One possible answer to this that I have gotten in the past with 
>>> another project, is that noone is stopping me from just grabbing a 
>>> bunch of PR:s and posting follow-ups saying I've tested something, or 
>>> otherwise giving feedback.
>>
>> Yes, but that's open-loop as well.  It's the same situation as with 
>> submitting
>> the patch in the first place: it's going to get frustrating if no 
>> committer
>> picks it up and commits it.
>>
>> There's not too many people so thick-skinned and persistent as to keep
>> going forward when no one's using their work.
>>
>>> For example, patches with a high confidence of correctness due to 
>>> many people affirming that they have tested it could be automatically 
>>> prioritized for committers to deal with, and there will be a clear 
>>> and systematic record of what testing was done, and by who.
>>
>> Right.  The "priority" field in GNATS has been so abused as to become
>> useless.  (People assume that setting it higher will cause some kind of
>> magic to happen.)
>>
>> My current thinking is that what committers ought to see is a metric of
>> correctness, and a metric of priority, _as set by someone who's vetted
>> the PR_.  The weekly mailings are too poor an approximation of either
>> to be useful.
>>
>> Adding the second metric would cure one problem that you don't mention --
>> which is that few people have the interest and patience to plow through
>> N-thousand PRs.  It's not humanly possible to look at them all -- even
>> the new ones as they come in.  There's simply too many.  So, you create
>> an expectation "why bother, there's so many anyways".  We need to break
>> that chain of expectation.  A good fix is a good fix.  The PR count will
>> never get to zero; I (with bugmaster hat on) would be thrilled if we can
>> get to the point of just steady-state.
> 
> DB Admins, SysAdmins, 'Management' need BSD, too and not all the work 
> that needs to be done requires 'C' expertise.
> 
> Some of us also have spare bandwidth and hardware as well as time, so...
> 
>>
>>> When I talk about priority above, I do not mean to imply that any 
>>> committer should work on things they do not want to work on. But I 
>>> have to assume that a lot of patches get committed because a 
>>> committer decided to go and pick a few PR:s to process, rather than 
>>> the committer necessarily has a burning interest in that particular 
>>> patch.
>>
>> I think most get committed because a committer sees a PR come in on the
>> mailing list and grabs it.  Much less often do committers go through the
>> database looking for things to fix.  Again, the lousy "search/browse"
>> capabilities of the existing tool let us down here.
>>
>>> As such, if said committer could spend those <n> minutes closing five 
>>> times as many PR:s because the up-front confidence in the correctness 
>>> of the patch is so much higher, perhaps it would help the system to 
>>> "scale", moving some of the burden off the committers.
>>
>> Right.  What we need is to start small and create a _culture_ of where
>> it's fun (or at least intellectually interesting) to work on this stuff.
>> I think the IRC channel may still be the best bet for this.
>>
>> Once I finish fiddling around with the sparc64-6 and sparc64-7 release
>> builds (the first approximation is finished, but I believe we can still
>> add some more packages), I intend to task-switch over to looking towards
>> defining a PR workflow and floating some proposals.  I hope to have
>> something concrete to present at BSDCan.  Please join us on bugbusters@
>> to discuss any more ideas, too.
>>
>> mcl
> 
> Coders will always set their own priorities. Even when working for a wage.
> 
> ;-)
> 
> But all-hands benefit from greater stability and fewer 'critical' things 
> breaking, so rational self-interest might be more effectively applied by 
> each participant if the 'big picture' - in several flavors - was simply 
> published in some 'better' manner.
> 
> You've ID'ed some of the possibilities - there are bound to be more.
> 
> Perhaps some of the committers - or at least the project in general - 
> could benefit from more 'virtual office staff' - folks who specifically 
> are NOT coders - to remove obstacles and help leverage their expertise?
> 
> Background research, documentation, finding and enlisting testers who 
> can help, building and maintaining databases  - *doing* tests.  Some of 
> that is more amenable to being 'directed' than the coding efforts 
> themselves. Different personality types...
> 
> JMHK$2CW..

Hi Bill,

After your last anti-FreeBSD rant didn't you promise to unsubscribe from 
the mailing lists and not come back?  What went wrong? :(

Kris