[FreeBSD-Announce] Final report on funded FreeBSD development project

[Poul-Henning Kamp]

Final report on funded FreeBSD development project [1]
------------------------------------------------------

    As promised, here follows the status report on how my six months
    of user-community funded FreeBSD development went.

    The goal of this effort was to get as far in the "buf-junta" plans
    (http://people.freebsd.org/~phk/plan.html) as possible, and as I
    have outlined below, a significant dent was made in that list
    over the last six months.

    I would like to thank 187 persons and companies for, though their
    donations small and large, making this work possible.  The future
    of FreeBSD is very much in the hands of the developers, and by
    helping this particular developer get his hands free for half a
    year certainly moved a lot of stuff from the future into the present.

    If you, my donors, are not happy with what you got out of your
    donations, the fault is entirely mine. (and I would like you to tell
    me about it as fast as possible.)  Please do not fault the FreeBSD
    project as such or the FreeBSD Foundation for that matter, if this
    was not a success, it is all my personal fault.

    If on the other hand you think you got your moneys worth and you
    would be willing to help support FreeBSD development financially
    also in the future, then, and only then, will I declare my project
    to have been a complete success.

    Feedback either way is most welcome!


The future
----------

    A lot of people have asked me "what about the future ?"

    I promised them to make up my mind and let them know in this
    report: I have decided to not do another fundraising drive at
    this time.

    There are a number of reasons for this.

    The primary reason is that I do not want to be stuck in an ivory
    tower doing only the FreeBSD work I can see from my lab in the
    basement.  I need to get out amongst people and get inspiration
    and input from real-world usage of FreeBSD.  To that end I hope
    to land a couple of contracts currently under discussion and
    occupy half of my time this way.

    The other reason is that I hope the FreeBSD Foundation will be
    the place where this kind of fundraising happens.

    Channeling the funds through the FreeBSD Foundation means that
    this will not grow into a personality cult around my person and
    that money can be spent where it is most needed rather than on
    where I happen to fancy spending it.
    
    The minor reason is that the paperwork associated with fundraising
    is not a favourite hobby of mine and I'm still not done resolving
    various sticky details [4]


    But since there are still a lot of work to be done in this area
    of the kernel and since that is what I think I am best at, I
    have sent a proposal to the FreeBSD Foundation for funding me
    part time for a period to continue working on this stuff.

    I was somewhat late in sending my proposal to the Foundation,
    and have not yet received an answer from them, so the outcome
    of this is not yet known.  If it falls through, I will find a
    couple of contracts for now, and then reconsider the situation
    later in the 2005.


    If you want to support FreeBSD development financially, the way
    to do it right now is to donate money to the FreeBSD Foundation.
    If you do so, some of the money may or may not end up funding
    my work, either way, it will be money well spent.


High-lights from my personal score-card:
----------------------------------------

    June ... October  -- TTY code/driver cleanup

	In order to not clash with the RELENG_5 release work, I
	attacked the tty code/drivers in order to try to drive Giant
	out of the pty driver.

	I found that a massive amount of code which should have
	lived in the generic tty code was copy&pasted into almost
	every and all tty device drivers.

	In the end I did not manage to do any of the actual locking
	work, but I did get a lot of preparation for it done: I
	eliminated the 3100 lines of copy&paste'd code so that there
	will be many fewer places where locking will have to be
	handled (and bringing a lot of consistency to our tty drivers
	as a side effect).

	I also managed to get struct tty properly reference counted,
	but some sticky issues about unloading tty drivers with
	dangling sessions still need to be handled.

    June ... November -- Device and Fifo vnode bypass

	In terms of architecture, this is one of the most significant
	changes I have made in the last six months.

	Despite being a relatively small change in terms of number
	of lines of code changed, this project was spread out over
	a long time in order to be sure to catch all the little
	details in the corners.  One of my test machines have been
	more or less dedicated to to this subproject for 5 months.

	The end result is that access to devices and fifos ('named
	pipes"), from programs now does not go through the vnode
	layer for the "traffic" calls, read, write & ioctl.  (For
	device drivers we may still need Giant if the driver is
	marked as needing this.)

    August -- Rewrite Floppy driver.

	The internal state engine was rewritten and the driver made
	INTR_FAST, Giant safe and GEOM aware. 

    September ... October -- DEVFS/SPECFS consolidation

	Devices are now only supported in the devfs filesystem, and
	supporting code was removed from ufs, ext2fs, cd9660 and a
	few minor filesystems.

	In practice this have had very little impact when seen from
	outside the kernel, but the simplifications in the kernel
	are extensive and very significant.

	The end result is that specfs is retired and only devfs
	handles devices now.

	A significant part of this work has been to properly reference
	count access into device drivers in order to make it safe(r)
	to unload device drivers.  There are still many hurdles here
	before it is 100% safe to unload a driver, but at least now
	the cdevsw interface should not be one of them.

    October ... [ongoing] -- struct buf/vnode -> buf/bufobj/vnode

	This is also one of the architecturally very significant
	changes which had its fingers all over filesystems and buffer
	cache code.

	The work is not carried all the way through, but has progressed
	sufficiently far to reap the first major benefits (see below).

	It is hard to fairly summarize this stuff without it turning
	into a long architectural monologue, but the gist of it is
	that the aspect of vnods which serve as the attachment point
	for stuff in the buffer cache has been given an indenpendent
	data structure (bufobj) which in the future will not be tied
	to vnodes only.

	The untangling has reached and stopped right in front of the
	syncer code, and will continue from there as time permits.

    October -- Move copyonwrite and prewrite into FFS where they belong.

	These two functions are private to snapshots and softupdates,
	and with a private strategy implementation in FFS they can be
	isolated there entirely.

	This simplifies the buffer cache code a bit.

    October -- Proper waiting for pending geom events

	A process which caused events to be created in geom will now
	wait for them to be completed before returning to userland.
	This removed an insufficient lot of explicit waits.

    October ... --- Put local filesystems directly on GEOM

	Another piece of significant architecture [2]

	Instead of taking a tour through the vnode layer, DEVFS and
	geom_dev, go directly to the new Geom class geom_vfs to access
	disk devices for local filesystems.

	Amongst many other benefits, this gives us correct read/write
	tracking on devices, and it is now possible to mount the same
	filesystem r/o many places.

	Together with the device vnode bypass, this entirely takes
	I/O traffic from devices out of the vnode layer (and more
	significantly: vnode locking).

    November ... December -- FILEDESC locking

	In order to make the locking of file descriptors work better
	with devices and fifos bypassing vnodes, the locking was 
	reworked and struct filedesc got a reference count more for
	protecting the memory allocation.

    November ... December -- omount(2) -> nmount(2) kernel transition.

	All filesystems converted to implement the nmount(2) API
	now.  Calls to the old omount(2) API are converted in a
	special compatibility function in the filesystems.

	Userland conversion to nmount(2) should happen in the spring
	so that RELENG_6 will be entirely nmount(2) but still support
	omount(2) calls.  Support for omount(2) will be dropped in
	RELENG_7.

    December -- Make VOP_* implementations typesafe.

	By using sparse struct initialization, it was possible to avoid
	a lot of the magic pointer gymnastics in creation of the VOP
	method vectors for filesystems.  The result is typesafe and
	faster VOP_ method calling.

    December -- Root filesystem mount rework

	Since all filesystems now accept nmount(2) calls, the magic
	code mounting the root filesystem was cleaned up, removing
	the need for magic per-filesystem code to handle the root
	file system.

	This eliminated the main source of "bogo-vnodes" in the
	system, leaving only the syncer-vnodes now.

    June ... December -- Bugs introduced, code broken.

	Throughout a number of mistakes, minunderstandings, keyboard
	misfires and just plain stupidity on my part have introduced
	bugs and broken code.

	This will be fixed as soon and as fast as I can.


Pointless statistics:
---------------------

In the mock sprit of sports pages everywhere, and general journalistic
"that was the year that was" reporting approaching the end of a
year:

	Total donations:			        33336 USD
	Average donation (sans outliers):		   72 USD
	Total number of commits to CVS:			  577
	Cost per commit:				   58 USD
	Average time between commits to CVS:		7h36m
	Number of submits to p4:			  996 (minium ?)
	Average time between submits to p4:		4h25m
	Netto lines of code added,
	    not counting retired device drivers:	-7291
	Pointy hats awarded:				    3
	Cost per codeline:				 -.22 USD
	Disks crashed:					    3
	Super strong fridge magnets retrieved:		    6
	Shoulder injuries:				    1
	Electricity used by computers			 5275 kWh
	Successful attempts to communicate 
	   non-criminal status to Paypal:		    0
	Futile attempts to communicate 
	   non-criminal status to Paypal:		   10
	Random acts of kindness received:		    3
	Random acts of kindness not delivered
	   due to botch by amazon.com:			    3  [3]


Merry X-mas & happy new-year,

Poul-Henning



[1] For those of you who don't know what I'm talking about in this
email, a quick visit to http://people.freebsd.org/~phk/funding.html
would be a good idea.

[2] I rather fear I sound as boring as the tourist guides on
the ruins of Akropolis, my apologies.

[3] Amazon.com had a wrong shipping address on my "wish-list", so
at least three packages of gifts from kind people have travelled
all the way from Amazon to my old address and back again.  I'm told
people will get a full refund from Amazon once the package makes
it back to them.  My apologies for this mess.  Much appreciated
anyway.

[4] Paypal is very well and nice, but trying to get through their
CMS system to explain to them that just because you are located in
the EU and received money doesn't make you a member of the mafia
is still not an art I have managed.  If you happen to know an email
address of a senior executive let me know.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.