FreeBSD Kernel Internals Documentation

Da Rock freebsd-questions at herveybayaustralia.com.au
Fri Dec 30 11:56:00 UTC 2011


On 12/30/11 16:39, Polytropon wrote:
> On Thu, 29 Dec 2011 22:43:16 -0500, David Jackson wrote:
>> However, My finding is that due to poor documentation, [...]
> That kind of statement doesn't fit well to FreeBSD
> which is known for its excellent documentation, often
> considered superior to other open source projects.
I'll second that. As a sysadmin its brilliant at documenting the "core" 
system (maybe needs some improvement in some of the more obscure areas 
that aren't used often). On top of that it does well at providing 
support for common uses and apps.

Developers probably couldn't ask for more- linux (for eg.) is a 
"general" thing, with the distros leaving the docs to the "common" linux 
community at large. FreeBSD docs are here: at FreeBSD.
>> [...] the FreeBSD kernel
>> is nearly impenetrable to an outsider. I have been able to find no
>> comprehensive documentation of kernel internals. I have found it nearly
>> impossible, due to lack of comprehensive documentation, much of any of the
>> kernel internals. What I see is an internal cliche of developers who are
>> aware of its myraid of undocumented esoteric secrets, and very little to
>> actually help anyone else to understand it.
> The source code itself is very verbose and well documented.
> Unlike other operating systems, FreeBSD provides man pages
> that can be accessed offline and provide information about
> many kernel aspects, such as interfaces and device drivers.
>
> Searching "http://www.google.com/search?q=freebsd+kernel+hacker"
> or something of that kind will also bring further information
> and and resources.
Yes it is good, but can be obtuse to someone unfamiliar. But then I'm a 
very green developer and more a sysadmin myself. The lists are 
fantastic! Very handy for someone seeking "the truth"... :)

I haven't looked, but it would be good to have some irc channels handled 
by the team though.
>> Any good, well designed software projects will have comprehensive
>> documentation of the source code, this includes code comments, information
>> on what every piece of code does, how the entire system fits together, and
>> descriptions of every variable and function. Any well run project would
>> insist that code contributors upload full and comprehensive documentation
>> of how their source code is written, how it works, etc.
> I think this is what you'll find in the FreeBSD kernel
> sources. I had a random peek at some files in the
> /usr/src/sys subtree (where the kernel sources reside)
> and found that it fits your claim.
>
>
>
>> Documentation is vital and good practice because it saves time, it prevents
>> people new to the project having to waste immense amounts of time trying to
>> figure out a vast and cryptic puzzle. Without good documentation software
>> can be nearly useless, unmaintainable and difficult for an outsider to
>> learn, to the point where it may actually take less time to just throw it
>> out and start from scratch.
> I couldn't agree more. As a developer, I've worked with
> many operating systems, and finally I found FreeBSD which
> provides the documentation in a way I feel comfortable
> with, unlike its predecessors in my "IT career". :-)
>
> However, there are differences in how you judge documentation
> to be _good_. Talk to a mainframer, and he will tell you a
> different story. Then talk to a "Windows" person and explain
> what documentation is, and he'll tell you that you don't
> need it. :-)
>
>
>
>> These are reasons that FreeBSD needs better documentation, documentation of
>> how the entire system fits together, what lines of code do, the purpose of
>> variables and functions, etc, in descriptive English. This is key to
>> developing maintainable software.
> As I said, that _is_ what you already find in the source
> tree. Allow me to give you one example (out of many):
>
> File randomly picked: /usr/src/sys/nfs/nfs_common.c
>
> /*
>   * Help break down an mbuf chain by setting the first siz bytes contiguous
>   * pointed to by returned val.
>   * This is used by the macros nfsm_dissect for tough
>   * cases. (The macros use the vars. dpos and dpos2)
>   */
> void *
> nfsm_disct(struct mbuf **mdp, caddr_t *dposp, int siz, int left, int how)
> {
> 	...
>
> and so on. Descriptive english text, usable variable
> and type names.
>
> Or how about /usr/sys/boot/i386/btx/btxldr/btxldr.S
>
> /*
>   * BTX program loader for ELF clients.
>   */
> start:          cld                             # String ops inc
>                  testl $OPT_SET(RBX_MUTE), 4(%esp) # Check first argument
>                  setnz muted                     #  for RBX_MUTE, set flag
>                  movl $m_logo,%esi               # Identify
>                  call putstr                     #  ourselves
>                  movzwl BDA_MEM,%eax             # Get base memory
>                  shll $0xa,%eax                  #  in bytes
>                  movl %eax,%ebp                  # Base of user stack
> #ifdef BTXLDR_VERBOSE
>                  movl $m_mem,%esi                # Display
>                  call hexout                     #  amount of
>                  call putstr                     #  base memory
> #endif
>                  lgdt gdtdesc                    # Load new GDT
>
> If that is insufficient, maybe you can give an
> example for comparison of how you imagine the
> documentation should be provided.
>
>
>
>> I saw where someone automatically generated "documentation" with Doxygen.
>> This is nearly useless, because all it shows is a huge list of functions
>> and variables but does not include any text on what they do. At best,
>> Doxygen can only provide a template for documentation that can be filled in
>> with descriptive English information on what everything does.
> If properly used Doxygen can help, but it's not the
> entire solution, I fully agree. Some people need to
> learn the hard way, just like me when I visited my
> _own_ source code a decade later and found something
> like kk[w.q].x = fbx(x,x0+a0+1+aa[q].t,dt,h,dh-1+3);
> which is totally ugly. :-)
Seen that too. But I am very pedantic to probably a fault with my 
comments and even then I can get confused as to where I was going.
>> One idea might be to have an official wiki that contains the template
>> generated by Doxygen which can then be filled in. When changes to the
>> source code is made, it is good practice for the commiter of such changes
>> to document their code as it is submitted.
> Also Wikis aren't for everything. I don't know how
> _you_ as a developer think about it, but _I_ like
> to have the documentation available _in_ the code
> I consult, offline, without the need to use a web
> browser, everything based on _files_ so I can use
> the system's powerful tools to work with them (e.
> g. search for information, filter, create lists
> and so on).
>
> FreeBSD has an official Wiki, and there are several
> others. Again a question arises: Would you like the
> documentation to be spread across arbitrary Wikis,
> user home pages, web forums and blogs?
Everyone has their own methods. I'd like to see a wiki for developers 
myself, hosted by FreeBSD as all other docs, I'd find that very useful; 
especially if all the info was centralised and updated to include the 
odd parts currently distributed in articles. All the info in one place 
and easy to see how it fits together.

As the info for functions is resolved it can be added to the wiki, and 
upcoming changes can be appended to the end of functions/modules/etc.

The handbook should be untouched though, its for a whole different set 
of users.
>> This allows others who come along who need to maintain the code to more
>> easily understand what the code does.
> History teaches that this is achieved by using the
> code itself. :-)
>> Another idea which would also improve the useability of FreeBSD would be to
>> have a wiki which would be updated by kernel contributors whenever they add
>> support for a certain piece of hardware. This would make finding hardware
>> compatability information easier from one central, up to date and current
>> source of information.
> Those informations will typically be added to the UPDATING
> information, as well as to the release notes of an upcoming
> RELEASE. Deltas in the src/ subtree also indicate when
> something new is present. As FreeBSD is a _quality_ operating
> system, developers pay attention to write the documentation
> along with releasing new drivers, so you aren't stuck with
> an ugly-hacked kernel module full of lines that look like
> the fbx() example I gave above. :-)
>> These documentaiton ideas, for commiters to document their code when they
>> upload it, and document their hardware support additions, are just good
>> software practices that should be highly recommended and encouraged
> I fully agree with that statement. If you could, for example,
> point to documentation you judge _good_, or point at some
> FreeBSD documentation that you think needs heavy improvement,
> it would help to back your argumentation. Otherwise, it can
> be seen as "already done".
I'd put my hand up for that. I'd be willing to host a mirror as well if 
needed.


More information about the freebsd-questions mailing list