Kernel Support for System Call Performance Monitoring

[Bosko Milekic]

On Mon, Jun 16, 2003 at 01:20:03PM -0400, Yaoping Ruan wrote:
> We have been working on improving Web server performance on
> FreeBSD, and think you may be interested in the results and
> techniques we used. Specifically, we focus on the SpecWeb99
> benchmark and the Flash Web Server, and have roughly quadrupled
> its performance. We did this by adding support for a very
> low-cost kernel performance monitoring system, which allowed
> us to find and fix a number of bad interactions between the
> server and the OS. We additionally augmented one of the system
> calls, sendfile, to be more useful for this kind of server.
> We think that our observations may be useful for other servers,
> and may present opportunities for performance improvement in
> FreeBSD.
> 
> A paper describing our system can be found at
> http://www.cs.princeton.edu/~yruan/DeBox and we can provide the
> patches we made if anyone's interested. We welcome any comments
> and feedback that you have.

  First off, thank you for choosing FreeBSD for your research.
  The more effort is put into doing this sort of research, the better it
  is for both the academic community and the industry.

  I've read your paper and have a few brief notes:

  - On DeBox implementation.  I understand that the DeBox implementation
    is primarily a tool used for tracking down potential application
    bottlenecks and so the relative importance of the crudeness of the
    implementation is not so high.  However, I'm looking at this from
    the perspective of introducing DeBox as a permanent option in
    FreeBSD, and two immediate problems are:

    1) User-visible DeBoxInfo structure has the magic number "5"
       PerSleepInfo structs and the magic number "200" CallTrace
       structs.  It seems that it would be somewhat less crude to turn
       the struct arrays in DeBoxInfo into pointers in which case you
       have several options.  You could provide a library to link
       applications compiled for DeBox use with that would take care of
       allocating the space in which to store maxSleeps and
       maxTrace-worth of memory and hooking the data into resultBuf or
       providing the addresses as separate arguments to the
       DeBoxControl() system call.  For what concerns the kernel, you
       could take a similar approach and dynamically pre-allocate the
       PerSleepInfo and CallTrace structures, based on the requirements
       given by the DeBoxControl system call.

    2) The problem of modifying entry-exit paths in function calls.
       Admittedly, this is hard, but crudely modifying a select number
       of functions to Do The Right Thing for what concerns call tracing
       is hard to justify from a general perspective.  I don't mean to
       spread FUD here; the change you made is totally OK from a
       measurement perspective and serves great for the paper, it's just
       tougher to integrate this stuff into the mainline code.

  - On the Case Study.  I was most interested in the sendfile
    modifications you talk about and would be interested in seeing
    patches.  I know that some of the modifications you mention have
    already been done in 5.x; Notably, if you have not already, you'll
    want to glance at:

http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/uipc_syscalls.c? \
    rev=1.144&content-type=text/x-cvsweb-markup 

    (regarding your mapping caching in sf_bufs)

    and this [gigantic] thread:

http://www.freebsd.org/cgi/getmsg.cgi?fetch=12432+15802+ \
    /usr/local/www/db/text/2003/freebsd-arch/20030601.freebsd-arch
    
    (subject: sendfile(2) SF_NOPUSH flag proposal on freebsd-arch@, at
     least).

   You may want to contact Igor Sysoev or other concerned parties in
   that thread to show them that you actually have performance results
   resulting from such a change.

   Finally, I'd like to sort of make a longshot proposal; more of a "if
   you have the time" follow-up to your work that someone could be able
   to perform, and that would certainly be interesting to see: how all
   this works out when forward-ported to FreeBSD 5.x.

> Sincerely
> 
> - Yaoping
> yruan at cs.princeton.edu

Regards,
-- 
Bosko Milekic  *  bmilekic at technokratis.com  *  bmilekic at FreeBSD.org
TECHNOkRATIS Consulting Services  *  http://www.technokratis.com/