Scheduler question

[Matthew Dillon]

   It sounds like there are at least two issues involved.

   The first could be a buffer cache starvation issue due to the load on
   the filesystem from the tar.  If the usb program is doing any filesystem
   operation at all, even at low bandwidths, it could be hitting blockages
   due to the disk intensive tar eating up available buffer cache buffers
   (e.g. causing an excessive ratio of dirty buffers vs clean buffers).
   This would NOT be a scheduler problem per-say, but instead a kernel
   resource management problem.

   The way to test this is to double-buffer or tripple-buffer the output
   via shared memory.  A pipe might not do the job if it gets stuck doing
   direct transfers (I eventually gave up trying to optimize pipes in DFly
   due to a similar problem and just run everything through a kernel buffer
   now).  Still, it may be possible to test against this particular problem
   by having the program write to a pipe and another program or fork handle
   the actual writing to the disk or filesystem.

   Another way to test this is to comment out the writing in the usb program
   entirely and see if things improve.

   --

   The second issue sounds more scheduler-related.  Try running the
   usb program at nice -20?  You could even run it at a pseudo-realtime
   priority using rtprio but nice -20 had better work properly against
   a md5 or there is something seriously broken in the scheduler.

   Dynamic priority handling is supposed to deal with this sort of thing
   automatically, particularly if the usb program is not using a lot of
   cpu, but sometimes it can't tell whether a newly-exec'd program is
   going to be interactive or batch until after it has run for a while.

   Tuning initial conditions after an exec for the scheduler is not an
   easy task.  Simply giving a program a more batch/bulk-run priority on
   exec and letting the dynamic priority shift it more to interactive
   operation tends to mess up interactive shells in the face of
   cpu-intensive system operation, for example.  Theoretically dynamic
   priority handling should bump up the priority for the usb program well
   beyond any initial conditions for exec once it has been running a while,
   assuming it doesn't use tons of cpu.

   --

   An md5, or any single-file reading operation, would not overload the
   buffer cache.  File writing and/or multi-file operations (such as a
   tar extraction or a tar-up) can create blockages in the buffer cache.

   It takes a considerable amount of VM/buffer-cache tuning to get those
   subsystems to pipeline properly and sometimes things can go stale and
   stop pipelining properly for months without anyone realizing it.

						-Matt