unexpected idprio 31 behavior on 9.2-BETA2 and 9.2-RC1

[Eric van Gyzen]

On 08/06/2013 10:31, Eric van Gyzen wrote:
> On 08/05/2013 16:15, Dave Mischler wrote:
>> I have an i5-2500 machine 8GB RAM now running 9.2-RC1 amd64 with the
>> GENERIC kernel. Today, while still running 9.2-BETA2, I updated my
>> source tree and started building world with idprio 31 and I looked back
>> a while later and all the CPU cores and disk were essentially idle, and
>> hardly any progress had been made on the build. I stopped and restarted
>> the build without the idle priority setting and it ran fine. Anybody
>> else seen any of this? Anybody know about any fairly recent changes that
>> might account for it?
>>
>> I did a "rm -rf /usr/src /usr/obj" and loaded a new source tree before
>> going to RC1.  I still see odd behavior at RC1.  Sometimes it works just
>> like it should (i.e. compute bound processes use most/all of the
>> available CPU time), but a lot of the time both the CPU and disk are
>> idle (e.g. CPU 97.8% idle, disk 1% busy per systat).  I don't think I
>> ever saw this behavior before while running "make buildworld -j4".  Can
>> anyone else confirm/rebut my findings?  Thanks.
> I can confirm your findings, on 9.1-RELEASE-p5 amd64 GENERIC.
>
> I ran
>
>     $ idprio 31 make buildworld buildkernel > /tmp/build.log 2>&1 <
> /dev/null &
>
> on an otherwise idle amd64 system with 4 CPUs.  The first command in the
> build.log file:
>
>     rm -rf /usr/obj/home/freebsd/tmp
>
> took over three minutes.  It should have taken about three /seconds/.
>
> "uptime" reported a load average of around 1.00.
> "top" showed no threads (user or kernel) using CPU.
> "iostat" showed an average of less than 20 tps on ada0.
> "rm" was usually in the RUN state.
>
> /home/freebsd (src) is UFS+SUJ.
> /usr/obj is UFS+SU.
> /tmp/build.log is tmpfs.
>
> Both UFS file systems are on ada0:
>
> ada0 at ata2 bus 0 scbus2 target 0 lun 0
> ada0: <WDC WD2502ABYS-18B7A0 02.03B05> ATA-8 SATA 2.x device
> ata2: <ATA channel> at channel 0 on atapci0
> atapci0: <Intel 5 Series/3400 Series PCH SATA300 controller>
>
> CPU: Intel(R) Xeon(R) CPU           X3430  @ 2.40GHz (2394.04-MHz
> K8-class CPU)
> real memory  = 8589934592 (8192 MB)
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> FreeBSD/SMP: 1 package(s) x 4 core(s)
>
> FreeBSD 9.1-RELEASE-p5 #0 r+94f2ad5: Tue Aug  6 09:40:22 CDT 2013    
> root at srv5:/usr/obj/home/freebsd/sys/GENERIC  amd64
>
> /boot/loader.conf contains:
>
>     console="comconsole vidconsole"
>     comconsole_speed="115200"
>     comconsole_port="0x2f8"
>
> /etc/sysctl.conf is empty.
>
> I'll update to releng/9.2 and try again.

I see more-or-less the same behavior on 9.2-RC1 (r253912).  It seems to
be faster than 9.1, but it's still much slower than I would expect.

idprio 30 is much, much faster than 31.  It's about as fast as I would
expect (for this idle machine).  So, the problem seems to affect only
idprio 31.  (Off-by-one / fencepost problem?)

CPU-bound processes, such as c++, seem to run at the normal speeds, so
the problem seems to affect system- or I/O-bound work.

Can anyone try this on a 9.0-RELEASE system?

Eric