CURRENT: make -jX buildworld doesn't work

[Ian Lepore]

On Wed, 2012-03-14 at 18:08 +0100, O. Hartmann wrote: 
> On 03/14/12 16:08, Ian Lepore wrote:
> > On Wed, 2012-03-14 at 14:47 +0100, O. Hartmann wrote:
> >> This is no compalin, since make buildworld works with one thread.
> >>
> >> But I'd like to report a funny thing I witnessed.
> >>
> >> On two boxes equipted with Core2Dou CPUS (E8500,  2 cores/threads,
> >> Q6600, 4 cores/threads) a parallel make buildworld works fine with the
> >> most recent sources of FreeBSD 10.0-CURRENT/amd64 (both boxes have 8 GB
> >> RAM, both boxes use a very close/similar setup and configuration).
> >>
> >> I moved in my lab towards a brand new Sandy-Bridge-E box with 32GB RAM
> >> and the CPU is a Core i7-3930X with six cores/12 threads. On this box,
> >> even "make buildworld" with -j2 fails to build. It builds fine with a
> >> vanilla "make buildworld".
> >>
> >> Also funny is, that even with only one thread, the 3 GHz Core2Duo
> >> methusalem systems "outperform" in compile time the 3,2 GHZ driven Intel
> >> youngster.
> >>
> >> Maybe there is an issue in FreeBSD 10 with the TURBO BOOST? I'm a bit
> >> time constraint now, but I'm willing to do some tests with advices from
> >> the experts.
> >>
> >> Lets give you some informations I think it could be valuable. Please
> >> request more, if this isn't sufficient.
> >>
> >> I realize/I'm aware that both hardware and OS are brandnew! But
> >> hopefully this is something important and could be "fixed" - if my
> >> observation is a real observation ...
> >>
> >> Regards,
> >> Oliver
> 
> [SNIP]
> 
> > 
> > There was a change (r232793) a few days ago to make turbo boost work,
> > more info in this thread:
> > 
> > http://lists.freebsd.org/pipermail/freebsd-current/2012-March/032434.html
> > 
> > I wasn't able to get it to work just by tweaking the rc.conf knobs, but
> > I suspect the reason may be because I have non-standard devd.conf. I
> > worked around it and haven't had time to look for the cause yet.
> > 
> > In trying to explain the compile time differences between two systems,
> > one of the first things I'd look at would be differences in the disks.
> > In my experience, IO performance has as big an influence on build time
> > as processor speed and number of cores.
> > 
> > I just updated my -current sources and did a fresh buildworld and
> > buildkernel using both -j2 and -j12 and had no problems on a 6-core
> > Xeon.  I know "make xdev" fails with -jN but I haven't seen failures on
> > any other targets.  There was a checkin recently that added some .ORDER
> > stuff for -jN but it only affected building usr.sbin/acpi.
> > 
> > -- Ian
> 
> The change has already been used. But no change.
> 
> The disks are on all systems the same, but the new box has SATA 6GB and
> the WD 640GB "Caviar black" disk claims also be SATA 6GB.
> You're right, disk I/O has a great impact. And I realized that FreeBSD
> 9/10 have a big problem with the Patsburg-based X79 chipset, as far as I
> can see - or it is simply a coincidence. Since the disks are attached to
> the X79 chipset's SATA 6GB port, I have sometimes strange elongated
> access times. When doing a simple diskper measurement, the raw
> performance of the disk reveals itself as slightly better than with SATA
> 3GB.
> 
> By the way, my last buildworl took 1 hour. The buildworld before 3
> hours. Same load, same box, same OS revision. funny.
> 
> I stay tuned. At the time, it doesn't bother me much. I thought it is
> just worth to be mentioned ...
> 
> Regards,
> 
> Oliver
> 

Just to give you a point of comparison...  My build machine is a 6-core
Xeon W3680 running at 4.25GHz (yes, that's overclocked from stock 3.3),
hyperthreading disabled, 12G ram, and all builds are done using
filesystems on SSD drives connected to a SATA-2 controller.
Using /dev/null for src.conf and make.conf so that it's a completely
stock build, I get these build times for a "make buildworld buildkernel"

  -j1   63 minutes
  -j2   35 minutes
  -j6   19 minutes
  -j12  18 minutes

I wonder if your 1hr/3hr difference is due to caching?  If so, that
would seem to point to big trouble with disk performance.  I re-did the
-j6 build after unmounting/remounting the filesystem to clear the cache
and the difference was less than 30 seconds, but that's with SSD drives.

Hmmm, I just realized I'm using MODULES_OVERRIDE="agp drm" so it's not
quite a completely-stock/GENERIC build.

-- Ian