svn commit: r44734 - head/en_US.ISO8859-1/books/handbook/cutting-edge

Wed May 7 17:57:58 UTC 2014

On May 6, 2014, at 9:09 PM, Benjamin Kaduk <kaduk at MIT.EDU> wrote:

> Warner, can you please double-check the facts about make variables?
> I am not confident I have the new world order correct.
> 
> On Thu, 1 May 2014, Dru Lavigne wrote:
> 
>> 
>> Modified: head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml
>> ==============================================================================
>> --- head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml	Thu May  1 18:12:22 2014	(r44733)
>> +++ head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml	Thu May  1 18:50:44 2014	(r44734)
>> @@ -1396,6 +1396,25 @@ before running "/usr/sbin/freebsd-update
>> 	sequence  described in
>> 	the following procedure.</para>
>> 
>> +      <note>
>> +      <para>It is a good idea to save the output from running
>> +	<command>make</command> to a file.  If something goes wrong, a copy of
>> +	the error message can be posted to one of the &os; mailing
>> +	lists.</para>
>> +
>> +      <para>The easiest way to do this is to use <command>script</command> with a
>> +	parameter that specifies the name of the file to save all
>> +	output to.  Do not save the output to
>> +	<filename>/tmp</filename> as this directory may be cleared at
>> +	next reboot.  A better place to save the file is
>> +	<filename>/var/tmp</filename>.  Run this command immediately before rebuilding
>> +	the world, and then type <userinput>exit</userinput> when the
>> +	process has finished:</para>
>> +
>> +      <screen>&prompt.root; <userinput>script <replaceable>/var/tmp/mw.out</replaceable></userinput>
>> +Script started, output file is /var/tmp/mw.out</screen>
>> +      </note>
>> +
> 
> (I mentioned on IRC that this could be "script /var/tmp/mw.out make buildworld", but only on BSD boxes -- Linux's script(1) has a different syntax.  This is more portable as-is, so I don't think we should change it.)

make buildworld |& tee foo.log

is what I use, but that’s csh/tcsh syntax and only catches one command. when I have to do multiples, I’ll add -a to the tee command. Sometimes I use emacs’ shell to get the same effect.

>> @@ -1597,140 +1620,70 @@ before running "/usr/sbin/freebsd-update
>> 	behavior of the <filename>Makefile</filename> is controlled by
>> -	these variables.  These are the same variables as are set in
>> -	<filename>/etc/make.conf</filename>, and this provides
>> -	another way of setting them.  For example:</para>
>> +	variables.  These can either be set in
>> +	<filename>/etc/make.conf</filename> or they can be specified
> 
> I think we must also mention /etc/src.conf here, in the file entitled "make world”.

src.conf is used for all builds in /usr/src (no matter where /usr/src is located). If you are building things like ls from hand, you’ll need to also define MAKESYSPATH to be/have “…/share/mk” in it.

>> +	when using <command>make</command>.  For example, this
>> +	variable specifies that profiled libraries
>> +	should not be built:</para>
>> 
>>      <screen>&prompt.root; <userinput>make -DNO_PROFILE <replaceable>target</replaceable></userinput></screen>
> 
> Using the NO_FOO form of variables is highly deprecated at this point, at least in HEAD.  (I think we're trying to deprecate it on the other branches, too.)  While reviewing some of Warner's commits to look at the current state of things, I saw a comment in src.opts.mk that "Old instances [of NO_FOO] should be removed since they were just to bridge the gap between FreeBSD 4 and FreeBSD 5."  The normal way to disable the building of profiled libraries would be to set WITHOUT_PROFILE in /etc/src.conf; I think that -DWITHOUT_PROFILE on the command line should also work, but would like confirmation from Warner.

In 11 and forward, his is correct. Set WITHOUT_PROFILE=yes somewhere and all profiling is disabled. In some rare cases you may need to set MK_PROFILE=no. This replaces the NO_FOO syntax that was supposed to be a stop-gap measure between 4 and 5, but the gap never stopped. NO_FOO is limited to CTF, DEBUG_FILES, INSTALLLIB, MAN and profile, and so was irregular with the rest of things. A few more NO items may be added to the list in my next round of changes, but that’s likely a ways away.

WITHOUT_PROFILE should work in 10 and earlier too, but that should be tested. I don’t plan on MFCing the NO_foo deprecation warning, and in fact realize that it is premature in head.

>> -      <para>is another way of specifying that profiled libraries
>> -	should not be built, and corresponds with the</para>
>> +      <para>It corresponds with this setting in
>> +	<filename>/etc/make.conf</filename>:</para>
>> 
>>      <programlisting>NO_PROFILE=    true     #    Avoid compiling profiled libraries</programlisting>
> 
> (Per the above, this is WITHOUT_PROFILE=true in /etc/src.conf.  Maybe we should note that it doesn't matter if it's set to true or false of arglebarglepants; the fact that it's set is the only thing that matters. The inverse of WITHOUT_PROFILE is setting WITH_PROFILE explicitly.)

Yes.

>> -      <para>line in <filename>/etc/make.conf</filename>.</para>
>> -
> [...]
>> -      <para>On a multi-CPU machine using an SMP configured kernel, try
>> -	values between 6 and 10 and see how they speed things
>> +      <screen>&prompt.root; <userinput>make -j4 buildworld</userinput></screen>
>> +
>> +      <para>On a multi-<acronym>CPU</acronym> machine, try
>> +	values between <literal>6</literal> and <literal>10</literal> to see how they speed things
> 
> -j10 is suboptimal on a 32-core machine, surely.  The last advice I remember seeing on the mailing lists was that the limit was about 1.5 times the number of cores.

Somewhere between 1.5x and 2.0x is where we saturate. 10 used to be a very optimistic number, in that the typical parallelism rarely got above 5 jobs running at any time. Now, however, running around #CPUs jobs tends to approach optimal, although some small but worthwhile gains can be had at 1.5 to 2.0 times the number of CPUs, especially if hyper threading is disabled. However, this massive parallelism is relatively recent, and problems exist starting in the -j 15-20 range and getting really bad in the 30 and up range.

I’ll be hacking a bit in this area for some time to come, so maybe it would be good to do a review when I’m done?

Warner

> -Ben
> 
>> 	up.</para>
>> 
>>      <indexterm>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/svn-doc-all/attachments/20140507/015b4628/attachment.sig>