portmaster, portupgrade, etc

[Don Lewis]

On  4 Oct, Steve Kargl wrote:
> On Wed, Oct 04, 2017 at 05:29:14PM -0500, Mark Linimon wrote:
>> Please understand that I'm not trying to be obstinate,
>> I'm trying to understand.
> 
> Me too.
> 
>> Background: years ago I managed the cluster of i386 blades
>> that we used in package building.  933MHz and 512MB IIRC.
>> So I am familiar with constraint problems.
> 
> The system in question is my last i686 laptop, which I 
> use for libm development and testing.  Once I cannot use
> that laptop (whether hardware failure or inability to 
> update the installed ports), I'll stop worrying about a
> functional libm on 32-bit hardware.

As an aside, this sort of thing could be done in an i386 VM or maybe an
i386 jail on amd64 hardware.

>> On Wed, Oct 04, 2017 at 02:22:25PM -0700, Steve Kargl wrote:
>> > Can't find info on whether jails can be avoided.
>> 
>> I have not checked the code but IIRC, no.  I thought jails
>> had low memory overhead, though.
> 
> That's good, but memory overhead isn't the problem with a
> jail.  It's the diskspace used to duplicate everything
> already available in /bin:/usr/bin:/usr/local/bin:... and
> storage to hold the packages as things get built.
> 
>> > If you only have 1 Gb of memory and 5-10 GB diskspace,
>> > then using poudriere with zfs and jails is a nonstarter.
>> 
>> For point of comparison, with those constraints, I do not
>> understand how modern llvms can build at all.
>> 
>> What happens if you use the manual approach on this same
>> system?  e.g.
>> 
>>   cd /usr/ports/devel/llvm40
>>   mkdir -p /usr/ports/packages
>>   make && make package
>>   pkg install /usr/ports/packages/llvm<whatever>.txz
> 
> I normally do 
> 
> portmaster -Byd devel/llvm40
> 
> and wait a day for the build to complete.  I have 
> /usr/ports/distfiles symlinked to a USB2 /mnt/distfiles
> (40 MB/s max throughput).  I may also need to set
> DISABLE_MAKE_JOBS="YES", but can't remember offhand.
> 
>> 
>> Do you still run out of resources?
>> 
> 
> No. I have 4 GB of swap space, which is well used during the
> build.  The system is actually quite usable as portmaster
> runs.  I also build libreoffice and octave on the system.

That must be fun ...

At some point you'll run into the problem that even one copy of the
ports that you want to install and just one copy of all their
dependencies will no longer fit and then you have to start tossing
out stuff that you would really like to install and use.  I'm pretty
sure that I no longer have enough room to build all the stuff on my
laptop that I currently have installed (openoffice warns that it needs
11GB of free space). In that case, if you move the package building off
to another machine, you don't even have to install many of the build
dependencies on the target machine.  For instance, I have a list of 214
ports that I want to install on my desktop machine. When I feed that
into poudriere, it builds something on the order of 1900 ports.  When I
do the pkg install on my desktop machine with the same list, only 1431
packages get installed.

>> In that case, there's not much that can be done.  The
>> compilers, the office suites, and certain math packages
>> are huge beasts.  However you try to build them won't
>> matter.
>> 
>> I would think having a copy of the llvm workfiles in a jail
>> is going to be equivalent to having them in /tmp?
>> 
>> I must be missing something.
> 
> portmaster satisfies its dependencies from already installed
> ports in /usr/local.  There isn't a clean room full of pkg'd
> dependencies sitting in some jail while llvm40 builds.

The llvm40 problem should be helped a lot by subpackages.  If you
install a package that was built by llvm40 and uses its shared
libraries, hopefully pkg will install a subpackage that contains the
llvm40 .so files and not all of the compiler bits.  You would only get
those if you manually installed llvm40 so that you could use the
compiler bits.  In that case, poudriere would only cause two copies of
the .so files to be installed and the compiler bits would only be
installed on the machine (in the build jail) when poudriere was using a
port that needed the llvm40 compiler.  The same will be true of other
things that are installed because of LIB_DEPENDS.  Things like the
include files, docs, and static libraries won't get automatically
installed just because a port is a dependency of something else.  They
would only need to be manually installed if you want to build something
manually outside of the ports framework.