stagedir vs. jail

Sun Oct 13 14:31:12 UTC 2013

On 14/10/2013 12:19 AM, Ekkehard Gehm wrote:
> A quick addition:
>
> My Jail is buid exactly as discribed in the jail handbook:
> http://www.freebsd.org/doc/handbook/jails-application.html
>
> So hanging around with this issue because of this staging thing is a bit *argl*
>
> Cheers,
>
> Ekki
>
> On Sun, 13 Oct 2013, Ekkehard Gehm wrote something like:
>
>> Ahoi!
>>
>> On Sun, 13 Oct 2013, Matthew Seaman wrote something like:
>>
>>> On 13/10/2013 11:58, Ekkehard Gehm wrote:
>>>> I resently have some problems installing/updating ports. The main
>>>> problem is that the stagedir path is somehow messed up.
>>>>
>>>> The system is running FreeBSD 9.1-RELEASE-p6 and I'm working in a
>>>> jail. While building it creates the stagedir in
>>>> /s/portbuild/usr/ports/...../usr/local/ BUT: When it comes to the pkg
>>>> building it suddently is looking in .../usr-local/.. witch acctualy
>>>> is a part of the symlink. (In the jail /usr/local is a symlink to
>>>> /s/usr-local). Resulting in an failure.
>>>>
>>>> The only workiaround  is disabling stage completly thru make.conf... 
>>>> As this is very quick'n'dirty I wonder if there is any other
>>>> solution!
>>> Your subject line is (perhaps) a bit misleading: this seems to be
>>> nothing specific to the use of jails, but due to having sym-links in
>>> various paths.  It could happen just the same if you laid out your host
>>> filesystem using sym-links.
>>>
>> Right. That might be...
>>
>>> Anyhow, this looks like a bug to me -- using sym-links to put your
>>> filesystem together should not result in chaos.
>>>
>>
>> This is alo my understanding of this issue...
>>
>>> Are you using pkg(8)?  There are differences in the way a package is
>>> generated from the staging directory between pkg(8) and pkg_create(1)
>>> which might make all the difference.
>>>
>> Here is a short cut out of a build (portmaster portmaster):
>>
>> install  -o root -g wheel -m 444 /s/portbuild/usr/ports/ports-mgmt/portmaster/wo                                                      rk/freebsd-portmaster-7359019/files/bash-completions  /s/portbuild/usr/ports/por                                                      ts-mgmt/portmaster/work/stage/usr/local/etc/bash_completion.d/portmaster.sh
>> ====> Compressing man pages
>> ===>  Building package for portmaster-3.17.2
>> Creating package /s/portbuild/usr/ports/ports-mgmt/portmaster/work/portmaster-3.                                                      17.2.tbz
>> Registering depends:.
>> Creating bzip'd tar ball in '/s/portbuild/usr/ports/ports-mgmt/portmaster/work/p                                                      ortmaster-3.17.2.tbz'
>> tar: could not chdir to '/s/portbuild/usr/ports/ports-mgmt/portmaster/work/stage                                                      /s/usr-local'
>>
>> pkg_create: make_dist: tar command failed with code 256
>> *** [do-package] Error code 1
>>
>> Stop in /usr/ports/ports-mgmt/portmaster.
>> *** [install] Error code 1
>>
>> Stop in /usr/ports/ports-mgmt/portmaster.
>>
>> ===>>> A backup package for portmaster-3.17.1 should
>>        be located in /var/ports/packages/portmaster-backup
>>
>> ===>>> Installation of portmaster-3.17.2 (ports-mgmt/portmaster) failed
>> ===>>> Aborting update
>>
>> ===>>> Killing background jobs
>> Terminated
>>
>> ===>>> You can restart from the point of failure with this command line:
>>        portmaster <flags> ports-mgmt/portmaster
>>
>> ===>>> Exiting
>>
>> As a result portmaster is not working anymore...
>>> Is it all ports that are affected?  Does the use of PLIST_FILES or
>>> PLIST_DIRS in the port Makefile make any difference compared to having a
>>> pkg-plist file?
>>>
>>> A possible work-around: instead of sym-linking /s/usr-local ->
>>> /usr/local use a nullfs mount instead.
>>>
>>>     mount -t nullfs /s/usr-local /usr/local
>>>
>>> (Actually, you'ld probably do that from outside the jail so adjust the
>>> paths accordingly.)
>>>
>>> Another workaround:  set up your own poudriere instance to build
>>> packages for all your jails.  poudriere(8) will create its own jails to
>>> do package building in, and manage paths etc. itself.  It's a natural
>>> partner to pkg(8) and zfs(8) but it will work without either of those.
>>>
>>> 	Cheers,
>>>
>>> 	Matthew
>>>
>>> -- 
>>> Dr Matthew J Seaman MA, D.Phil.
>>> PGP: http://www.infracaninophile.co.uk/pgpkey
>>>
>>>
>>
>>
>> -- 
>> Ekkehard Gehm           *       mailto:gehm at doom-labs.net
>> Doom-Labs Inc.          *       http://www.doom-labs.net
>> Frag Content            *       PGP-Key: http://www.physik.tu-berlin.de/~gehm/pubkey.asc
>>
>> Experience is what you get when you were expecting something else.
>>
>> Microsoft:      "Where do you want to go today?"
>> Linux:          "Where do you want to go tomorrow?"
>> FreeBSD:        "Are you guys coming or what?"
>
>
My setup is slightly different, where I do use as Matthew recommends, 
nullfs is used without symlinks; and pkg_* and portmaster are used for
the build process.  We've been using jails to build ports for different
architectures for years, the builds are currently on a FreeBSD 9.2Stable
(Built from source on Oct 8) system.

Building all ports after a complete wipe,  using the portmaster commands
fails due to missing file
+ portmaster --no-term-title --no-confirm -H -K -D -g -G -B -v -m
__MAKE_CONF=/etc/make_P3.conf -m -DBUILD__PRODUCTION mail/sendmail
tar: etc/rc.d/saslauthd: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
pkg_create: make_dist: tar command failed with code 256

In fact the file does exist:
# ls -lh /usr/staging/usr/local/etc/rc.d/saslauthd
-r-xr-xr-x  1 root  wheel   940B Oct 14 00:55
/usr/staging/usr/local/etc/rc.d/saslauthd
and not in /usr/local/etc/rc.d/

As you can see sendmail builds saslauthd as a prerequisite.  The build
process also fails in the same way, missing file in etc/rc.d/... for
samba36, samba4, isc-dhcp42-server and cyrus-sasl2-saslauthd, but works
correctly for some 40 other packages.

The make.conf includes:
WRKDIRPREFIX=/var/ports
DISTDIR=/distfiles
TMPDIR=/tmp
PACKAGES=/packages
STAGEDIR=/usr/staging
FAVORITE_COMPILER=gcc
DEFAULT_VERSIONS=perl5=5.16 python=2.7 python2=2.7 apache=22
DISABLE_LICENSES=yes  # Not that I want to, only that it prevented the
build (a PR existed for this)
WITH_CCACHE_BUILD=yes
MAKE_JOBS_UNSAFE=yes

I suspect the problem lies within the if/then block ".if
!target(install-rc-script)" in bsd.port.mk, but lack the know-how to
proceed further.

The same problem, missing etc/rc.d/$FILE arises on both the i386 and
amd64 architectures (once these work, we move to others).

I suspect that Ekkehard issue and mine are probably related.  We're
both: in a  jail environment using pkg_* tools and portmaster.   (Though
I do test everything using make, to eliminate portmaster as a cause
before emailing or creating a PR).

Finally prior to a build, the following steps are take for a clean rebuild
pkg_delete -a
rm -R $WRKDIRPREFIX/* $STAGEDIR/*
ccache -C && ccache -z

Any advice appreciated.
Dewayne.
GMT+10 (so goodnight)