poudriere merging multiple ports trees

Fri Jan 29 16:38:36 UTC 2021

Greetings,

i created a simple patch for this, this may not be the best solution but 
my tests seems to be successful to get the desired result:

Without NOT_MOVED:

=>> Debug: Reading /usr/local/etc/poudriere.conf
[00:00:00] Creating the reference jail... done
[00:00:02] Mounting system devices for j12p64-default-server
[00:00:02] Mounting ports/packages/distfiles
[00:00:02] Using packages from previously failed build: 
/home/poudriere/data/packages/j12p64-default-server/.building
[00:00:02] Mounting ccache from: /home/poudriere/ccache
[00:00:02] Mounting packages from: 
/home/poudriere/data/packages/j12p64-default-server
[00:00:02] Copying /var/db/ports from: 
/usr/local/etc/poudriere.d/j12p64-options
[00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/server-make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/default-make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/j12p64-make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/default-server-make.conf
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools
/etc/resolv.conf -> 
/home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf
[00:00:02] Starting jail j12p64-default-server
[00:00:02] Logs: 
/home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_11h14m59s
[00:00:02] WWW: 
http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_11h14m59s
[00:00:02] Loading MOVED for 
/home/poudriere/data/.m/j12p64-default-server/ref/usr/ports
[00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:03] Gathering ports metadata
[00:00:03] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has expired: 
Ancient and unsupported release
[00:00:03] Error: Fatal errors encountered gathering initial ports metadata
[00:00:03] Cleaning up
[00:00:03] Unmounting file systems

With NOT_MOVED:

=== root at dsspkg (pts/1) /home/poudriere/ports/dss_overlay 35 -> cat 
NOT_MOVED
sysutils/sge62

=== root at dsspkg (pts/1) /home/poudriere/ports/dss_overlay 97 -> 
poudriere bulk -vv -O dss_overlay -j j12p64 -z server -p default 
sysutils/sge62
=>> Debug: Reading /usr/local/etc/poudriere.conf
[00:00:00] Creating the reference jail... done
[00:00:02] Mounting system devices for j12p64-default-server
[00:00:02] Mounting ports/packages/distfiles
[00:00:02] Using packages from previously failed build: 
/home/poudriere/data/packages/j12p64-default-server/.building
[00:00:02] Mounting ccache from: /home/poudriere/ccache
[00:00:02] Mounting packages from: 
/home/poudriere/data/packages/j12p64-default-server
[00:00:02] Copying /var/db/ports from: 
/usr/local/etc/poudriere.d/j12p64-options
[00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/server-make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/default-make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/j12p64-make.conf
[00:00:02] Appending to make.conf: 
/usr/local/etc/poudriere.d/default-server-make.conf
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): comms/chan_capi
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-kmod
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): comms/isdn4bsd-utils
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): sysutils/env4801
[00:00:02] Warning: Blacklisting (from 
/usr/local/etc/poudriere.d/j12p64-blacklist): security/ipsec-tools
/etc/resolv.conf -> 
/home/poudriere/data/.m/j12p64-default-server/ref/etc/resolv.conf
[00:00:02] Starting jail j12p64-default-server
[00:00:03] Logs: 
/home/poudriere/data/logs/bulk/j12p64-default-server/2021-01-29_17h28m52s
[00:00:03] WWW: 
http://dsspkg.incore/poudriere//build.html?mastername=j12p64-default-server&build=2021-01-29_17h28m52s
[00:00:03] Loading MOVED for 
/home/poudriere/data/.m/j12p64-default-server/ref/usr/ports
[00:00:03] Loading NOT_MOVED from overlaydir: 
/home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay
[00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:03] Gathering ports metadata
[00:00:03] Debug: sysutils/sge62 found in 
/home/poudriere/data/.m/j12p64-default-server/ref/overlays/dss_overlay/NOT_MOVED, 
skipped checking MOVED File of master portstree
[00:00:03] Debug: queueing sysutils/sge62 into gatherqueue (rdep=listed)
[00:00:03] Debug: Processing gatherqueue
[00:00:03] Warning: (sysutils/sge62): Debug: gather_port_vars_port 
(sysutils/sge62): LOOKUP
[00:00:03] Warning: (sysutils/sge62): Debug: deps_fetch_vars: discovered 
sysutils/sge62 is sge-6.2.2.1_6
[00:00:03] Warning: (sysutils/sge62): Debug: WILL BUILD sysutils/sge62

Patch leading to this:

--- /usr/local/share/poudriere/common.sh.1st    2021-01-27 
15:44:15.000000000 +0100
+++ /usr/local/share/poudriere/common.sh        2021-01-29 
17:27:14.810441000 +0100
@@ -6815,8 +6815,11 @@
                         continue
                 fi
                 origin_listed="${origin}"
-               if shash_get origin-moved "${origin}" new_origin; then
-                       if [ "${new_origin%% *}" = "EXPIRED" ]; then
+               # check against the overlay list, skip port if its in there
+               if (grep -q "${origin}" 
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED); then
+                       msg_debug "${origin} found in 
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED, skipped checking 
MOVED File of master portstree"
+               elif shash_get origin-moved "${origin}" new_origin; then
+                       if [ "${new_origin%% *}" = "EXPIRED" ] ; then
                                 msg_error "MOVED: ${origin} ${new_origin}"
                                 set_dep_fatal_error
                                 continue
@@ -7085,6 +7088,10 @@
         fi
         [ -f ${MASTERMNT}${PORTSDIR}/MOVED ] || return 0
         msg "Loading MOVED for ${MASTERMNT}${PORTSDIR}"
+       # Respect overlays, ${OVERLAYS} has a leading whitepace
+       if [ -f ${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }/NOT_MOVED ]; then
+               msg "Loading NOT_MOVED from overlaydir: 
${MASTERMNT}${OVERLAYSDIR}/${OVERLAYS# }"
+       fi
         bset status "loading_moved:"
         awk -f ${AWKPREFIX}/parse_MOVED.awk \
             ${MASTERMNT}${PORTSDIR}/MOVED | \

Maybe someone else could test this, seems to work for me.


Kind regards,

Henrik Rosenke


Am 28.01.21 um 21:56 schrieb Henrik Rosenke:
>
> Am 28.01.21 um 21:14 schrieb Miroslav Lachman:
>> On 28/01/2021 17:47, Henrik Rosenke wrote:
>>> I tried this with a patched MOVED file, copied from the master 
>>> portstree, deleted the entry and tried to build but this File seems 
>>> to be ignored. I dont really understand where the MOVED file is 
>>> handled, it seems to be bsd.port.subdir.mk but i am not sure.
>>
>> MOVED is handled by bsd.port.subdir.mk but overlays do not account 
>> with it. Overlays are handled on more places
>
> i thought this too but it seems poudriere is canceled before this, i 
> even tried to move the bsd.port.subdir.mk and commented out the 
> function load_moved in the common.sh of poudriere and can start a 
> build. Correct me if iam wrong but it seems that the common.sh from 
> poudriere needs to be patched to handle this rather than the Mk files 
> itself. I added set -x in the common.sh to get a better understanding 
> what is happening and it seems that shash_set sucks in the MOVED file 
> and this is checked later against the ports we want to build:
>
> + shash_set origin-moved sysutils/sge62 'EXPIRED 2013-06-05 Has 
> expired: Ancient and unsupported release'
>
> later:
>
> + originspec_decode sysutils/sge62 origin '' flavor
> + local -
> + set +x -f
> + [ -n '' ]
> + origin_listed=sysutils/sge62
> + shash_get origin-moved sysutils/sge62 new_origin
> + local -
> + set +x
> + [ EXPIRED '=' EXPIRED ]
> + msg_error 'MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has expired: 
> Ancient and unsupported release'
> + local -
> + set +x
> [00:00:07] Error: MOVED: sysutils/sge62 EXPIRED 2013-06-05 Has 
> expired: Ancient and unsupported release
>
> Kind regards,
>
> Henrik Rosenke
>
>>
>> Scripts/do-depends.sh
>> Scripts/depends-list.sh
>>
>> Next is bsd.port.mk:
>>
>> .if defined(USE_LOCAL_MK)
>> .include "${PORTSDIR}/Mk/bsd.local.mk"
>> .endif
>> .for odir in ${OVERLAYS}
>> .sinclude "${odir}/Mk/bsd.overlay.mk"
>> .endfor
>>
>> But there is no bsd.overlay.mk and I don't know what should be in it 
>> (file in the overlay ports tree)
>>
>> The AWK code in bsd.port.subdir.mk can be changed to use MOVED from 
>> overlay if it exists insead of the default one.
>>
>>> I think the best way to handle this would be a NOT_MOVED file in the 
>>> overlay portstree with the ports to ignore from the MOVED file with 
>>> entries like "sysutils/sge62".
>>
>> This can be possible too. NOT_MOVED can be used as additional INDEXFILE
>>
>> Or maybe start with PORTSEARCH_MOVED=0 (not tried yet)
>>
>> Kind regards
>> Miroslav Lachman
