From nobody Sat Dec 18 21:37:17 2021 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B55C718F6815; Sat, 18 Dec 2021 21:37:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JGfMK3T5nz3lPq; Sat, 18 Dec 2021 21:37:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 535B9167CA; Sat, 18 Dec 2021 21:37:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1BILbHX7099704; Sat, 18 Dec 2021 21:37:17 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1BILbHtc099703; Sat, 18 Dec 2021 21:37:17 GMT (envelope-from git) Date: Sat, 18 Dec 2021 21:37:17 GMT Message-Id: <202112182137.1BILbHtc099703@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: "Simon J. Gerraty" Subject: git: 3b26e5a453df - main - Update dirdeps.mk et al to 20211212 versions List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: sjg X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 3b26e5a453dfd7fa96368a745e8f66b3ed911a9e Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1639863437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=51u8MZyU/b7oVADmoRzsJnotjXoYbj6lGRXkBYZlSFw=; b=jJh9mb4hv76K/zPa1cP2VFAFFG6+y/I2EvEdAtE9hsg/BDYSTH5heb6f9nwRVUqQDleme8 j2753RqCTeLisACErJwUrMmbhUu53vVYoPD7gy1RwzW6JZQWAam3mdqHs01l1mAK486Rdz z7uyTCxXcfNppGsBFIlpXBWjVEZ2xWpZtg0tVhiokjJuPZsTMZOTM3Z/54yBM2S9sQ3hbt qKGuSGxNQAxDAeZRykJSPad4vgs+OVbBmlenOOGNjKKV/vc6bISDf4PNv9Vm8KvlNJfb+G 0mHcQ95Yrr5bJFU+MsgMlFxI0VOQ5fyVxXAbep8ch312YrCmlxtHV+nFQyg85w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1639863437; a=rsa-sha256; cv=none; b=axkmbX6LbhzlEep302f+m1POyOUnq0N/jMyP8N3fjb2c//FoybENBuPxyc+kiVhUd0klzm B87JIb61k9FgBRpOzL4R+ZQV2KKUCypLI0qR373eDQccOGaXCXdlIHJra1NHQOvLTV0Oqn jIwjraAbHspeWKn1hYl4kWSs/Ma1CMzfAfLuMAAn37CDpj9iUdRHIP8tjkBXlvZZ6NtH1q xygRAxSJKmlEKlFFiOfChxhN6yNR+tAgTSCdvnNWVFrI0Z/wUNfyDsmgh/luye6WyzWUj9 vc6SfR6vI65T71VRBUmAYk7w2TGW6dwHpWTErXMTsOoJozOE/o9SLAU4aR63tA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by sjg: URL: https://cgit.FreeBSD.org/src/commit/?id=3b26e5a453dfd7fa96368a745e8f66b3ed911a9e commit 3b26e5a453dfd7fa96368a745e8f66b3ed911a9e Author: Simon J. Gerraty AuthorDate: 2021-12-18 21:37:07 +0000 Commit: Simon J. Gerraty CommitDate: 2021-12-18 21:37:07 +0000 Update dirdeps.mk et al to 20211212 versions dirdeps.mk: simplify computation of qualified vs unqualified dirdeps. Note the importance of avoiding unresolved variables in DIRDEPS Fix DIRDEP_LOADAVG_REPORT - expr(1) fails if expression evaluates to 0 Trim ${SRCTOP}/ from debug etc messages to make comparison easier. Include number of DIRDEPS in debug output. Trim ${SRCTOP}/ when exporting to env while building DIRDEPS_CACHE to help avoid env exhaustion. Use DIRDEPS_ALL_MACHINES_FILTER to better handle ALL_MACHINES in complex build environments. dirdeps-options.mk: use separate .undef for each variable dirdeps-targets.mk: allow for '.' in DIRDEPS_TARGETS_DIRS meta.autodep.mk: leverage ${.SUFFIXES} if we can. meta.sys.mk: allow use in META_MODE vs DIRDEPS_BUILD meta2deps.py: compute a list of dirdep extensions that map to current TARGET_SPEC to avoid confusion. Reviewed by: stevek --- share/mk/dirdeps-options.mk | 5 +- share/mk/dirdeps-targets.mk | 7 +- share/mk/dirdeps.mk | 152 +++++++++++++++++++++++++++++--------------- share/mk/meta.autodep.mk | 16 ++++- share/mk/meta.sys.mk | 26 ++++++-- share/mk/meta2deps.py | 79 +++++++++++++---------- 6 files changed, 187 insertions(+), 98 deletions(-) diff --git a/share/mk/dirdeps-options.mk b/share/mk/dirdeps-options.mk index c59a5600923e..bf5bb964ab47 100644 --- a/share/mk/dirdeps-options.mk +++ b/share/mk/dirdeps-options.mk @@ -1,5 +1,5 @@ # $FreeBSD$ -# $Id: dirdeps-options.mk,v 1.17 2020/08/07 01:57:38 sjg Exp $ +# $Id: dirdeps-options.mk,v 1.18 2020/12/22 18:10:34 sjg Exp $ # # @(#) Copyright (c) 2018-2020, Simon J. Gerraty # @@ -60,7 +60,8 @@ DIRDEPS_OPTIONS_QUALIFIER_LIST ?= \ # note that we need to include $o in the variable _o$o # to ensure correct evaluation. .for o in ${DIRDEPS_OPTIONS} -.undef _o$o _v$o +.undef _o$o +.undef _v$o .for x in ${DIRDEPS_OPTIONS_QUALIFIER_LIST} .if defined(MK_$o.$x) _o$o ?= MK_$o.$x diff --git a/share/mk/dirdeps-targets.mk b/share/mk/dirdeps-targets.mk index e65ad89742bd..cd1bcdb65250 100644 --- a/share/mk/dirdeps-targets.mk +++ b/share/mk/dirdeps-targets.mk @@ -1,6 +1,6 @@ # $FreeBSD$ # RCSid: -# $Id: dirdeps-targets.mk,v 1.22 2020/08/15 18:00:11 sjg Exp $ +# $Id: dirdeps-targets.mk,v 1.24 2020/12/11 18:15:43 sjg Exp $ # # @(#) Copyright (c) 2019-2020 Simon J. Gerraty # @@ -42,6 +42,7 @@ .-include # for DIRDEPS_BUILD this is how we prime the pump +# include . to allow any directory to work as a target DIRDEPS_TARGETS_DIRS ?= targets targets/pseudo # these prefixes can modify how we behave # they need to be stripped when looking for target dirs @@ -77,7 +78,7 @@ DIRDEPS_TARGETS_MACHINE_LIST += \ DIRDEPS_TARGETS_MACHINE_LIST := ${DIRDEPS_TARGETS_MACHINE_LIST:O:u} # raw Makefile.depend* list -tdeps != 'cd' ${SRCTOP} && 'ls' -1 ${tdirs:O:u:@d@$d/${.MAKE.DEPENDFILE_PREFIX}*@} 2> /dev/null; echo +tdeps != 'cd' ${SRCTOP} && 'ls' -1 ${tdirs:O:u:@d@$d/${.MAKE.DEPENDFILE_PREFIX}*@:S,^./,,} 2> /dev/null; echo .if ${DEBUG_DIRDEPS_TARGETS:U:Mdep*} != "" .info tdeps=${tdeps} .endif @@ -136,7 +137,7 @@ DIRDEPS := ${DIRDEPS:O:u} # if we got DIRDEPS get to work .if !empty(DIRDEPS) DIRDEPS.dirs := ${DIRDEPS:S,^,${SRCTOP}/,:@d@${exists($d):?$d:${d:R}}@} -# some targets what to tweak options we might want to process now +# some targets want to tweak options we might want to process now .for m in ${DIRDEPS.dirs:S,$,/Makefile.dirdeps.options,} .-include <$m> .endfor diff --git a/share/mk/dirdeps.mk b/share/mk/dirdeps.mk index c7842b49d16b..b0d4a761281e 100644 --- a/share/mk/dirdeps.mk +++ b/share/mk/dirdeps.mk @@ -1,7 +1,7 @@ # $FreeBSD$ -# $Id: dirdeps.mk,v 1.130 2020/11/02 00:34:30 sjg Exp $ +# $Id: dirdeps.mk,v 1.147 2021/12/14 02:09:53 sjg Exp $ -# Copyright (c) 2010-2020, Simon J. Gerraty +# Copyright (c) 2010-2021, Simon J. Gerraty # Copyright (c) 2010-2018, Juniper Networks, Inc. # All rights reserved. # @@ -67,6 +67,10 @@ # processing is recursive and results in .MAKE.LEVEL 0 learning the # dependencies of the tree wrt the initial directory (_DEP_RELDIR). # +# NOTE: given the extent of processing that DIRDEPS undergoes it +# is important that any variables in entries use :U to guard +# against surprises when undefined. +# # TARGET_SPEC_VARS # The default value is just MACHINE, and for most environments # this is sufficient. The _DIRDEP_USE target actually sets @@ -266,24 +270,9 @@ N_notmachine := ${.MAKE.DEPENDFILE_PREFERENCE:E:N*${MACHINE}*:${M_ListToSkip}} # if we were included recursively _DEP_TARGET_SPEC should be valid. .if empty(_DEP_TARGET_SPEC) -# we may or may not have included a dependfile yet -.if defined(.INCLUDEDFROMFILE) -_last_dependfile := ${.INCLUDEDFROMFILE:M${.MAKE.DEPENDFILE_PREFIX}*} -.else -_last_dependfile := ${.MAKE.MAKEFILES:M*/${.MAKE.DEPENDFILE_PREFIX}*:[-1]} -.endif -.if ${_debug_reldir:U0} -.info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: _last_dependfile='${_last_dependfile}' -.endif - -.if empty(_last_dependfile) || ${_last_dependfile:E:${N_notmachine}} == "" -# this is all we have to work with DEP_MACHINE = ${TARGET_MACHINE:U${MACHINE}} _DEP_TARGET_SPEC := ${DEP_TARGET_SPEC} -.else -_DEP_TARGET_SPEC = ${_last_dependfile:${M_dep_qual_fixes:ts:}:E} -.endif -.if !empty(_last_dependfile) +.if ${.INCLUDEDFROMFILE:U:M${.MAKE.DEPENDFILE_PREFIX}*} != "" # record that we've read dependfile for this _dirdeps_checked.${_CURDIR}.${TARGET_SPEC}: .endif @@ -402,8 +391,10 @@ DIRDEP_LOADAVG_LAST = 0 # yes the expression here is a bit complicated, # the trick is to only eval ${DIRDEP_LOADAVG_LAST::=${now_utc}} # when we want to report. +# Note: expr(1) will exit 1 if the expression evaluates to 0 +# hence the || true DIRDEP_LOADAVG_REPORT = \ - test -z "${"${expr ${now_utc} - ${DIRDEP_LOADAVG_INTEVAL:U60} - ${DIRDEP_LOADAVG_LAST}:L:sh:N-*}":?yes${DIRDEP_LOADAVG_LAST::=${now_utc}}:}" || \ + test -z "${"${expr ${now_utc} - ${DIRDEP_LOADAVG_INTEVAL:U60} - ${DIRDEP_LOADAVG_LAST} || true:L:sh:N-*}":?yes${DIRDEP_LOADAVG_LAST::=${now_utc}}:}" || \ echo "${TRACER}`${DIRDEP_LOADAVG_CMD}`" # we suppress SUBDIR when visiting the leaves @@ -413,7 +404,7 @@ DIRDEP_LOADAVG_REPORT = \ _DIRDEP_USE: .USE .MAKE @for m in ${.MAKE.MAKEFILE_PREFERENCE}; do \ test -s ${.TARGET:R}/$$m || continue; \ - echo "${TRACER}Checking ${.TARGET:S,${SRCTOP}/,,} for ${.TARGET:E} ..."; \ + echo "${TRACER}Checking ${.TARGET:S,^${SRCTOP}/,,} for ${.TARGET:E} ..."; \ ${DIRDEP_USE_PRELUDE} \ MACHINE_ARCH= NO_SUBDIR=1 ${DIRDEP_USE_ENV} \ TARGET_SPEC=${.TARGET:E} \ @@ -423,23 +414,72 @@ _DIRDEP_USE: .USE .MAKE done .ifdef ALL_MACHINES -# this is how you limit it to only the machines we have been built for -# previously. .if empty(ONLY_TARGET_SPEC_LIST) && empty(ONLY_MACHINE_LIST) -.if !empty(ALL_MACHINE_LIST) -# ALL_MACHINE_LIST is the list of all legal machines - ignore anything else -_machine_list != cd ${_CURDIR} && 'ls' -1 ${ALL_MACHINE_LIST:O:u:@m@${.MAKE.DEPENDFILE:T:R}.$m@} 2> /dev/null; echo +# we start with everything +_machine_list != echo; 'ls' -1 ${_CURDIR}/${.MAKE.DEPENDFILE_PREFIX}* 2> /dev/null + +# some things we know we want to ignore +DIRDEPS_TARGETS_SKIP_LIST += \ + *~ \ + *.bak \ + *.inc \ + *.old \ + *.options \ + *.orig \ + *.rej \ + +# first trim things we know we want to skip +# and provide canonical form +_machine_list := ${_machine_list:${DIRDEPS_TARGETS_SKIP_LIST:${M_ListToSkip}}:T:E} + +# cater for local complexities +# local.dirdeps.mk can set +# DIRDEPS_ALL_MACHINES_FILTER and +# DIRDEPS_ALL_MACHINES_FILTER_XTRAS for final tweaks + +.if !empty(ALL_TARGET_SPEC_LIST) +.if ${_debug_reldir} +.info ALL_TARGET_SPEC_LIST=${ALL_TARGET_SPEC_LIST} +.endif +DIRDEPS_ALL_MACHINES_FILTER += \ + @x@$${ALL_TARGET_SPEC_LIST:@s@$${x:M$$s}@}@ +.elif !empty(ALL_MACHINE_LIST) +.if ${_debug_reldir} +.info ALL_MACHINE_LIST=${ALL_MACHINE_LIST} +.endif +.if ${TARGET_SPEC_VARS:[#]} > 1 +# the space below can result in extraneous ':' +DIRDEPS_ALL_MACHINES_FILTER += \ + @x@$${ALL_MACHINE_LIST:@m@$${x:M$$m,*} $${x:M$$m}@}@ .else -_machine_list != 'ls' -1 ${_CURDIR}/${.MAKE.DEPENDFILE_PREFIX}.* 2> /dev/null; echo +DIRDEPS_ALL_MACHINES_FILTER += \ + @x@$${ALL_MACHINE_LIST:@m@$${x:M$$m}@}@ +.endif +.endif +# add local XTRAS - default to something benign +DIRDEPS_ALL_MACHINES_FILTER += \ + ${DIRDEPS_ALL_MACHINES_FILTER_XTRAS:UNbak} + +.if ${_debug_reldir} +.info _machine_list=${_machine_list} +.info DIRDEPS_ALL_MACHINES_FILTER=${DIRDEPS_ALL_MACHINES_FILTER} .endif -_only_machines := ${_machine_list:${NIgnoreFiles:UN*.bak}:E:O:u} + +_only_machines := ${_machine_list:${DIRDEPS_ALL_MACHINES_FILTER:ts:}:S,:, ,g} .else _only_machines := ${ONLY_TARGET_SPEC_LIST:U} ${ONLY_MACHINE_LIST:U} .endif .if empty(_only_machines) # we must be boot-strapping -_only_machines := ${TARGET_MACHINE:U${ALL_MACHINE_LIST:U${DEP_MACHINE}}} +_only_machines := ${TARGET_MACHINE:U${ALL_TARGET_SPEC_LIST:U${ALL_MACHINE_LIST:U${DEP_MACHINE}}}} +.endif + +# cleanup the result +_only_machines := ${_only_machines:O:u} + +.if ${_debug_reldir} +.info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: ALL_MACHINES _only_machines=${_only_machines} .endif .else # ! ALL_MACHINES @@ -466,6 +506,10 @@ _only_machines := ${_only_machines:${NOT_TARGET_SPEC_LIST:${M_ListToSkip}}} # clean up _only_machines := ${_only_machines:O:u} +.if ${_debug_reldir} +.info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: _only_machines=${_only_machines} +.endif + # make sure we have a starting place? DIRDEPS ?= ${RELDIR} .endif # target @@ -523,7 +567,7 @@ ${DIRDEPS_CACHE}: .META .NOMETA_CMP ${"${DEBUG_DIRDEPS:Nno}":?DEBUG_DIRDEPS='${DEBUG_DIRDEPS}':} \ ${.MAKEFLAGS:tW:S,-D ,-D,g:tw:M*WITH*} \ ${.MAKEFLAGS:tW:S,-d ,-d,g:tw:M-d*} \ - 3>&1 1>&2 | sed 's,${SRCTOP},$${SRCTOP},g;s,_{,$${,g' >> ${.TARGET}.new && \ + 3>&1 1>&2 | sed 's,${SRCTOP},_{SRCTOP},g;s,_{,$${,g' >> ${.TARGET}.new && \ mv ${.TARGET}.new ${.TARGET} .endif @@ -599,6 +643,7 @@ _build_dirs += ${_machines:@m@${_CURDIR}.$m@} .endif .if ${_debug_reldir} +.info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: nDIRDEPS=${DIRDEPS:[#]} .info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: DIRDEPS='${DIRDEPS}' .info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: _machines='${_machines}' .endif @@ -617,11 +662,10 @@ DEP_DIRDEPS_FILTER = U # this is what we start with __depdirs := ${DIRDEPS:${NSkipDir}:${DEP_DIRDEPS_FILTER:ts:}:C,//+,/,g:O:u:@d@${SRCTOP}/$d@} -# some entries may be qualified with . -# the :M*/*/*.* just tries to limit the dirs we check to likely ones. -# the ${d:E:M*/*} ensures we don't consider junos/usr.sbin/mgd -__qual_depdirs := ${__depdirs:M*/*/*.*:@d@${exists($d):?:${"${d:E:M*/*}":?:${exists(${d:R}):?$d:}}}@} -__unqual_depdirs := ${__depdirs:${__qual_depdirs:Uno:${M_ListToSkip}}} +# some entries may be qualified with . or . +# we can tell the unqualified ones easily - because they exist +__unqual_depdirs := ${__depdirs:@d@${exists($d):?$d:}@} +__qual_depdirs := ${__depdirs:${__unqual_depdirs:Uno:${M_ListToSkip}}} .if ${DEP_RELDIR} == ${_DEP_RELDIR} # if it was called out - we likely need it. @@ -632,9 +676,9 @@ __qual_depdirs += ${__hostdpadd} .if ${_debug_reldir} .info DEP_DIRDEPS_FILTER=${DEP_DIRDEPS_FILTER:ts:} -.info depdirs=${__depdirs} -.info qualified=${__qual_depdirs} -.info unqualified=${__unqual_depdirs} +.info depdirs=${__depdirs:S,^${SRCTOP}/,,} +.info qualified=${__qual_depdirs:S,^${SRCTOP}/,,} +.info unqualified=${__unqual_depdirs:S,^${SRCTOP}/,,} .endif # _build_dirs is what we will feed to _DIRDEP_USE @@ -655,13 +699,14 @@ _build_all_dirs := ${_build_all_dirs:O:u} # Normally if doing make -V something, # we do not want to waste time chasing DIRDEPS # but if we want to count the number of Makefile.depend* read, we do. -.if ${.MAKEFLAGS:M-V${_V_READ_DIRDEPS}} == "" +.if ${.MAKEFLAGS:M-V${_V_READ_DIRDEPS:U}} == "" .if !empty(_build_all_dirs) .if ${BUILD_DIRDEPS_CACHE} == "yes" x!= echo; { echo; echo '\# ${DEP_RELDIR}.${DEP_TARGET_SPEC}'; } >&3 # guard against _new_dirdeps being too big for a single command line -_new_dirdeps := ${_build_all_dirs:@x@${target($x):?:$x}@} -.export _build_xtra_dirs _new_dirdeps +_new_dirdeps := ${_build_all_dirs:@x@${target($x):?:$x}@:S,^${SRCTOP}/,,} +_cache_xtra_deps := ${_build_xtra_dirs:S,^${SRCTOP}/,,} +.export _cache_xtra_deps _new_dirdeps .if !empty(DEP_EXPORT_VARS) # Discouraged, but there are always exceptions. # Handle it here rather than explain how. @@ -674,7 +719,7 @@ dirdeps: ${_build_all_dirs} ${_build_all_dirs}: _DIRDEP_USE .if ${_debug_reldir} -.info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: needs: ${_build_dirs} +.info ${DEP_RELDIR}.${DEP_TARGET_SPEC}: needs: ${_build_dirs:S,^${SRCTOP}/,,} .endif .if !empty(DEP_EXPORT_VARS) @@ -685,36 +730,37 @@ DEP_EXPORT_VARS= # this builds the dependency graph .for m in ${_machines} .if ${BUILD_DIRDEPS_CACHE} == "yes" && !empty(_build_dirs) -x!= echo; { echo; echo 'DIRDEPS.${_this_dir}.$m = \'; } >&3 _cache_deps = +x!= echo; { echo; echo 'DIRDEPS.${_this_dir}.$m = \'; } >&3 .endif # it would be nice to do :N${.TARGET} .if !empty(__qual_depdirs) .for q in ${__qual_depdirs:${M_dep_qual_fixes:ts:}:E:O:u:N$m} .if ${_debug_reldir} || ${DEBUG_DIRDEPS:@x@${${DEP_RELDIR}.$m:L:M$x}${${DEP_RELDIR}.$q:L:M$x}@} != "" -.info ${DEP_RELDIR}.$m: graph: ${_build_dirs:M*.$q} +.info ${DEP_RELDIR}.$m: graph: ${_build_dirs:M*.$q:S,^${SRCTOP}/,,} .endif .if ${BUILD_DIRDEPS_CACHE} == "yes" -_cache_deps += ${_build_dirs:M*.$q} +_cache_deps += ${_build_dirs:M*.$q:S,^${SRCTOP}/,,} .else ${_this_dir}.$m: ${_build_dirs:M*.$q} .endif .endfor .endif .if ${_debug_reldir} -.info ${DEP_RELDIR}.$m: graph: ${_build_dirs:M*.$m:N${_this_dir}.$m} +.info ${DEP_RELDIR}.$m: graph: ${_build_dirs:M*.$m:N${_this_dir}.$m:S,^${SRCTOP}/,,} .endif .if ${BUILD_DIRDEPS_CACHE} == "yes" .if !empty(_build_dirs) -_cache_deps += ${_build_dirs:M*.$m:N${_this_dir}.$m} +_cache_deps += ${_build_dirs:M*.$m:N${_this_dir}.$m:S,^${SRCTOP}/,,} .if !empty(_cache_deps) .export _cache_deps -x!= echo; for x in $$_cache_deps; do echo " $$x \\"; done >&3 +x!= echo; for x in $$_cache_deps; do echo " _{SRCTOP}/$$x \\"; done >&3 .endif +# anything in _{build,env}_xtra_dirs is hooked to dirdeps: only x!= echo; { echo; echo '${_this_dir}.$m: $${DIRDEPS.${_this_dir}.$m}'; \ echo; echo 'dirdeps: ${_this_dir}.$m \'; \ - for x in $$_build_xtra_dirs; do echo " $$x \\"; done; \ - echo; for x in $$_new_dirdeps; do echo "$$x: _DIRDEP_USE"; done; } >&3 + for x in $$_cache_xtra_deps; do echo " _{SRCTOP}/$$x \\"; done; \ + echo; for x in $$_new_dirdeps; do echo "_{SRCTOP}/$$x: _DIRDEP_USE"; done; } >&3 .endif .else ${_this_dir}.$m: ${_build_dirs:M*.$m:N${_this_dir}.$m} @@ -729,7 +775,7 @@ ${_this_dir}.$m: ${_build_dirs:M*.$m:N${_this_dir}.$m} # once only _dirdeps_checked.$d: .if ${_debug_search} -.info checking $d +.info checking ${d:S,^${SRCTOP}/,,} .endif # Note: _build_all_dirs is fully qualifed so d:R is always the directory .if exists(${d:R}) @@ -749,7 +795,7 @@ DEP_MACHINE := ${_DEP_MACHINE} # Warning: there is an assumption here that MACHINE is always # the first entry in TARGET_SPEC_VARS. # If TARGET_SPEC and MACHINE are insufficient, you have a problem. -_m := ${.MAKE.DEPENDFILE_PREFERENCE:T:S;${TARGET_SPEC}$;${d:E};:S;${MACHINE};${d:E:C/,.*//};:@m@${exists(${d:R}/$m):?${d:R}/$m:}@:[1]} +_m := ${.MAKE.DEPENDFILE_PREFERENCE:T:S;${TARGET_SPEC}$;${d:E};:C;${MACHINE}((,.+)?)$;${d:E:C/,.*//}\1;:@m@${exists(${d:R}/$m):?${d:R}/$m:}@:[1]} .if !empty(_m) # M_dep_qual_fixes isn't geared to Makefile.depend _qm := ${_m:C;(\.depend)$;\1.${d:E};:${M_dep_qual_fixes:ts:}} @@ -758,7 +804,7 @@ _qm := ${_m:C;(\.depend)$;\1.${d:E};:${M_dep_qual_fixes:ts:}} .endif # set this "just in case" # we can skip :tA since we computed the path above -DEP_RELDIR := ${_m:H:S,${SRCTOP}/,,} +DEP_RELDIR := ${_m:H:S,^${SRCTOP}/,,} # and reset this DIRDEPS = .if ${_debug_reldir} && ${_qm} != ${_m} @@ -815,6 +861,7 @@ _reldir_failed: .NOMETA .endif # bootstrapping new dependencies made easy? +.if !target(bootstrap-empty) .if !target(bootstrap) && (make(bootstrap) || \ make(bootstrap-this) || \ make(bootstrap-recurse) || \ @@ -870,3 +917,4 @@ bootstrap-empty: .NOTMAIN .NOMETA @{ echo DIRDEPS=; echo ".include "; } > ${_want} .endif +.endif diff --git a/share/mk/meta.autodep.mk b/share/mk/meta.autodep.mk index 4dda9970774e..842554beea45 100644 --- a/share/mk/meta.autodep.mk +++ b/share/mk/meta.autodep.mk @@ -1,5 +1,5 @@ # $FreeBSD$ -# $Id: meta.autodep.mk,v 1.53 2020/11/08 05:47:56 sjg Exp $ +# $Id: meta.autodep.mk,v 1.55 2021/12/13 08:12:01 sjg Exp $ # # @(#) Copyright (c) 2010, Simon J. Gerraty @@ -24,8 +24,12 @@ __${_this}__: .NOTMAIN PICO?= .pico .if defined(SRCS) +.if ${MAKE_VERSION:U0} >= 20211212 +OBJ_EXTENSIONS += ${.SUFFIXES:M*o} +.else # it would be nice to be able to query .SUFFIXES -OBJ_EXTENSIONS+= .o .po .lo ${PICO} +OBJ_EXTENSIONS += .o .po .lo ${PICO} +.endif # explicit dependencies help short-circuit .SUFFIX searches SRCS_DEP_FILTER+= N*.[hly] @@ -164,7 +168,11 @@ _depend = .depend # it would be nice to be able to get .SUFFIXES as ${.SUFFIXES} # we actually only care about the .SUFFIXES of files that might be # generated by tools like yacc. +.if ${MAKE_VERSION:U0} >= 20211212 +DEPEND_SUFFIXES += ${.SUFFIXES:N.sh:N*[0-9aFfglopmnrSsty]} +.else DEPEND_SUFFIXES += .c .h .cpp .hpp .cxx .hxx .cc .hh +.endif .depend: .NOMETA $${.MAKE.META.CREATED} ${_this} @echo "Updating $@: ${.OODATE:T:[1..8]}" @egrep -i '^R .*\.(${DEPEND_SUFFIXES:tl:O:u:S,^.,,:ts|})$$' /dev/null ${.MAKE.META.FILES:T:O:u:${META_FILE_FILTER:ts:}:M*o.meta} | \ @@ -299,16 +307,20 @@ start_utc := ${now_utc} meta_stats= meta=${empty(.MAKE.META.FILES):?0:${.MAKE.META.FILES:[#]}} \ created=${empty(.MAKE.META.CREATED):?0:${.MAKE.META.CREATED:[#]}} +.if !target(_reldir_finish) #.END: _reldir_finish .if target(gendirdeps) _reldir_finish: gendirdeps .endif _reldir_finish: .NOMETA @echo "${TIME_STAMP} Finished ${RELDIR}.${TARGET_SPEC} seconds=$$(( ${now_utc} - ${start_utc} )) ${meta_stats}" +.endif +.if !target(_reldir_failed) #.ERROR: _reldir_failed _reldir_failed: .NOMETA @echo "${TIME_STAMP} Failed ${RELDIR}.${TARGET_SPEC} seconds=$$(( ${now_utc} - ${start_utc} )) ${meta_stats}" +.endif .if !defined(WITHOUT_META_STATS) && ${.MAKE.LEVEL} > 0 .END: _reldir_finish diff --git a/share/mk/meta.sys.mk b/share/mk/meta.sys.mk index 94e6ef862fdb..cccab7ca952d 100644 --- a/share/mk/meta.sys.mk +++ b/share/mk/meta.sys.mk @@ -1,8 +1,8 @@ # $FreeBSD$ -# $Id: meta.sys.mk,v 1.38 2020/08/19 17:51:53 sjg Exp $ +# $Id: meta.sys.mk,v 1.42 2021/12/13 05:50:55 sjg Exp $ # -# @(#) Copyright (c) 2010-2020, Simon J. Gerraty +# @(#) Copyright (c) 2010-2021, Simon J. Gerraty # # This file is provided in the hope that it will # be of use. There is absolutely NO WARRANTY. @@ -31,21 +31,31 @@ SYS_MK_DIR := ${_PARSEDIR} .endif META_MODE += meta verbose +.if ${MAKE_VERSION:U0} > 20130323 && empty(.MAKE.PATH_FILEMON) +# we do not support filemon +META_MODE += nofilemon +MKDEP_MK ?= auto.dep.mk +.endif + .MAKE.MODE ?= ${META_MODE} -.if ${.MAKE.LEVEL} == 0 +_filemon := ${.MAKE.PATH_FILEMON:U/dev/filemon} + +.if empty(UPDATE_DEPENDFILE) _make_mode := ${.MAKE.MODE} ${META_MODE} .if ${_make_mode:M*read*} != "" || ${_make_mode:M*nofilemon*} != "" # tell everyone we are not updating Makefile.depend* UPDATE_DEPENDFILE = NO .export UPDATE_DEPENDFILE .endif -.if ${UPDATE_DEPENDFILE:Uyes:tl} == "no" && !exists(/dev/filemon) +.if ${_filemon:T:Mfilemon} == "filemon" +.if ${UPDATE_DEPENDFILE:Uyes:tl} == "no" && !exists(${_filemon}) # we should not get upset META_MODE += nofilemon .export META_MODE .endif .endif +.endif .if !defined(NO_SILENT) .if ${MAKE_VERSION} > 20110818 @@ -107,7 +117,7 @@ _metaError: .NOMETA .NOTMAIN # Are we, after all, in meta mode? .if ${.MAKE.MODE:Uno:Mmeta*} != "" -MKDEP_MK = meta.autodep.mk +MKDEP_MK ?= meta.autodep.mk .if ${.MAKE.MAKEFILES:M*sys.dependfile.mk} == "" # this does all the smarts of setting .MAKE.DEPENDFILE @@ -141,12 +151,13 @@ META_NOECHO= : .warning Setting UPDATE_DEPENDFILE=NO due to -k UPDATE_DEPENDFILE= NO .export UPDATE_DEPENDFILE -.elif !exists(/dev/filemon) -.error ${.newline}ERROR: The filemon module (/dev/filemon) is not loaded. +.elif ${_filemon:T} == "filemon" && !exists(${_filemon}) +.error ${.newline}ERROR: The filemon module (${_filemon}) is not loaded. .endif .endif .if ${.MAKE.LEVEL} == 0 +.if ${MK_DIRDEPS_BUILD:Uyes} == "yes" # make sure dirdeps target exists and do it first all: dirdeps .WAIT dirdeps: @@ -157,6 +168,7 @@ dirdeps: # by default dirdeps is all we want at level0 .MAIN: dirdeps .endif +.endif .endif .else diff --git a/share/mk/meta2deps.py b/share/mk/meta2deps.py index 42e4bb353446..1bcf1baedaaa 100755 --- a/share/mk/meta2deps.py +++ b/share/mk/meta2deps.py @@ -38,7 +38,7 @@ We only pay attention to a subset of the information in the """ RCSid: $FreeBSD$ - $Id: meta2deps.py,v 1.34 2020/10/02 03:11:17 sjg Exp $ + $Id: meta2deps.py,v 1.40 2021/12/13 19:32:46 sjg Exp $ Copyright (c) 2011-2020, Simon J. Gerraty Copyright (c) 2011-2017, Juniper Networks, Inc. @@ -69,12 +69,6 @@ RCSid: import os, re, sys -def getv(dict, key, d=None): - """Lookup key in dict and return value or the supplied default.""" - if key in dict: - return dict[key] - return d - def resolve(path, cwd, last_dir=None, debug=0, debug_out=sys.stderr): """ Return an absolute path, resolving via cwd or last_dir if needed. @@ -153,7 +147,10 @@ def abspath(path, cwd, last_dir=None, debug=0, debug_out=sys.stderr): return path def sort_unique(list, cmp=None, key=None, reverse=False): - list.sort(cmp, key, reverse) + if sys.version_info[0] == 2: + list.sort(cmp, key, reverse) + else: + list.sort(reverse=reverse) nl = [] le = None for e in list: @@ -169,6 +166,19 @@ def add_trims(x): x + '/', x] +def target_spec_exts(target_spec): + """return a list of dirdep extensions that could match target_spec""" + + if target_spec.find(',') < 0: + return ['.'+target_spec] + w = target_spec.split(',') + n = len(w) + e = [] + while n > 0: + e.append('.'+','.join(w[0:n])) + n -= 1 + return e + class MetaFile: """class to parse meta files generated by bmake.""" @@ -225,22 +235,23 @@ class MetaFile: """ self.name = name - self.debug = getv(conf, 'debug', 0) - self.debug_out = getv(conf, 'debug_out', sys.stderr) - - self.machine = getv(conf, 'MACHINE', '') - self.machine_arch = getv(conf, 'MACHINE_ARCH', '') - self.target_spec = getv(conf, 'TARGET_SPEC', '') - self.curdir = getv(conf, 'CURDIR') - self.reldir = getv(conf, 'RELDIR') - self.dpdeps = getv(conf, 'DPDEPS') + self.debug = conf.get('debug', 0) + self.debug_out = conf.get('debug_out', sys.stderr) + + self.machine = conf.get('MACHINE', '') + self.machine_arch = conf.get('MACHINE_ARCH', '') + self.target_spec = conf.get('TARGET_SPEC', self.machine) + self.exts = target_spec_exts(self.target_spec) + self.curdir = conf.get('CURDIR') + self.reldir = conf.get('RELDIR') + self.dpdeps = conf.get('DPDEPS') self.line = 0 if not self.conf: # some of the steps below we want to do only once self.conf = conf - self.host_target = getv(conf, 'HOST_TARGET') - for srctop in getv(conf, 'SRCTOPS', []): + self.host_target = conf.get('HOST_TARGET') + for srctop in conf.get('SRCTOPS', []): if srctop[-1] != '/': srctop += '/' if not srctop in self.srctops: @@ -254,10 +265,10 @@ class MetaFile: trim_list = add_trims(self.machine) if self.machine == 'host': trim_list += add_trims(self.host_target) - if self.target_spec: + if self.target_spec != self.machine: trim_list += add_trims(self.target_spec) - for objroot in getv(conf, 'OBJROOTS', []): + for objroot in conf.get('OBJROOTS', []): for e in trim_list: if objroot.endswith(e): # this is not what we want - fix it @@ -277,13 +288,14 @@ class MetaFile: self.srctops.sort(reverse=True) self.objroots.sort(reverse=True) - self.excludes = getv(conf, 'EXCLUDES', []) + self.excludes = conf.get('EXCLUDES', []) if self.debug: print("host_target=", self.host_target, file=self.debug_out) print("srctops=", self.srctops, file=self.debug_out) print("objroots=", self.objroots, file=self.debug_out) print("excludes=", self.excludes, file=self.debug_out) + print("ext_list=", self.exts, file=self.debug_out) self.dirdep_re = re.compile(r'([^/]+)/(.+)') @@ -359,10 +371,10 @@ class MetaFile: ddep = open(ddepf, 'r').readline().strip('# \n') if self.debug > 1: print("found %s: %s\n" % (ddepf, ddep), file=self.debug_out) - if ddep.endswith(self.machine): - ddep = ddep[0:-(1+len(self.machine))] - elif self.target_spec and ddep.endswith(self.target_spec): - ddep = ddep[0:-(1+len(self.target_spec))] + for e in self.exts: + if ddep.endswith(e): + ddep = ddep[0:-len(e)] + break if not ddep: # no .dirdeps, so remember that we've seen the raw input @@ -469,8 +481,8 @@ class MetaFile: if pid != last_pid: if last_pid: pid_last_dir[last_pid] = self.last_dir - cwd = getv(pid_cwd, pid, self.cwd) - self.last_dir = getv(pid_last_dir, pid, self.cwd) + cwd = pid_cwd.get(pid, self.cwd) + self.last_dir = pid_last_dir.get(pid, self.cwd) last_pid = pid # process operations @@ -558,7 +570,10 @@ class MetaFile: # to the src dir, we may need to add dependencies for each rdir = dir dir = abspath(dir, cwd, self.last_dir, self.debug, self.debug_out) - rdir = os.path.realpath(dir) + if dir: + rdir = os.path.realpath(dir) + else: + dir = rdir if rdir == dir: rdir = None # now put path back together @@ -726,7 +741,7 @@ def main(argv, klass=MetaFile, xopts='', xoptf=None): for a in eaten: args.remove(a) - debug_out = getv(conf, 'debug_out', sys.stderr) + debug_out = conf.get('debug_out', sys.stderr) if debug: print("config:", file=debug_out) @@ -753,9 +768,9 @@ def main(argv, klass=MetaFile, xopts='', xoptf=None): print(m.src_dirdeps('\nsrc:')) - dpdeps = getv(conf, 'DPDEPS') + dpdeps = conf.get('DPDEPS') if dpdeps: - m.file_depends(open(dpdeps, 'wb')) + m.file_depends(open(dpdeps, 'w')) return m