From nobody Sat Jan 22 11:48:21 2022 X-Original-To: dev-commits-ports-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7C9491958894; Sat, 22 Jan 2022 11:48:21 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jgvdd31fVz3NPh; Sat, 22 Jan 2022 11:48:21 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642852101; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XnwZOTse5AyAF4vTUxGdZxpfFXfRIO+qZxUu/frKxRs=; b=Oct7r6fFUueihjLkNo7eht627R4eDjbueVFcJ4nS7FsOplnl9N4NA0EMXIFmOFglWbpQrE dUpOdiBG7s4L9ygz0nGKhbBpXidjQCiDXAtYFq1mAYBY9gzHTGrwaTgDM4r6IY96X/qqs7 wt+Zyr0EViU/GNEqubFn0Mha9NkbX4U4rqT8RxlF3kTRwiiOKk/9CpjpbilX7UHwXBZ9sA cHveqgZsViy0qg+MhUl67uf8e5Jc29bk8WtaRwaxPWW5CIsDeJQ+bKQ/UO4idkZe5rqCcR XJVBm33zwq0nirU+FAyEEh/JlLJ/zKXRYbjFXiUnB/a6B1UpgsDDYk1NWweA5A== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 43D8E6B69; Sat, 22 Jan 2022 11:48:21 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20MBmLsp073503; Sat, 22 Jan 2022 11:48:21 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20MBmLhs073501; Sat, 22 Jan 2022 11:48:21 GMT (envelope-from git) Date: Sat, 22 Jan 2022 11:48:21 GMT Message-Id: <202201221148.20MBmLhs073501@gitrepo.freebsd.org> To: ports-committers@FreeBSD.org, dev-commits-ports-all@FreeBSD.org, dev-commits-ports-main@FreeBSD.org From: =?utf-8?Q?Stefan E=C3=9Fer?= Subject: git: 59c30220dc28 - main - Tools/scripts: Add port_conflicts_check.lua List-Id: Commits to the main branch of the FreeBSD ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-ports-main@freebsd.org X-BeenThere: dev-commits-ports-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: se X-Git-Repository: ports X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 59c30220dc28b4bbc419e8a7624d6572cd0b13be Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642852101; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=XnwZOTse5AyAF4vTUxGdZxpfFXfRIO+qZxUu/frKxRs=; b=FA6Ge1XnL66mqNrOPDSDBZ4Bd0tzXoenpBDEpvOeC+DZgJZXSERli/AYniXk3SdF72Kgdi MoRy5D2NkGmVCkbizAmy8QW7v5fvftEU6qRtGi2TxxfNZydPdsZvJFOi4mF/pP4D8tcMHg 9up3CEnhPga0ZdMrYg/NxjbTPuqsOHGaATeAm7AFLqgT+iFs0Qboyg2VNMAHx9fS07Wvau 1h+RDHRRNf92OIcdpF4gkQ9KCHT2iJrlNfl91CygTu5EZ21ZH7uIUKWqkQReSeHYe+rf1v KcFXz9VNFkCTXHtYvVazkZeAa8LT6FQXE7vntdNVUI1D5Bt44OW9QEJ9Da3E4A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642852101; a=rsa-sha256; cv=none; b=e5HQwNUcskO3pk+XaOQic6jOmoK7WNtyOi+D5F91PggKz+vS4x43wJmjpIPUFXHH2x+qxP ZazzwnK1nLaoxEk5hebT4AV+yvJ5OLgDlbizxVAycgeXI9dH2OyMMNYvVdut1B4XzGeoqo dMSdGfsTLm9K3+VXzuwWssst/ibMkIzURoLZn+xKTjJFplixgy6iG4buWV3z6UCzrznkCp cecXGYyVbyq/F+apQT4mXkUKwwqv4A63T1kb+986I7ijteNJFGREejFgF7qM3bwdLAAODy AABV7DhVn+iSCb2zRMyira3odKzP9M4Ccs8jR3U4MxQqXIaX/wR2VFdxasNkjg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by se: URL: https://cgit.FreeBSD.org/ports/commit/?id=59c30220dc28b4bbc419e8a7624d6572cd0b13be commit 59c30220dc28b4bbc419e8a7624d6572cd0b13be Author: Stefan Eßer AuthorDate: 2022-01-22 11:08:07 +0000 Commit: Stefan Eßer CommitDate: 2022-01-22 11:48:03 +0000 Tools/scripts: Add port_conflicts_check.lua Add a script to check the CONFLICTS and CONFLICTS_INSTALL parameters of ports for completeness and correctness. This script uses the "hidden" LUA interpreter in the FreeBSD base system and the pkg-provides extension of the pkg command to check for conflicting files in all packages available for the architecture and version of the base system this command is run on. It generates output in the following format: portedit merge -ie 'CONFLICTS_INSTALL=kicad-library-footprints-devel \ # share/kicad/template/fp-lib-table' \ /usr/ports/cad/kicad-library-footprints (The last line is shown wrapped for the text of this commit message.) The portedit command is provided by the port-fmt package. It takes care of placing the CONFLICTS_ENTRY into the correct position of the port's Makefile (and removes prior definitions). The files listed with each result are examples of files that are in conflict between the port and the packages in the list after ">". The main purpose of the files list is to help distinguish between conflicts that affect all flavors or versions of a port, or whether the files are placed in version specific sub-directories or use other mechanisms to allow e.g. multiple Python versions to co-exist. (In the latter case ${PYTHON_PKGNAMEPREFIX} can be used to limit the CONFLICTS_INSTALL entry to conflicting packages using the same Python interpreter version, for example, else a prefix like py*- might be required for a version independent pattern). Users of this feature are highly advised to check each Makefile by comparing it with pre-edit version before the changes are committed! There are several limitations that can cause incorrect or undesirable changes: - The list of files installed by each port is only available for the officially built packages (and the flavors selected from the set of available flavors). It does not include ports that may not be packaged or that are broken or ignored due to a dependency on a broken port (or for other reasons). As a result, there may be undected conflicts with ports for which no official package is available. - The CONFLICTS_INSTALL line is not always inserted into the correct position in the Makefile, typically due to out-of-order entries used by portedit to locate the desired position. - Complex ports may have conditional CONFLICTS_INSTALL entries, depending on port options or flavors that are in effect. It is not possible to deal with that kind of Makefiles in an automated way. - The union of all CONFLICTS and CONFLICTS_INSTALL entries is used as the list of install conflicts of a port. But only CONFLICTS_INSTALL entries are generated by this tool. Quite a lot of ports have CONFLICTS entries where CONFLICTS_INSTALL would suffice (i.e. there is no build conflict, actually), but there are ports that need to keep the conflicts listed as CONFLICTS. Such issues can be found by comparing the before and after versions of the edited Makefiles. - Conflicting ports that have been removed from the ports system will only be found as long as their official package files are still available. (There is a recommendation that conflicts with removed ports are kept for a few months.) - If all packages conflicting with a given port have been removed from the ports system and the official packages repository, the now superfluous CONFLICTS_INSTALL definition will not be detected. This is due to only Makefiles of ports being parsed that install files in the same place as some other port. Parsing all Makefiles instead would increase the run-time of this script by more than a factor of 10. --- Tools/scripts/README | 3 + Tools/scripts/port_conflicts_check.lua | 346 +++++++++++++++++++++++++++++++++ 2 files changed, 349 insertions(+) diff --git a/Tools/scripts/README b/Tools/scripts/README index aa9b32f612e4..e3712d07d345 100644 --- a/Tools/scripts/README +++ b/Tools/scripts/README @@ -30,6 +30,9 @@ gnomedepends - Analyse pkg/PLIST and give an advice as to which GNOME ports should be listes in {RUN,LIB}_DEPENDS for this port mark_safe.pl - utility to set subsets of ports to MAKE_JOBS_(UN)SAFE=yes neededlibs.sh - Extract direct library dependencies from binaries. +port_conflicts_check.lua - Verify that files installed by more than 1 port are covered + in CONFLICTS or CONFLICTS_INSTALL entries (and generate portedit commands + to fix those issues)x portsearch - A utility for searching the ports tree. It allows more detailed search criteria than ``make search key='' and accepts all perl(1) regular expressions. diff --git a/Tools/scripts/port_conflicts_check.lua b/Tools/scripts/port_conflicts_check.lua new file mode 100755 index 000000000000..49d8579e58be --- /dev/null +++ b/Tools/scripts/port_conflicts_check.lua @@ -0,0 +1,346 @@ +#!/usr/libexec/flua + +--[[ +SPDX-License-Identifier: BSD-2-Clause-FreeBSD + +Copyright (c) 2022 Stefan Esser + +Generate a list of existing and required CONFLICTS_INSTALL lines +for all ports (limited to ports for which official packages are +provided). + +This script depends on the ports-mgmt/pkg-provides port for the list +of files installed by all pre-built packages for the architecture +the script is run on. + +The script generates a list of ports by running "pkg provides ." and +a mapping from package base name to origin via "pkg rquery '%n %o'". + +The existing CONFLICTS and CONFLICTS_INSTALL definitions are fetched +by "make -C $origin -V CONFLICTS -V CONFLICTS_INSTALL". This list is +only representative for the options configured for each port (i.e. +if non-default options have been selected and registered, these may +lead to a non-default list of conflicts). + +The script detects files used by more than one port, than lists by +origin the existing definition and the list of package base names +that have been detected to cause install conflicts followed by the +list of duplicate files separated by a hash character "#". + +This script uses the "hidden" LUA interpreter in the FreeBSD base +systems and does not need any port except "pkg-provides" to be run. + +The run-time on my system checking the ~32000 packages available +for -CURRENT on amd64 is less than 250 seconds. + +Example output: + +# Port: games/sol +# Files: bin/sol +# < aisleriot gnome-games +# > aisleriot +portedit merge -ie 'CONFLICTS_INSTALL=aisleriot # bin/sol' /usr/ports/games/sol + +The output is per port (for all flavors of the port, if applicable), +gives examples of conflicting files (mostly to understand whether +different versions of a port could co-exist), the current CONFLICTS +and CONFLICTS_INSTALL entries merged, and a suggested new entry. +This information is followed by a portedit command line that should +do the right thing for simple cases, but the result should always +be checked before the resulting Makefile is committed. +--]] + +require "lfs" + +------------------------------------------------------------------- +local function table_sorted_keys(t) + local result = {} + for k, _ in pairs(t) do + result[#result + 1] = k + end + table.sort(result) + return result +end + +local function table_sort_uniq(t) + local result = {} + if t then + local last + table.sort(t) + for _, entry in ipairs(t) do + if entry ~= last then + last = entry + result[#result + 1] = entry + end + end + end + return result +end + +local function fnmatch(name, pattern) + local function fnsubst(s) + s = string.gsub(s, "%%", "%%%%") + s = string.gsub(s, "%+", "%%+") + s = string.gsub(s, "%-", "%%-") + s = string.gsub(s, "%.", "%%.") + s = string.gsub(s, "%?", ".") + s = string.gsub(s, "%*", ".*") + return s + end + local rexpr = "" + local left, middle, right + while true do + left, middle, right = string.match(pattern, "([^[]*)(%[[^]]+%])(.*)") + if not left then + break + end + rexpr = rexpr .. fnsubst(left) .. middle + pattern = right + end + rexpr = "^" .. rexpr .. fnsubst(pattern) .. "$" + return string.find(name, rexpr) +end + +------------------------------------------------------------------- +local function fetch_pkgs_origin() + local pkgs = {} + local pipe = io.popen("pkg rquery '%n %o'") + for line in pipe:lines() do + local pkgbase, origin = string.match(line, "(%S+) (%S+)") + pkgs[origin] = pkgbase + end + pipe:close() + pipe = io.popen("pkg rquery '%n %o %At %Av'") + for line in pipe:lines() do + local pkgbase, origin, tag, value = string.match(line, "(%S+) (%S+) (%S+) (%S+)") + if tag == "flavor" then + pkgs[origin] = nil + pkgs[origin .. "@" .. value] = pkgbase + end + end + pipe:close() + return pkgs +end + +------------------------------------------------------------------- +local function read_files(pattern) + local files_table = {} + local pkgbase + local pipe = io.popen("pkg provides " .. pattern) + for line in pipe:lines() do + local label = string.sub(line, 1, 10) + if label == "Name : " then + local name = string.sub(line, 11) + pkgbase = string.match(name, "(.*)-[^-]*") + elseif label == " " or label == "Filename: " then + local file = string.sub(line, 11) + if file:sub(1, 10) == "usr/local/" then + file = file:sub(11) + else + file = "/" .. file + end + local t = files_table[file] or {} + t[#t + 1] = pkgbase + files_table[file] = t + end + end + pipe:close() + return files_table +end + +------------------------------------------------------------------- + +local function fetch_pkg_pairs(pattern) + local pkg_pairs = {} + for file, pkgbases in pairs(read_files(pattern)) do + if #pkgbases >= 2 then + for i = 1, #pkgbases -1 do + local pkg_i = pkgbases[i] + for j = i + 1, #pkgbases do + local pkg_j = pkgbases[j] + if pkg_i ~= pkg_j then + local p1 = pkg_pairs[pkg_i] or {} + local p2 = p1[pkg_j] or {} + p2[#p2 + 1] = file + p1[pkg_j] = p2 + pkg_pairs[pkg_i] = p1 + end + end + end + end + end + return pkg_pairs +end + +------------------------------------------------------------------- +local function conflicts_delta(old, new) + local old_seen = {} + local changed + for i = 1, #new do + local matched + for j = 1, #old do + if fnmatch(new[i], old[j]) then + new[i] = old[j] + old_seen[j] = true + matched = true + break + end + end + changed = changed or not matched + end + if not changed then + for j = 1, #old do + if not old_seen[j] then + changed = true + break + end + end + end + if changed then + return table_sort_uniq(new) + end +end + +------------------------------------------------------------------- +local function fetch_port_conflicts(origin) + local dir, flavor = origin:match("([^@]+)@?(.*)") + if flavor ~= "" then + flavor = " FLAVOR=" .. flavor + end + local seen = {} + local pipe = io.popen("make -C /usr/ports/" .. dir .. flavor .. " -V CONFLICTS -V CONFLICTS_INSTALL 2>/dev/null") + for line in pipe:lines() do + for word in line:gmatch("(%S+)%s?") do + seen[word] = true + end + end + pipe:close() + return table_sorted_keys(seen) +end + +------------------------------------------------------------------- +local function conflicting_pkgs(conflicting) + local pkgs = {} + for origin, pkgbase in pairs(fetch_pkgs_origin()) do + if conflicting[pkgbase] then + pkgs[origin] = pkgbase + end + end + return pkgs +end + +------------------------------------------------------------------- +local function collect_conflicts(pkg_pairs) + local pkgs = {} + local files = {} + for pkg_i, p1 in pairs(pkg_pairs) do + for pkg_j, p2 in pairs(p1) do + pkgs[pkg_i] = pkgs[pkg_i] or {} + pkgs[pkg_j] = pkgs[pkg_j] or {} + table.insert(pkgs[pkg_i], pkg_j) + table.insert(pkgs[pkg_j], pkg_i) + files[pkg_i] = files[pkg_i] or {} + files[pkg_j] = files[pkg_j] or {} + for _, file in ipairs(p2) do + table.insert(files[pkg_i], file) + table.insert(files[pkg_j], file) + end + end + end + return pkgs, files +end + +------------------------------------------------------------------- +local function split_origins(origin_list) + local port_list = {} + local flavors = {} + local last_port + for _, origin in ipairs(origin_list) do + local port, flavor = string.match(origin, "([^@]+)@?(.*)") + if port ~= last_port then + port_list[#port_list + 1] = port + if flavor ~= "" then + flavors[port] = {flavor} + end + else + table.insert(flavors[port], flavor) + end + last_port = port + end + return port_list, flavors +end + +------------------------------------------------------------------- +-- TODO: Collect FLAVORs and report for port directory + +local function merge_table(t1, t2) + table.move(t2, 1, #t2, #t1 + 1, t1) +end + +local PKG_PAIR_FILES = fetch_pkg_pairs(".") +local CONFLICT_PKGS = collect_conflicts(PKG_PAIR_FILES) +local PKGBASE = conflicting_pkgs(CONFLICT_PKGS) +local ORIGIN_LIST = table_sorted_keys(PKGBASE) +local PORT_LIST, FLAVORS = split_origins(ORIGIN_LIST) + +local function conflicting_files(pkg_i, pkgs) + local files = {} + local f + local p1 = PKG_PAIR_FILES[pkg_i] + if p1 then + for _, pkg_j in ipairs(pkgs) do + f = p1[pkg_j] + if f then + table.sort(f) + files[#files + 1] = f[1] + end + end + end + for _, pkg_j in ipairs(pkgs) do + p1 = PKG_PAIR_FILES[pkg_j] + f = p1 and p1[pkg_i] + if f then + table.sort(f) + files[#files + 1] = f[1] + end + end + return files +end + +for _, port in ipairs(PORT_LIST) do + local port_conflicts = {} + local files = {} + local conflict_pkgs = {} + local function merge_data(origin) + local pkgbase = PKGBASE[origin] + merge_table(files, conflicting_files(pkgbase, CONFLICT_PKGS[pkgbase])) + merge_table(conflict_pkgs, CONFLICT_PKGS[pkgbase]) + merge_table(port_conflicts, fetch_port_conflicts(origin)) + end + local flavors = FLAVORS[port] + if flavors then + for _, flavor in ipairs(flavors) do + merge_data(port .. "@" .. flavor) + end + else + merge_data(port) + end + local conflicts_new = table_sort_uniq(conflict_pkgs) + if #port_conflicts then + port_conflicts = table_sort_uniq(port_conflicts) + conflicts_new = conflicts_delta(port_conflicts, conflicts_new) + end + if conflicts_new then + local conflicts_string = table.concat(port_conflicts, " ") + local conflicts_string_new = table.concat(conflicts_new, " ") + local file_list = table.concat(table_sort_uniq(files), " ") + print("# Port: " .. port) + print("# Files: " .. file_list) + if conflicts_string ~= "" then + print("# < " .. conflicts_string) + end + print("# > " .. conflicts_string_new) + print("portedit merge -ie 'CONFLICTS_INSTALL=" .. conflicts_string_new .. " # " .. file_list .. "' /usr/ports/" .. port) + print() + end +end