From nobody Thu Jan 13 23:13:04 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A61981959AC8; Thu, 13 Jan 2022 23:13:16 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JZgG44GHKz3JLt; Thu, 13 Jan 2022 23:13:16 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642115596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jsE86FyAoG6v4urrPu0WtWgnfS+YjPqx+xogIBYQKwk=; b=Dh+2FVojoEIoAuZNzrEZaNd2+KJTjVqLkVvCxInwBNl1Al1TmRbIqCaIGsU2/x2M4JDNfd gDzpGK9cc9+svPImKmWRaQ1l76go4CPs5Sgyd6mpcO9gm1s7fIHAsrMx915Swn157baUHs jiOsZCycJSNitMyXVbp4C0LJjjO+vNhs9ExTV4EQiu0A4v8RlrIkkDSqI+JkYeBisRd26y G38rj2rQ28PqeOfsr/5FwfsBUmTiuTuHaR1JNilvkc1cNaLBpyQUid7RRd3c0a/bBHm74m jH9QYHqLZPHHh54tPQ91EnQ6jGLIZpdhB9ZqUAw0H6LS1EpRMByh9kM1vudTbg== Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 6E4A3AA7; Thu, 13 Jan 2022 23:13:16 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qk1-f171.google.com with SMTP id c190so7571124qkg.9; Thu, 13 Jan 2022 15:13:16 -0800 (PST) X-Gm-Message-State: AOAM5325E8jqnKzb4lrmAoUvo1mou96w5MQIEFMO9p95z8wKgKRT3X6F K89+jmzYwt7SaOCsS++4lKZoiM8+TJhYQbfTc6A= X-Google-Smtp-Source: ABdhPJz9aJHtJYKRLpgXciPDy9gEttxEJqjhjsGgLil2Ustj0qvauxatmC+cxiyW2saZUBPwQYU7tRBPF9FnMay9x10= X-Received: by 2002:a05:620a:f0e:: with SMTP id v14mr4832992qkl.365.1642115595962; Thu, 13 Jan 2022 15:13:15 -0800 (PST) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 References: <202201132138.20DLccSU044307@gitrepo.freebsd.org> In-Reply-To: <202201132138.20DLccSU044307@gitrepo.freebsd.org> From: Kyle Evans Date: Thu, 13 Jan 2022 17:13:04 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: git: 82bfeeff10da - main - tools/portconflicts/port_conflicts_check.lua: detect port conflicts To: =?UTF-8?B?U3RlZmFuIEXDn2Vy?= Cc: src-committers , "" , dev-commits-src-main@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642115596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jsE86FyAoG6v4urrPu0WtWgnfS+YjPqx+xogIBYQKwk=; b=UqmEVswAIF/JPlqhER6V9y114/g3OwlVJvaMACXwi/gTGvmqRNGSXfUFdUBEnth1zgff5b zgl4lc9cx1MRHNMF36KH+XyTi8zskcMrrQsNgtAw1ycJiICh/frrQn1iSFHMLDYvg0vVmL +TEu/z26ftbtFtD3EF8X/4OApvzdDyjwxnjV45cIPGc1S+1YslRSKw+9yMuN5SOYQu6ChP 0lc8pgXkrtCyf+oG4coVb1dRDPKbOC0R0Myqg9wqcMkN0YYK5p5SzkVTVnCX2+5pF8ujIa VdtIB0K/3Y6Bo3GAvqXbCesLy1aDZ9tqfsLKbMgbqsJLEZ65bOzGVnkuNNjcQw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642115596; a=rsa-sha256; cv=none; b=E+QDIGvpvY8QJvR1FHrzD/24zxh0r1D08unWwvyJXEdorgRYz6gbnqshDWa++LlgoKwiPy Xs9uZ/fAYmt4DJdTF2pBCKOdP0OPgN52m8o5exyby+V/n6a3WaeH9yNXirEHtDHEG5Ells nM9gNRlevzwu2ubaODJDvZDxw8lor/xMghv3Mtb9yD7nZoo9Zg9lx+ilYBkYwsdWbKbrlU FY9nU0a52s/qsZ8m5p6c6hN7vLYlwlWya1i1IbeT/1Z7kzLTp4LYNx93INv9R97vCQIHVV AJWfW5GRlzUlQMOxFdJ1WG4sMZ4twS/MBwMZUSZ3bVfRJqLQq+lSnq3PLe+tmg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N and runOn Thu, Jan 13, 2022 at 3:38 PM Stefan E=C3=9Fer w= rote: > > The branch main has been updated by se: > > URL: https://cgit.FreeBSD.org/src/commit/?id=3D82bfeeff10da860b1ed9f03a01= f3282b42b991be > > commit 82bfeeff10da860b1ed9f03a01f3282b42b991be > Author: Stefan E=C3=9Fer > AuthorDate: 2022-01-13 21:36:09 +0000 > Commit: Stefan E=C3=9Fer > CommitDate: 2022-01-13 21:36:09 +0000 > > tools/portconflicts/port_conflicts_check.lua: detect port conflicts > > This tool can be used to detect install conflicts (ports/packages tha= t > conflict with each other because of identically named files). > > MFC after: 3 days > Hi, Good to see more use of flua coming into the tree. :-) I recommend grabbing devel/lua-luacheck and running that over the file -- it's mostly pointing out complaints about things that should be declared `local`, but also notes some style-type issues and a write-only variable. > --- > tools/tools/portconflicts/port_conflicts_check.lua | 184 +++++++++++++++= ++++++ > 1 file changed, 184 insertions(+) > > diff --git a/tools/tools/portconflicts/port_conflicts_check.lua b/tools/t= ools/portconflicts/port_conflicts_check.lua > new file mode 100755 > index 000000000000..ced7cc619f4d > --- /dev/null > +++ b/tools/tools/portconflicts/port_conflicts_check.lua > @@ -0,0 +1,184 @@ > +#!/usr/libexec/flua > + > +--[[ > +SPDX-License-Identifier: BSD-2-Clause-FreeBSD > + > +Copyright (c) 2022 Stefan Esser > + > +Generate a list of existing and required CONFLICTS_INSTALL lines > +for all ports (limited to ports for which official packages are > +provided). > + > +This script depends on the ports-mgmt/pkg-provides port for the list > +of files installed by all pre-built packages for the architecture > +the script is run on. > + > +The script generates a list of ports by running "pkg provides ." and > +a mapping from package base name to origin via "pkg rquery '%n %o'". > + > +The existing CONFLICTS and CONFLICTS_INSTALL definitions are fetched > +by "make -C $origin -V CONFLICTS -V CONFLICTS_INSTALL". This list is > +only representative for the options configured for each port (i.e. > +if non-default options have been selected and registered, these may > +lead to a non-default list of conflicts). > + > +The script detects files used by more than one port, than lists by > +origin the existing definition and the list of package base names > +that have been detected to cause install conflicts followed by the > +list of duplicate files separated by a hash character "#". > + > +This script uses the "hidden" LUA interpreter in the FreeBSD base > +systems and does not need any port except "pkg-provides" to be run. > + > +The run-time on my system checking the ~32000 packages available > +for -CURRENT on amd64 is 150 seconds. > +--]] > + > +require "lfs" > + > +local index_file =3D "/usr/ports/INDEX-14" > + > +local function read_index () > + local ORIGIN =3D {} > + > + local pipe =3D io.popen("pkg rquery '%n %o'") > + for line in pipe:lines() do > + local pkgbase, origin =3D string.match(line, "(%S+) (%S+)") > + ORIGIN[pkgbase] =3D origin > + end > + pipe:close() > + return ORIGIN > +end > + > +local function read_files() > + local FILES_TABLE =3D {} > + > + local pkgbase, version > + local pipe =3D io.popen("pkg provides .") > + for line in pipe:lines() do > + local label =3D string.sub(line, 1, 10) > + if label =3D=3D "Name : " then > + name =3D string.sub(line, 11) > + pkgbase, version =3D string.match(name, "(.*)-([^-]*)") > + elseif label =3D=3D " " or label =3D=3D "Filename: " then > + local file =3D string.sub(line, 11) > + if file:sub(1, 10) =3D=3D "usr/local/" then > + file =3D file:sub(11) > + else > + file =3D "/" .. file > + end > + local t =3D FILES_TABLE[file] or {} > + t[#t + 1] =3D pkgbase > + FILES_TABLE[file] =3D t > + end > + end > + pipe:close() > + return FILES_TABLE > +end > + > +local PKG_PAIRS =3D {} > + > +for file, pkgbases in pairs(read_files()) do > + if #pkgbases > 1 then > + for i =3D 1, #pkgbases -1 do > + local pkg_i =3D pkgbases[i] > + for j =3D i + 1, #pkgbases do > + local pkg_j =3D pkgbases[j] > + if pkg_i ~=3D pkg_j then > + p1 =3D PKG_PAIRS[pkg_i] or {} > + p2 =3D p1[pkg_j] or {} > + p2[#p2 + 1] =3D file > + p1[pkg_j] =3D p2 > + PKG_PAIRS[pkg_i] =3D p1 > + end > + end > + end > + end > +end > + > +local CONFLICT_PKGS =3D {} > +local CONFLICT_FILES =3D {} > + > +for pkg_i, p1 in pairs(PKG_PAIRS) do > + for pkg_j, p2 in pairs(p1) do > + CONFLICT_PKGS[pkg_i] =3D CONFLICT_PKGS[pkg_i] or {} > + CONFLICT_PKGS[pkg_j] =3D CONFLICT_PKGS[pkg_j] or {} > + CONFLICT_FILES[pkg_i] =3D CONFLICT_FILES[pkg_i] or {} > + CONFLICT_FILES[pkg_j] =3D CONFLICT_FILES[pkg_j] or {} > + table.insert(CONFLICT_PKGS[pkg_i], pkg_j) > + table.insert(CONFLICT_PKGS[pkg_j], pkg_i) > + for _, file in ipairs(p2) do > + table.insert(CONFLICT_FILES[pkg_i], file) > + table.insert(CONFLICT_FILES[pkg_j], file) > + end > + end > +end > + > +local function table_sorted_keys(t) > + result =3D {} > + for k, _ in pairs(t) do > + result[#result + 1] =3D k > + end > + table.sort(result) > + return result > +end > + > +local function table_sort_uniq(t) > + local result =3D {} > + local last > + > + table.sort(t) > + for _, entry in ipairs(t) do > + if entry ~=3D last then > + last =3D entry > + result[#result + 1] =3D entry > + end > + end > + return result > +end > + > +local ORIGIN =3D read_index() > + > +local RESULT_PATTERN =3D {} > + > +for pkg, pkgs in pairs(CONFLICT_PKGS) do > + local origin =3D ORIGIN[pkg] > + > + if origin then > + table.sort(pkgs) > + RESULT_PATTERN[origin] =3D table.concat(pkgs, " ") > + end > +end > + > +local FILE_LIST =3D {} > + > +for pkg, files in pairs(CONFLICT_FILES) do > + local origin =3D ORIGIN[pkg] > + > + if origin then > + FILE_LIST[origin] =3D table.concat(table_sort_uniq(files), " ") > + end > +end > + > +for _, origin in ipairs(table_sorted_keys(RESULT_PATTERN)) do > + local pipe =3D io.popen("make -C /usr/ports/" .. origin .. " -V CONFL= ICTS -V CONFLICTS_INSTALL 2>/dev/null") > + local conflicts_table =3D {} > + local seen =3D {} > + for line in pipe:lines() do > + for word in line:gmatch("(%S*)%s?") do > + if word ~=3D "" and not seen[word] then > + table.insert(conflicts_table, word) > + seen[word] =3D true > + end > + end > + end > + pipe:close() > + table.sort(conflicts_table) > + conflicts_string =3D table.concat(conflicts_table, " ") > + local conflicts_new =3D RESULT_PATTERN[origin] > + if conflicts_string ~=3D conflicts_new then > + print("< " .. origin, conflicts_string) > + print("> " .. origin, conflicts_new .. " # " .. FILE_LIST[origin]) > + print() > + end > +end