From nobody Mon Jul 04 17:20:16 2022 X-Original-To: ports-bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4817D1BE12F0 for ; Mon, 4 Jul 2022 17:20:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LcCHM6PgCz4ckW for ; Mon, 4 Jul 2022 17:20:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BCE0D156C9 for ; Mon, 4 Jul 2022 17:20:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 264HKFLC035594 for ; Mon, 4 Jul 2022 17:20:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 264HKFNR035593 for ports-bugs@FreeBSD.org; Mon, 4 Jul 2022 17:20:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: ports-bugs@FreeBSD.org Subject: [Bug 264950] New port: misc/dedup Date: Mon, 04 Jul 2022 17:20:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: borodin@unix7.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: ports-bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Ports bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-ports-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports-bugs@freebsd.org X-BeenThere: freebsd-ports-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1656955216; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w3o9Y+Kz/y9IcdMXr/M189B/zFSvxwcJR1PU7NGIi0Y=; b=tmlKFBBTYvzlKxswh9Rd8mt2Y5Gqm84LN+MCeL4YeZhNKrImnDwqrp/3donCo6+I8EnVOM uXWsUHDVKq/lsD7/tBetG7bWfgGkkcw5OAJq/nVnPrU/qF8aREZQ1UbBHGlFGSHr7h8OcQ EZQXzIz3qjcAYRgSWoNGOwgmM/0aRsbse66SNjhSE3IS50SCJcH6XrHjohE+Cp7v0ga6qE E4JvyhOd2DfKSqsW++DyflMUYk2jABSjLQur5HfwrU26A+ukt8ep6Fu/Afp49dbcHoDgA6 QBrb7VdUoWBYtNlk3Iou4kipWouXxkH0GHs+ErZ4mcnbxY9p/4k+MO4thtucWQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1656955216; a=rsa-sha256; cv=none; b=BG3RpTa8hlPk1sam5MxDCTAo4t5mQmpMTQ1ZM/ITGF48PV/68px3bd4R1ZyhQiBlXBs3kg AFvqfH9+uwvIVSMWmVNx/iahMXghoIhOPB4tSGjyQwvyqJAACZIaD55V9Hyl4sd118xlLL fEYfMiCuHgntEkpUyM2O7ZvNs5r9krFP8zMSehDrFnAOVd+YsYtmBp3qLHpuRXpkRVbvaR NqVd/wo5kHAf0N2eirfrnktwyHykzxkxvudOBXFqA8gfvfI4GIElm5guMy9NxP6XHvXn+/ lk1DklbfEu4FCZLJYENnxmqmdxG09hKjIwGNfSfhSgJBvm9S4xINxbFVBsqSMw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264950 --- Comment #5 from Oleg Borodin --- (In reply to Daniel Engberg from comment #4) sysutils/duff - uses SHA1 to SHA256 sysutils/dupd - uses xxHash sysutils/fdupes - uses MD5=20 sysutils/filedupe - uses MD5 sysutils/jdupes - uses xxHash sysutils/rmlint - one utility which can use the HighwayHash algorithm, but = is overloaded with functionality and dependencies. I used one of the fastest HighwayHash which is comparable to xxHash in spee= d, but as far as I understand from tests it has less chance of collisions even with unchanged initial vector (I could be wrong here). All you need to do to find files with duplicate content is to hash sum the content and compare against a database in the file tree, and print the resu= lt (of course before doing so check that the file sizes match, because that's = easy to do). But if you think this port duplicates the functionality of the others and doesn't make sense to add it, so be it. Perhaps it makes sense to close this ticket. --=20 You are receiving this mail because: You are the assignee for the bug.=