From nobody Thu May 04 21:47:44 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QC6ql2rX8z49Z5Y for ; Thu, 4 May 2023 21:47:47 +0000 (UTC) (envelope-from kayasaman@optiplex-networks.com) Received: from mail.optiplex-networks.com (mail.optiplex-networks.com [212.159.80.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QC6qk5NCzz3R11 for ; Thu, 4 May 2023 21:47:46 +0000 (UTC) (envelope-from kayasaman@optiplex-networks.com) Authentication-Results: mx1.freebsd.org; none Received: from localhost (localhost [127.0.0.1]) by mail.optiplex-networks.com (Postfix) with ESMTP id 13A1F15C2C76; Thu, 4 May 2023 22:47:45 +0100 (BST) Received: from mail.optiplex-networks.com ([127.0.0.1]) by localhost (mail.optiplex-networks.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Y67IPZkktF1G; Thu, 4 May 2023 22:47:44 +0100 (BST) Received: from localhost (localhost [127.0.0.1]) by mail.optiplex-networks.com (Postfix) with ESMTP id 8DFCC15C2DBC; Thu, 4 May 2023 22:47:44 +0100 (BST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.optiplex-networks.com 8DFCC15C2DBC DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=optiplex-networks.com; s=AE93A2AC-7F67-11EA-90AE-8A1FE64F6997; t=1683236864; bh=5xK6M0QHr2aDp1L7iB1zTN9nCkRkLKovLp2scuttzlM=; h=Message-ID:Date:MIME-Version:To:From; b=EOMPDkY0opOsK1u0pfZTvDI6HXsDILYAo1+ZEOXg9jcEQREuqPWl6/TxluIfqegHC 2zU742uRyOk97sE1RpIXB+SLmYMZo6CWPzojsP8B1lr0U0Q5DUnzqG+kwIc/NnQUK2 bqLmkhFDCIXuL6tB0tXIzIplVWQogB8Qs2HgUgh/WhXqgLsR3BkZJB2znZ/hwS4f0/ NIf9RvtNhF7T2LdDRpkUqf/s+WBQSnHcRQjqPtHAQUlIuVUva2YDzv1Uh8FKNatNMg saCYL8xQRHaoRwEmo4W4SgZVmgi6Y0eXQrS19uVqfXOE7OsaavTBjNzvZHed0vO5ep M/sROcysuvFSQ== X-Virus-Scanned: amavisd-new at mail.optiplex-networks.com Received: from mail.optiplex-networks.com ([127.0.0.1]) by localhost (mail.optiplex-networks.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id gl1Bld1DpSHG; Thu, 4 May 2023 22:47:44 +0100 (BST) Received: from [192.168.20.23] (unknown [192.168.20.23]) by mail.optiplex-networks.com (Postfix) with ESMTPSA id 730FB15C2C76; Thu, 4 May 2023 22:47:44 +0100 (BST) Content-Type: multipart/alternative; boundary="------------nwzKkVn1jzK0pw3WOSUpGRKx" Message-ID: <344b29c6-3d69-543d-678d-c2433dbf7152@optiplex-networks.com> Date: Thu, 4 May 2023 22:47:44 +0100 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: Tool to compare directories and delete duplicate files from one directory Content-Language: en-US To: Paul Procacci Cc: freebsd-questions@freebsd.org References: <9887a438-95e7-87cc-a162-4ad7a70d744f@optiplex-networks.com> From: Kaya Saman In-Reply-To: X-Rspamd-Queue-Id: 4QC6qk5NCzz3R11 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6871, ipnet:212.159.64.0/18, country:GB] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------nwzKkVn1jzK0pw3WOSUpGRKx Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable On 5/4/23 17:29, Paul Procacci wrote: > > > On Thu, May 4, 2023 at 11:53=E2=80=AFAM Kaya Saman=20 > wrote: > > Hi, > > > I'm wondering if anyone knows of a tool like diff or so that can al= so > delete files based on name and size from either left/right or > source/destination directory? > > > Basically what I have done is performed an rsync without using the > --remove-source-files option onto a newly bought and created disk > pool > (yes zpool) that i am trying to consolidate my data - as it's > currently > spread out over multiple pools with the same folder name. > > > The issue I am facing mainly is that I perform another rsync and > use the > --remove-source-files option, rsync will delete files based on name > while there are some files that have the same name but not same > size and > I would like to retain these files. > > > Right now I have looked at many different options in both rsync and > other tools but found nothing suitable. I even tested using a few > test > dirs and files that I put into /tmp and whatever I tried, the > files of > different size either got transferred or deleted. > > > How would be a good way to approach this problem? > > > Even if I create some kind of shell script and use diff, I think > it will > only compare names and not file sizes. > > > I'm really lost here.... > > > Regards, > > > Kaya > > > > > It sounds like you want fdupes.=C2=A0 It's in the ports tree. > > ~Paul > > --=20 > __________________ > > :(){ :|:& };: I tried fdupes and installed it a while back. For me it felt like it=20 only works on a single directory. My dir structure is that I have" /dir <- main directory where everything has now been rsync'ed to /dir_1 <- old directory with partial content /dir_2 <- more partial content /dir_3 <- more partial content The key thing here is that I need to compare: /dir_(x) with /dir if the files are different sizes in /dir_(x) then leave them, otherwise=20 delete if both name and file size are the same. --------------nwzKkVn1jzK0pw3WOSUpGRKx Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On 5/4/23 17:29, Paul Procacci wrote:<= br>


On Thu, May 4, 2023 at 11:53=E2=80=AFAM Kaya Saman <= kayasaman@optiplex-networks.com> wrote:
Hi,


I'm wondering if anyone knows of a tool like diff or so that can also
delete files based on name and size from either left/right or
source/destination directory?


Basically what I have done is performed an rsync without using the
--remove-source-files option onto a newly bought and created disk pool
(yes zpool) that i am trying to consolidate my data - as it's currently
spread out over multiple pools with the same folder name.

The issue I am facing mainly is that I perform another rsync and use the
--remove-source-files option, rsync will delete files based on name
while there are some files that have the same name but not same size and
I would like to retain these files.


Right now I have looked at many different options in both rsync and
other tools but found nothing suitable. I even tested using a few test
dirs and files that I put into /tmp and whatever I tried, the files of
different size either got transferred or deleted.


How would be a good way to approach this problem?


Even if I create some kind of shell script and use diff, I think it will
only compare names and not file sizes.


I'm really lost here....


Regards,


Kaya




It sounds like you want fdupes.=C2=A0 It's in the ports tree= .

~Paul

--
__________________
:(){ :|:& };:



I tried fdupes and installed it a while back. For me it felt like it only works on a single directory.


My dir structure is that I have"


/dir <- main directory where everything has now been rsync'ed to

/dir_1 <- old directory with partial content

/dir_2 <- more partial content

/dir_3 <- more partial content


The key thing here is that I need to compare:


/dir_(x) with /dir


if the files are different sizes in /dir_(x) then leave them, otherwise delete if both name and file size are the same.

--------------nwzKkVn1jzK0pw3WOSUpGRKx--