From nobody Thu May 18 00:35:43 2023 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QM9xj0WV4z4Bn6p for ; Thu, 18 May 2023 00:35:53 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [IPv6:2001:470:0:19b::b869:801b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "holgerdanske.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QM9xg5WXFz3nMn for ; Thu, 18 May 2023 00:35:51 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=holgerdanske.com header.s=nov-20210719-112354 header.b=fFA1Fu0V; spf=pass (mx1.freebsd.org: domain of dpchrist@holgerdanske.com designates 2001:470:0:19b::b869:801b as permitted sender) smtp.mailfrom=dpchrist@holgerdanske.com; dmarc=pass (policy=none) header.from=holgerdanske.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=holgerdanske.com; s=nov-20210719-112354; t=1684370144; bh=OWDVekcoPvee0YzRi2wAI9m3KiX41Gaagka6crLgEgE=; h=Received:Message-ID:Date:MIME-Version:User-Agent:Subject:To: References:Content-Language:From:In-Reply-To:Content-Type: Content-Transfer-Encoding; b=fFA1Fu0VONLKZycv7BJ+3C+uBj1woILnmZ8CnWBAp5iUL9/TIgkGA/Hj1eDTGPmQj OqBOoIusxrLk18rmMXCZec90dDQyhqyYTfzZQ6JK7dO/NgvBy9UNgeItgFpZ5ODWPt gtrL0aFaVk1cNaGwSqdFWD4nai7ugZnwo4+WKCZoo+406ozri001l6eJcgaA5Lhn4e JlIc4npiafvaU95e0g0kURsFxtLdWnQXc/FDP8+VatA5LKjxLhyVSeamkJgLnv7x1y UvLW5tU9sxZJExR4cOyjj/PlKV7+gtqW3UQYncOHPSHsEslSO5SscSNIrjM1wi1BQa zmctuDl+3kw2Ab/4KLoA+dQVf5iOk5nQOZ5TsB3ouSMHf7/5Y82LW4OMQlzKa4udz4 CzH+NRzoe5OToMSQhPc5Dt+u99qsi524yNc1qxFjHojIwN6M/2WmUakV5zK65KwLWe rQw6O4O07JbPc6dBFMVFu/Pkp5ujygQA/p84CHtjBc+Dik2FfKf5xa37iN+r1B6VXo 7N9Mm7yBhbBVQjjFZK9VVlycEpmJcreVru5kCipicOCY5RNFjr1Hz36vHAXvyx/9pI hEs9n6Sa1pnOqV8uAoEQKWBcjswKqgTYzTF7aXRnvS5/f5dtsxGXXt2w0LcUkJR659 C/mgOU4yatRZGsFliqPJ8bfE= Received: from 99.100.19.101 (99-100-19-101.lightspeed.frokca.sbcglobal.net [99.100.19.101]) by holgerdanske.com with ESMTPSA (TLS_AES_128_GCM_SHA256:TLSv1.3:Kx=any:Au=any:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Wed, 17 May 2023 17:35:44 -0700 Message-ID: <3e2b4ee6-c098-456a-bb3a-4b1f45e4d888@holgerdanske.com> Date: Wed, 17 May 2023 17:35:43 -0700 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: Tool to compare directories and delete duplicate files from one directory To: questions@freebsd.org References: <9887a438-95e7-87cc-a162-4ad7a70d744f@optiplex-networks.com> <7c2429c5-55d0-1649-a442-ce543f2d46c2@holgerdanske.com> <6a0aba81-485a-8985-d20d-6da58e9b5580@optiplex-networks.com> <347612746.1721811.1683912265841@fidget.co-bxl> <08804029-03de-e856-568b-74494dfc81cf@holgerdansk e.com> <126434505.494354.1684104532813@ichabod.co-bxl> <818813a2-8ab0-df5 4-3c59-0e1ba9ce743d@holgerdanske.com> <941908372.622746.1684189567246@ichabod.co-bxl> <1e30ac66-a339-ce08-75ac-8e566f4d2278@optiplex-networks.com> Content-Language: en-US From: David Christensen In-Reply-To: <1e30ac66-a339-ce08-75ac-8e566f4d2278@optiplex-networks.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[holgerdanske.com,none]; R_SPF_ALLOW(-0.20)[+a:november.he.net]; R_DKIM_ALLOW(-0.20)[holgerdanske.com:s=nov-20210719-112354]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[holgerdanske.com:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MLMMJ_DEST(0.00)[questions@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[questions@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4QM9xg5WXFz3nMn X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 5/17/23 00:55, Kaya Saman wrote: > > On 5/15/23 23:26, Sysadmin Lists wrote: >>> ---------------------------------------- >>> From: David Christensen >>> Date: May 15, 2023, 1:43:38 AM >>> To: >>> Subject: Re: Tool to compare directories and delete duplicate files >>> from one directory >>> >>> >>> I looks like your script only finds duplicates when the subpath is >>> identical (?): >>> >> Yeah. Wasn't that the original problem description? I went off the >> example >> given by Paul earlier in this thread, and it looked like only files with >> matching subpaths were being considered (because the OP accidentally >> rsync'd >> files from a source to a bunch of destination dirs). >> > > Glad to see this thread has turned into an interesting discussion.... > > > Just as the OP :-) I will clarify.... > > There was no accidental rsync in place. > > > Due to lack of storage my files where basically all over the place on > different zpools. The problem is that most of those were on iscsi drives > (all running Freebsd), so I needed to get them in a single place. Of > course as the files where all over things became a mess. > > I bought a few new drives and created a new zpool just for this case. So > virtually I had to sync the multiple directories to a single > destination. *but* of course I didn't use the --remove-source-files > option as I didn't want things to be destructive. > > > But then I needed the extra space too and that's where this post came from. > > > Regards, > > > Kaya I seem to recall that you decided to run a Perl script posted by a reader. How has that worked out? My first response presupposed that you wanted to delete /dir1, /dir2, and /dir3. Further messages indicated that you wanted to keep those directories and any unique files they contain. Please clarify your plans for those directories and their contents. How do you plan to validate the consolidation process when it is complete? David