From nobody Wed Oct 02 15:29:00 2024 X-Original-To: git@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XJdyK4YMKz5Xgdr for ; Wed, 02 Oct 2024 15:29:13 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XJdyK2QQjz3wsy for ; Wed, 2 Oct 2024 15:29:13 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-20ba9f3824fso19126715ad.0 for ; Wed, 02 Oct 2024 08:29:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1727882952; x=1728487752; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=qYoQD0VNoLO0Dphsu7TJXsuUUDCPeRwKi+c68LT9hHw=; b=Cj/nQAgQ8yN1edU4KU1s64EjORy8K+mrDvRNfSATo3+al8GsD/4Oq39RcZ3x879wSg roYZVHgarD9FxCJdxXufNjrHoeeR0sd9hUBP3LV9VF2xI0s61PPvJX/kyTd+QhJNTSTI Kyj8FTUQoYzPvsRZuDJ7T3mdre50UfeQ8YfN3RjVs8dUgaT767LYSNsXPwROPJgfJAib yTE8UOkWVJqBft0ikr8nfV4BoOkNrS2ngI8nr4UI1JuIUVTvClCpEvbqm9WlwxeqzsiM E0vQxlKI62xXetJ5ZfoKxxxVN44iVviQy4GiwK+OT8LsipDB4V+O/JqNyicdtM06csjn ID4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727882952; x=1728487752; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qYoQD0VNoLO0Dphsu7TJXsuUUDCPeRwKi+c68LT9hHw=; b=ijAUKyLeMS+GI5VNhkbsCAbzlk5RRCKarhkn4qTGyX/Kr+WmkVO+N1R31sOM/TC0RW rGVD8gKeYlH5CozqpqUlVw4AwYv3RVHTqOGK3d3Cr8jd1u4YyUL1qUfPJM30r7jhYBNx k/H59labOP99nkgI4biR7UBcOFfaTEFcF7bn7onM2tdcD8Fo/iqjGgB2f8ZDDDHSpThs zRobCMMcrlbEx3CXaCNuHI/bfWu6nqw/0E0J0cAeHnh00ko32GGhrN3cFxX0tq0F2nnE uHIn9UHWqm1EZ8wW1quryli5K7M9Px+xCR0iOEIHKBt+zSohiYzUiVgu6qTVjy78lEXu hxAw== X-Forwarded-Encrypted: i=1; AJvYcCW4NS05hhqXWDIo4Dxe75UT0KGmsaFW91xFHJA63yYzPW8Qu3WMn5jGWIGwKQsSb7Jj9hw=@freebsd.org X-Gm-Message-State: AOJu0Yznj3w3Pg3vCYv+2sPEioRNnblWffMR8OszRzPFNHUKq6dKfnIx HgXpfjvFXhnWShZGrBgKvLm2Wa6QbPKyI0aper3KOThQAGroR26etKVubYcUY9wQG9iF6EC/pcK 1VPUCnX8h3LuyWa806SAoOhi0A/MqKx4q21fYGA== X-Google-Smtp-Source: AGHT+IF7J7UCHxdpLSjDDO6PVp4IncyMxHY+l7Vd8+J1QJ0UnckzWVmJ9Azkg0tfQLbB5hpdaCMeDS/m/+5YVTbbFZY= X-Received: by 2002:a17:90b:3444:b0:2d3:d728:6ebb with SMTP id 98e67ed59e1d1-2e18451dc60mr4658862a91.5.1727882951867; Wed, 02 Oct 2024 08:29:11 -0700 (PDT) List-Id: Discussion of git use in the FreeBSD project List-Archive: https://lists.freebsd.org/archives/freebsd-git List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-git@FreeBSD.org MIME-Version: 1.0 References: <483fb132-0d1c-443a-9b44-b7f2f087fb3d@freebsd.org> <736exensjgsyhbytunglk6qnk3dh4fyyvdc5jktzrv3zzujbgr@db6amzcu53h2> In-Reply-To: From: Warner Losh Date: Wed, 2 Oct 2024 09:29:00 -0600 Message-ID: Subject: Re: Impossible to push my ports directory to my CI/CD pipeline To: Brooks Davis Cc: Shawn Webb , Matthias Fechner , git@freebsd.org Content-Type: multipart/alternative; boundary="0000000000008f88d70623801996" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4XJdyK2QQjz3wsy X-Spamd-Bar: ---- --0000000000008f88d70623801996 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Oct 2, 2024, 9:26=E2=80=AFAM Brooks Davis wrot= e: > On Tue, Oct 01, 2024 at 07:37:00PM +0000, Shawn Webb wrote: > > On Mon, Sep 30, 2024 at 01:09:32PM UTC, Matthias Fechner wrote: > > > Dear all, > > > > > > I already tried to address that problem on the all-developer list, bu= t > it is > > > maybe better to put it here. > > > I think by accident someone pushed LFS object with a commit of > java/eclipse. > > > > > > git remote -v > > > freebsd git@gitrepo.freebsd.org:ports.git (fetch) > > > freebsd git@gitrepo.freebsd.org:ports.git (push) > > > freebsd-https?? https://git.freebsd.org/ports.git (fetch) > > > freebsd-https?? https://git.freebsd.org/ports.git (push) > > > githubfreebsd?? https://github.com/freebsd/freebsd-ports.git (fetch) > > > githubfreebsd?? https://github.com/freebsd/freebsd-ports.git (push) > > > origin? git@gitlab.fechner.net:mfechner/Gitlab.git (fetch) > > > origin? git@gitlab.fechner.net:mfechner/Gitlab.git (push) > > > > > > git lfs fetch --all freebsd > > > fetch: 22 object(s) found, done. > > > fetch: Fetching all references... > > > [99a670c083e000008ee29dd972b60cbad6ade1d6ec024b30ff45f41da3d19e1d] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [c89e491472c8af1d153e2e2801836fad0e38f428d0250ed982e16baa5effbe4f] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [f7db6794c6c3e7a03a3987537b8a0cdb1bde2996ed7c7eec50f2e5df1e7a30c2] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [e8e2d306fd11358a07b4ee10bfef19de3b518793568fc498cdd989170d31712b] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [5dbe70b5c83520df09c100815bb164c8903da5eb8bffbd094aaaed54b212d73a] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [8813907b0f7b001080a346ed2d0307a1c749b85688fdfa1d270fe59a07aa3909] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [1d41b04bdd4115c52aac8177e1b899403a3d7d9f6d147cd864463029648a4714] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [c0155fc8e7c4b9736d81db638fed95bc9ff297453bfd0bbd7dd5d882ebca6e89] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [6d0aef4ba9e96d4c63baae8db9cdce7c634140a2bda7da670e698e38e0df57ce] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [a928176ec92ad347404b2454e5f4f321aaef8ba862a60b0c28adc3bdcc619a4a] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [42fa73302624dc511e6d8958c66c82f31b8d9155f2980343b6bcdc25345e1dcd] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [ed324ddc16ccb8b9fab3041da394b844166b6778c39cd12708bcade05b41c0da] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [9ef847d4e699b5dbe49ac74ee35596fdddc22c6335bc96affee19a99ecd077f5] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [f5417c277c6ca509334010bb6999c56c3ab5d3751204c74dc00c2fa25d93dfae] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [a1ffa591430df806983c8798113264f12ea679bce4b785d70d2868dc91cec883] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [b2e525c38598dac6885a75e7eaffa346c8aca3dbdda73cb0275f50103ec3d245] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [5db9424884396cf5bfc46a6b22dacbe245f43a7e1eda6aeac831905dc921802e] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [b5e05ef77effb4048e72e38605700151fdf4eae214cfa358e9e81a48e124cb52] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [ca2c21bf21e1fa2f53c72653ea57e76506c305bdd44c6416542ebbf2ab8b1b83] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [4fc786126dd64f936554a12c4752787c4989453654fc5d6fd119b6e5eeafec6f] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > [f56db5e81304e441260b84d6c2805ae7050ea56760d763d2b32e6be928fa1cae] > Object > > > does not exist on the server: [404] Object does not exist on the serv= er > > > error: failed to fetch some objects from > > > 'https://github.com/freebsd/freebsd-ports.git/info/lfs' > > > > > > I have my own testing pipeline in place to automatically test my > commits > > > against changes I did to ports. > > > > > > But as LFS is now in the FreeBSD repository, but the files cannot be > > > retrieved from the FreeBSD repository I'm now locked out for my CI/CD > > > pipeline and my automated tests. > > > > > > If I now try to push to my local repo I get: > > > > > > git push origin > > > Warning: untrusted X11 forwarding setup failed: xauth key data not > generated > > > Locking support detected on remote "origin". Consider enabling it wit= h: > > > ? $ git config > > > lfs. > https://gitlab.fechner.net/mfechner/Gitlab.git/info/lfs.locksverify true > > > Unable to find source for object > > > 99a670c083e000008ee29dd972b60cbad6ade1d6ec024b30ff45f41da3d19e1d (try > > > running git lfs fetch --all) > > > Uploading LFS objects:?? 0% (0/21), 0 B | 0 B/s, done. > > > error: failed to push some refs to 'gitlab.fechner.net: > mfechner/Gitlab.git' > > > > > > Does anyone have an idea how that can be fixed? > > > > Hey Matthias, > > > > With commit c94e4ced329298efb06189d1c6db26e3d6ff3eef[1], FreeBSD > > introduced LFS objects into the ports git history. This commit was > > reverted with commit 3ec21e417cd5752c96b88587a5a605567ba5cd19[2]. > > > > Even though the offending commit was reverted, the git commit history > > still contains LFS references. This caused HardenedBSD's self-hosted > > GitLab to become unhappy in exactly the same way yours did. > > > > I ended up disabling LFS for the ports repo in the GitLab side. This > > caused GitLab to become happy again. Problem is, now everyone > > downstream from us (and FreeBSD, too) will need to disable LFS on > > their repos/forks, too. > > I'm curious if you can turn LFS back on once the broken change is past? > That that ports should ever use LFS, but it seems somewhat better if > GitLab configs only need to be non-default during bootstrapping. > > At this point there's probably nothing to be done other than workarounds > as rewriting history to remove this stuff would be quite disruptive > after several weeks of commits. > > We could really use some hooks to block both things that look like lfs > metadata and submodule bits. (The latter is somewhat challenging in that > we need to allow them on the vendor side to support OpenZFS.) > I didn't think we imported the submodules of OpenZFS to the vendor branch, except to the extent that data was plain files in that import not otherwise used... Warner > > --0000000000008f88d70623801996 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Oct 2, 2024, 9:26=E2=80=AFAM Brooks Davis <= brooks@freebsd.org> wrote:
=
On Tue, Oct 01, 2024 at 07:37:00PM +00= 00, Shawn Webb wrote:
> On Mon, Sep 30, 2024 at 01:09:32PM UTC, Matthias Fechner wrote:
> > Dear all,
> >
> > I already tried to address that problem on the all-developer list= , but it is
> > maybe better to put it here.
> > I think by accident someone pushed LFS object with a commit of ja= va/eclipse.
> >
> > git remote -v
> > freebsd git@gitrepo.freebsd.org:ports.git (fetch)
> > freebsd git@gitrepo.freebsd.org:ports.git (push)
> > freebsd-https?? https://git.freebsd.org/ports.= git (fetch)
> > freebsd-https?? https://git.freebsd.org/ports.= git (push)
> > githubfreebsd?? https://github.com/= freebsd/freebsd-ports.git (fetch)
> > githubfreebsd?? https://github.com/= freebsd/freebsd-ports.git (push)
> > origin? git@gitlab.fechner.net:mfechner/Gitlab.git (fetch)
> > origin? git@gitlab.fechner.net:mfechner/Gitlab.git (push)
> >
> > git lfs fetch --all freebsd
> > fetch: 22 object(s) found, done.
> > fetch: Fetching all references...
> > [99a670c083e000008ee29dd972b60cbad6ade1d6ec024b30ff45f41da3d19e1d= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [c89e491472c8af1d153e2e2801836fad0e38f428d0250ed982e16baa5effbe4f= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [f7db6794c6c3e7a03a3987537b8a0cdb1bde2996ed7c7eec50f2e5df1e7a30c2= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [e8e2d306fd11358a07b4ee10bfef19de3b518793568fc498cdd989170d31712b= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [5dbe70b5c83520df09c100815bb164c8903da5eb8bffbd094aaaed54b212d73a= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [8813907b0f7b001080a346ed2d0307a1c749b85688fdfa1d270fe59a07aa3909= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [1d41b04bdd4115c52aac8177e1b899403a3d7d9f6d147cd864463029648a4714= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [c0155fc8e7c4b9736d81db638fed95bc9ff297453bfd0bbd7dd5d882ebca6e89= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [6d0aef4ba9e96d4c63baae8db9cdce7c634140a2bda7da670e698e38e0df57ce= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [a928176ec92ad347404b2454e5f4f321aaef8ba862a60b0c28adc3bdcc619a4a= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [42fa73302624dc511e6d8958c66c82f31b8d9155f2980343b6bcdc25345e1dcd= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [ed324ddc16ccb8b9fab3041da394b844166b6778c39cd12708bcade05b41c0da= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [9ef847d4e699b5dbe49ac74ee35596fdddc22c6335bc96affee19a99ecd077f5= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [f5417c277c6ca509334010bb6999c56c3ab5d3751204c74dc00c2fa25d93dfae= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [a1ffa591430df806983c8798113264f12ea679bce4b785d70d2868dc91cec883= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [b2e525c38598dac6885a75e7eaffa346c8aca3dbdda73cb0275f50103ec3d245= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [5db9424884396cf5bfc46a6b22dacbe245f43a7e1eda6aeac831905dc921802e= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [b5e05ef77effb4048e72e38605700151fdf4eae214cfa358e9e81a48e124cb52= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [ca2c21bf21e1fa2f53c72653ea57e76506c305bdd44c6416542ebbf2ab8b1b83= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [4fc786126dd64f936554a12c4752787c4989453654fc5d6fd119b6e5eeafec6f= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > [f56db5e81304e441260b84d6c2805ae7050ea56760d763d2b32e6be928fa1cae= ] Object
> > does not exist on the server: [404] Object does not exist on the = server
> > error: failed to fetch some objects from
> > 'https://github.com/fr= eebsd/freebsd-ports.git/info/lfs'
> >
> > I have my own testing pipeline in place to automatically test my = commits
> > against changes I did to ports.
> >
> > But as LFS is now in the FreeBSD repository, but the files cannot= be
> > retrieved from the FreeBSD repository I'm now locked out for = my CI/CD
> > pipeline and my automated tests.
> >
> > If I now try to push to my local repo I get:
> >
> > git push origin
> > Warning: untrusted X11 forwarding setup failed: xauth key data no= t generated
> > Locking support detected on remote "origin". Consider e= nabling it with:
> > ? $ git config
> > lfs.https://= gitlab.fechner.net/mfechner/Gitlab.git/info/lfs.locksverify true
> > Unable to find source for object
> > 99a670c083e000008ee29dd972b60cbad6ade1d6ec024b30ff45f41da3d19e1d = (try
> > running git lfs fetch --all)
> > Uploading LFS objects:?? 0% (0/21), 0 B | 0 B/s, done.
> > error: failed to push some refs to 'gitlab.fechner.net:mfechn= er/Gitlab.git'
> >
> > Does anyone have an idea how that can be fixed?
>
> Hey Matthias,
>
> With commit c94e4ced329298efb06189d1c6db26e3d6ff3eef[1], FreeBSD
> introduced LFS objects into the ports git history. This commit was
> reverted with commit 3ec21e417cd5752c96b88587a5a605567ba5cd19[2].
>
> Even though the offending commit was reverted, the git commit history<= br> > still contains LFS references. This caused HardenedBSD's self-host= ed
> GitLab to become unhappy in exactly the same way yours did.
>
> I ended up disabling LFS for the ports repo in the GitLab side. This > caused GitLab to become happy again. Problem is, now everyone
> downstream from us (and FreeBSD, too) will need to disable LFS on
> their repos/forks, too.

I'm curious if you can turn LFS back on once the broken change is past?=
That that ports should ever use LFS, but it seems somewhat better if
GitLab configs only need to be non-default during bootstrapping.

At this point there's probably nothing to be done other than workaround= s
as rewriting history to remove this stuff would be quite disruptive
after several weeks of commits.

We could really use some hooks to block both things that look like lfs
metadata and submodule bits.=C2=A0 (The latter is somewhat challenging in t= hat
we need to allow them on the vendor side to support OpenZFS.)

I didn't t= hink we imported the submodules of OpenZFS to the vendor branch, except to = the extent that data was plain files in that import not otherwise used...

Warner

--0000000000008f88d70623801996--