Re: Impossible to push my ports directory to my CI/CD pipeline

On Tue, Oct 01, 2024 at 07:37:00PM +0000, Shawn Webb wrote:
> On Mon, Sep 30, 2024 at 01:09:32PM UTC, Matthias Fechner wrote:
> > Dear all,
> > 
> > I already tried to address that problem on the all-developer list, but it is
> > maybe better to put it here.
> > I think by accident someone pushed LFS object with a commit of java/eclipse.
> > 
> > git remote -v
> > freebsd git@gitrepo.freebsd.org:ports.git (fetch)
> > freebsd git@gitrepo.freebsd.org:ports.git (push)
> > freebsd-https?? https://git.freebsd.org/ports.git (fetch)
> > freebsd-https?? https://git.freebsd.org/ports.git (push)
> > githubfreebsd?? https://github.com/freebsd/freebsd-ports.git (fetch)
> > githubfreebsd?? https://github.com/freebsd/freebsd-ports.git (push)
> > origin? git@gitlab.fechner.net:mfechner/Gitlab.git (fetch)
> > origin? git@gitlab.fechner.net:mfechner/Gitlab.git (push)
> > 
> > git lfs fetch --all freebsd
> > fetch: 22 object(s) found, done.
> > fetch: Fetching all references...
> > [99a670c083e000008ee29dd972b60cbad6ade1d6ec024b30ff45f41da3d19e1d] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [c89e491472c8af1d153e2e2801836fad0e38f428d0250ed982e16baa5effbe4f] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [f7db6794c6c3e7a03a3987537b8a0cdb1bde2996ed7c7eec50f2e5df1e7a30c2] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [e8e2d306fd11358a07b4ee10bfef19de3b518793568fc498cdd989170d31712b] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [5dbe70b5c83520df09c100815bb164c8903da5eb8bffbd094aaaed54b212d73a] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [8813907b0f7b001080a346ed2d0307a1c749b85688fdfa1d270fe59a07aa3909] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [1d41b04bdd4115c52aac8177e1b899403a3d7d9f6d147cd864463029648a4714] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [c0155fc8e7c4b9736d81db638fed95bc9ff297453bfd0bbd7dd5d882ebca6e89] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [6d0aef4ba9e96d4c63baae8db9cdce7c634140a2bda7da670e698e38e0df57ce] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [a928176ec92ad347404b2454e5f4f321aaef8ba862a60b0c28adc3bdcc619a4a] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [42fa73302624dc511e6d8958c66c82f31b8d9155f2980343b6bcdc25345e1dcd] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [ed324ddc16ccb8b9fab3041da394b844166b6778c39cd12708bcade05b41c0da] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [9ef847d4e699b5dbe49ac74ee35596fdddc22c6335bc96affee19a99ecd077f5] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [f5417c277c6ca509334010bb6999c56c3ab5d3751204c74dc00c2fa25d93dfae] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [a1ffa591430df806983c8798113264f12ea679bce4b785d70d2868dc91cec883] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [b2e525c38598dac6885a75e7eaffa346c8aca3dbdda73cb0275f50103ec3d245] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [5db9424884396cf5bfc46a6b22dacbe245f43a7e1eda6aeac831905dc921802e] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [b5e05ef77effb4048e72e38605700151fdf4eae214cfa358e9e81a48e124cb52] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [ca2c21bf21e1fa2f53c72653ea57e76506c305bdd44c6416542ebbf2ab8b1b83] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [4fc786126dd64f936554a12c4752787c4989453654fc5d6fd119b6e5eeafec6f] Object
> > does not exist on the server: [404] Object does not exist on the server
> > [f56db5e81304e441260b84d6c2805ae7050ea56760d763d2b32e6be928fa1cae] Object
> > does not exist on the server: [404] Object does not exist on the server
> > error: failed to fetch some objects from
> > 'https://github.com/freebsd/freebsd-ports.git/info/lfs'
> > 
> > I have my own testing pipeline in place to automatically test my commits
> > against changes I did to ports.
> > 
> > But as LFS is now in the FreeBSD repository, but the files cannot be
> > retrieved from the FreeBSD repository I'm now locked out for my CI/CD
> > pipeline and my automated tests.
> > 
> > If I now try to push to my local repo I get:
> > 
> > git push origin
> > Warning: untrusted X11 forwarding setup failed: xauth key data not generated
> > Locking support detected on remote "origin". Consider enabling it with:
> > ? $ git config
> > lfs.https://gitlab.fechner.net/mfechner/Gitlab.git/info/lfs.locksverify true
> > Unable to find source for object
> > 99a670c083e000008ee29dd972b60cbad6ade1d6ec024b30ff45f41da3d19e1d (try
> > running git lfs fetch --all)
> > Uploading LFS objects:?? 0% (0/21), 0 B | 0 B/s, done.
> > error: failed to push some refs to 'gitlab.fechner.net:mfechner/Gitlab.git'
> > 
> > Does anyone have an idea how that can be fixed?
> 
> Hey Matthias,
> 
> With commit c94e4ced329298efb06189d1c6db26e3d6ff3eef[1], FreeBSD
> introduced LFS objects into the ports git history. This commit was
> reverted with commit 3ec21e417cd5752c96b88587a5a605567ba5cd19[2].
> 
> Even though the offending commit was reverted, the git commit history
> still contains LFS references. This caused HardenedBSD's self-hosted
> GitLab to become unhappy in exactly the same way yours did.
> 
> I ended up disabling LFS for the ports repo in the GitLab side. This
> caused GitLab to become happy again. Problem is, now everyone
> downstream from us (and FreeBSD, too) will need to disable LFS on
> their repos/forks, too.

I'm curious if you can turn LFS back on once the broken change is past?
That that ports should ever use LFS, but it seems somewhat better if
GitLab configs only need to be non-default during bootstrapping.

At this point there's probably nothing to be done other than workarounds
as rewriting history to remove this stuff would be quite disruptive
after several weeks of commits.

We could really use some hooks to block both things that look like lfs
metadata and submodule bits.  (The latter is somewhat challenging in that
we need to allow them on the vendor side to support OpenZFS.)

-- Brooks