504 errors from cgit-beta

[Davíð Steinn Geirsson]

On Mon, Nov 30, 2020 at 10:24:50PM +0100, Ulrich Spörlein wrote:
> On Mon, 2020-11-30 at 15:06:42 +0000, Davíð Steinn Geirsson wrote:
> > On Fri, Nov 13, 2020 at 05:33:12PM +0800, Li-Wen Hsu wrote:
> > > On Fri, Nov 13, 2020 at 2:28 AM Yasuhiro KIMURA <yasu at utahime.org> wrote:
> > > >
> > > > From: Davíð Steinn Geirsson <david at isnic.is>
> > > > Subject: 504 errors from cgit-beta
> > > > Date: Thu, 12 Nov 2020 15:56:59 +0000
> > > >
> > > > > We are getting frequent 504 errors when running `git fetch` against an
> > > > > existing checkout of `ports.git` from https://cgit-beta.freebsd.org/ports.git:
> > > > >
> > > > > ```
> > > > > $ git fetch cgit-beta
> > > > > error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
> > > > > fatal: the remote end hung up unexpectedly
> > > > > ```
> > > >
> > > > I experienced same error when accessing Emacs git remository with
> > > > HTTPS. Following is bug report that I submitted to report the issue.
> > > >
> > > > https://savannah.nongnu.org/support/?110322
> > > >
> > > > As you can see, site administrator fixed the issue by icreasing
> > > > `fastcgi_read_timeout` and `proxy_read_timeout` parameters of
> > > > nginx. Since cgit-beta also uses nginx this may also fix your
> > > > error. In my case, however, access always failed and never
> > > > succeeded. So cause may be different from the one of my case.
> > > 
> > > Thanks, I have checked this, indeed some requests' handlers don't have
> > > a long enough timeout setting and I've relaxed them.  Hope this solves
> > > some people's issues. Please check it again, and if it still fails for
> > > you, we might need to have more information to debug.
> > 
> > This problem disappeared after your changes, but as of this weekend seems
> > to be happening again:
> > 
> > user at ssh:~/foo/ports$ git fetch -v cgit-beta
> > POST git-upload-pack (gzip 3272 to 1703 bytes)
> > POST git-upload-pack (gzip 2577 to 1354 bytes)
> > error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
> > fatal: the remote end hung up unexpectedly
> > 
> > Is it possible some web server config got overwritten during the last
> > batch of changes?
> 
> This is most definitely fallout from the commit hashes changing. That means
> your client will upload basically all the hashes or packs for the server to
> compare what it does and does not have.
> 
> What is your up/downstream bandwidth situation like? Could you try some more
> tracing as outlined here: https://stackoverflow.com/questions/27442134/git-fetch-hangs-on-git-upload-pack
> What sort of custom work do you have in there (branches, etc)? I'm curious
> to find out a way to reset this non-destructively ... and I have an idea.

Up/downstream should be good. Speedtests show ~100-160Mbit/s in both
directions. Cloning a repo from cgit-beta.freebsd.org I see 7.75 MiB/s.

The checkout I was working from had two branches: `upstream` which is
a 1:1 clone of the state of the `main` branch on cgit-beta, and `main`
which is the same but also has a couple of local ports in commits that
get rebased on top of `upstream` when it's updated. When this error
occurred I was on the `upstream` branch.

This was a manual test, but normally the same update-then-rebase process
happens as part of a CI job which was also failing.

It seems this is fixed now, as the last 4 runs of the CI job were successful
(first successful run was at 18:42 UTC). If I see a similar error again I'll
follow the linked steps and send a more detailed trace.

Thanks for your assistance,
-Davíð

> 
> Li-Wen, we need to further up the timeouts here, freebsd repos are big,
> after all.
> 
> Cheers
> Uli
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-git/attachments/20201201/08777cf9/attachment.sig>