From nobody Wed Jul 12 04:05:34 2023 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R140f02F3z4mbg1 for ; Wed, 12 Jul 2023 04:05:54 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R140d53K8z4Qb7; Wed, 12 Jul 2023 04:05:53 +0000 (UTC) (envelope-from kob6558@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-5768a7e3adbso5185067b3.0; Tue, 11 Jul 2023 21:05:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689134752; x=1691726752; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=3mNFnJuNrZqyUAkr7rLDdwjP9KwzG+S0P/Y2zQfU4bU=; b=jtnMOKEKL40JhuffLaS9n5zvm16pIYEWTJXMBaCWX7fC/bY9pGewHdLgUWjA/TAqtR NF08diSeOpA8ldXoKZqaI9KPdSCx7sUO9SVRlPcbnrcWm1OGosRTX0So6dGmkH6CuXrm SpMrzOvduxumbqqJB47A74ZIdEcYlPDTQE9CGwhFkoRnpc54khhda/w7po9FJ7zhKaOs K4ETeSoar+pMcNopnKIOSQAgE1vZ3dOF2zgVm88R7N0R8dLnPm2e3ss8ZgS41lUSsc0y JvP/yYxJGxvAHY8mFHiJXkt1786O3H4C0/608ru1fFfNPzIUp+E2OykjZXI2QeF9iiX5 +bEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689134752; x=1691726752; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3mNFnJuNrZqyUAkr7rLDdwjP9KwzG+S0P/Y2zQfU4bU=; b=KvFvX7SK87v2qxdLKxB1RK3ArF9+KA8Ou+GC+r9XmK3wG2sqv7ue7OCcCLI3AlHeTN u7WamCTLkXl7rzuC2bOZrAPS9ojYGAxIdVEJzwx1V3zIZSWynj2Zrx1pIi+z6ioAmYni Rbai03D0AlB4RiNV+IdlZKRvkzge6j/5X5v2JOABd2Rxg/jOixVJW/NbZld3XC+BtO+P X6nuSXMorfLc4f7aAlvW/1+hQJRLSOYwqGh3C9iOYs2u8hxD947ZasylGxEDPX/6keDQ QoLDkbOBVBbGcnMU3aEBzNTQkgMUHcatPVZmjZqNhLycf/88/uuWjcunFsjVOz3lP7Tw 5ewQ== X-Gm-Message-State: ABy/qLZ9xw9EGZH3t/MlX+JA/uh//KOACfxP837ZRG7YEErUpO4n7HNW olwPjn1IMnkQGxJxFbHdn/9f+FbGIAErAps7LmE= X-Google-Smtp-Source: APBJJlEHEtf7d4heNYrIgPpM8MOh/4F4YLmVoFFlqZkal2ewFlnnwbbxFnv0K3o+xuzPcTxsfIvSvvnXs5QkZjKZFOA= X-Received: by 2002:a0d:dd0c:0:b0:579:e6e4:a165 with SMTP id g12-20020a0ddd0c000000b00579e6e4a165mr909020ywe.10.1689134751701; Tue, 11 Jul 2023 21:05:51 -0700 (PDT) List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 References: <2DCC0132-0307-4A73-A473-B749ABF87367.ref@yahoo.com> <2DCC0132-0307-4A73-A473-B749ABF87367@yahoo.com> <1aaeafcc-9812-64bd-a369-bae7b9fc0e5f@gwdg.de> <422DBC3B-D85F-4AFB-ABDE-842A08482EC8@yahoo.com> <58250d08-ce0a-c1ac-ed15-7d55d517218e@gwdg.de> <2ea168aa-3e0e-ee31-d4a3-82b4ce46b330@gwdg.de> <78c1497b-1ec7-08b6-a407-d671e3c4f9dd@gwdg.de> <44201182-73C9-46BC-A846-7DC765F5B81F@yahoo.com> In-Reply-To: <44201182-73C9-46BC-A846-7DC765F5B81F@yahoo.com> From: Kevin Oberman Date: Tue, 11 Jul 2023 21:05:34 -0700 Message-ID: Subject: Re: Problem with the package builds To: Mark Millard Cc: "Hurling, Rainer" , FreeBSD Mailing List Content-Type: multipart/alternative; boundary="000000000000da5e4c0600425494" X-Rspamd-Queue-Id: 4R140d53K8z4Qb7 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000da5e4c0600425494 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jul 11, 2023 at 1:10=E2=80=AFPM Mark Millard wr= ote: > On Jul 11, 2023, at 12:57, Kevin Oberman wrote: > > > On Mon, Jul 10, 2023 at 9:42=E2=80=AFPM Mark Millard wrote: > > On Jul 10, 2023, at 21:27, Rainer Hurling wrote: > > > > > As I understand it, the ports-mgmt/pkg of the system running Poudrier= e > must be updated beforehand? > > > > > > At least on my side, this seems to work as expected :) > > > > > > > poudriere builds pkg updates first (if needed) and then uses the pkg it > > built for building the later ports into packages. > > > > But, after the restarts of main-* builds, the FreeBSD build servers are > > still showing examples were, after an 1hr, some builds are still in > > build-depends. Also there was an example I saw were after 1.5 hr it was > > still in run-depends. > > > > It may be that things are improved but not fully fixed relative to > > some performance issues. > > > > =3D=3D=3D > > Mark Millard > > marklmi at yahoo.com > > A new build started this morning at 1:06 UTC and, with pkg-1.20.2, it'= s > better, but not much. It's running at 21 packages/hour, a 100% improvemen= t > on the last attempt which appears to have been killed last night. The log= s > indicate the installation of dependencies, but I don't see any sign of > caching. It's a re-install every time. (I may not understand how poudrier= e > does things, but I am pretty sure that caching is done.) > > Just about "caching" relative to "poudriere bulk" builds . . . > > Nope. At the end of a builder run of a port build the context is destroye= d. > At the start of the builder building its next port the context is recreat= ed > from scratch. The only ports installed are exactly the declared > dependencies, > no more, no less, for the new port to be built. > > Caching installed state would imply access to ports from prior build > activity that do not apply: It would make the build environment polluted > with > irrelevant history. poudriere's purpose is to have a "clean-room" context > for > each port build. Thus its construction of such a context for each port > build. > > Caching vs. not is not the source of the large increase in how long thing= s > take to build. > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > OK. I knew it started in a clean jail. I just thought that it might use some caching technique to speed repeated installs of a single port... not that I have any idea of how this might be done. I think I'll monitor the speed of installs. I iish build logs included timestamps --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000da5e4c0600425494 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Jul 11, 2023 at 1:10=E2= =80=AFPM Mark Millard <marklmi@yaho= o.com> wrote:
On Jul 11, 2023, at 12:57, Kevin Oberman = <rkoberman@gmai= l.com> wrote:

> On Mon, Jul 10, 2023 at 9:42=E2=80=AFPM Mark Millard <marklmi@yahoo.com> wrote:<= br> > On Jul 10, 2023, at 21:27, Rainer Hurling <rhurlin@gwdg.de> wrote:
>
> > As I understand it, the ports-mgmt/pkg of the system running Poud= riere must be updated beforehand?
> >
> > At least on my side, this seems to work as expected :)
> >
>
> poudriere builds pkg updates first (if needed) and then uses the pkg i= t
> built for building the later ports into packages.
>
> But, after the restarts of main-* builds, the FreeBSD build servers ar= e
> still showing examples were, after an 1hr, some builds are still in > build-depends. Also there was an example I saw were after 1.5 hr it wa= s
> still in run-depends.
>
> It may be that things are improved but not fully fixed relative to
> some performance issues.
>
> =3D=3D=3D
> Mark Millard
> marklmi at yahoo.com
>=C2=A0 A new build started this morning at 1:06 UTC and, with pkg-1.20.= 2, it's better, but not much. It's running at 21 packages/hour, a 1= 00% improvement on the last attempt which appears to have been killed last = night. The logs indicate the installation of dependencies, but I don't = see any sign of caching. It's a re-install every time. (I may not under= stand how poudriere does things, but I am pretty sure that caching is done.= )

Just about "caching" relative to "poudriere bulk" build= s . . .

Nope. At the end of a builder run of a port build the context is destroyed.=
At the start of the builder building its next port the context is recreated=
from scratch. The only ports installed are exactly the declared dependencie= s,
no more, no less, for the new port to be built.

Caching installed state would imply access to ports from prior build
activity that do not apply: It would make the build environment polluted wi= th
irrelevant history. poudriere's purpose is to have a "clean-room&q= uot; context for
each port build. Thus its construction of such a context for each port buil= d.

Caching vs. not is not the source of the large increase in how long things<= br> take to build.

=3D=3D=3D
Mark Millard
marklmi at yahoo.com
OK. I knew= it started in a clean jail. I just thought that it might use some caching = technique to speed repeated installs of a single port... not that I have an= y idea of how this might be done.

I th= ink I'll monitor the speed of installs. I iish build logs included time= stamps
-- <= br>
Kevin Oberman, Part t= ime kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
= PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
=
--000000000000da5e4c0600425494--