From nobody Mon Aug 21 16:26:35 2023 X-Original-To: freebsd-pkgbase@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTyY41m2rz4r0hw for ; Mon, 21 Aug 2023 16:26:48 +0000 (UTC) (envelope-from dfr@rabson.org) Received: from mail-yw1-x1130.google.com (mail-yw1-x1130.google.com [IPv6:2607:f8b0:4864:20::1130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTyY371Ndz4Q8Y for ; Mon, 21 Aug 2023 16:26:47 +0000 (UTC) (envelope-from dfr@rabson.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1130.google.com with SMTP id 00721157ae682-58df8cab1f2so39385287b3.3 for ; Mon, 21 Aug 2023 09:26:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rabson-org.20221208.gappssmtp.com; s=20221208; t=1692635207; x=1693240007; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=1NPhLDxRDTyjBu6YZND1zQOYWHmXLMNk/QRJLT/jVi4=; b=oudt1aYwdbUc5STBHZjCCXW02ah9xxqZOZ9GM8hWbaXQGUfEYN1MmdJAUzdIT8F3Tv 1XnNZFrH0sJyvGJ/gqD+2PaK01e9yxm8LGcrj72SnGBT2JKYUPnM+49eRxvkw19SMj5d a1xp7Q2w6KFbEV+wDyfqQ/gK1iRe1Kwie8xom95iYToRwD0vn3OtJue3p/kwUoSB6Vqe /YDkPpfWQ0nRybFUnZxur59rLbpguA6Osj0KRjkBq1QT1xdrklda4ORpefBlUEl0PKIB ECzZg4K6ST0Mncz8bSldjbZO5swXR8nstbX5b1RqTTfCLqAd5K1gratGSBoYsGrKuiEK rxZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692635207; x=1693240007; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1NPhLDxRDTyjBu6YZND1zQOYWHmXLMNk/QRJLT/jVi4=; b=Hj9DajrkQGpf6DvHaQHgQ6kq/XydwCp335/p6FewiJGVdaPi01TVLOo2Vtx/C4X9EO BcFIuqu3EDer7TpfTyB1es7DSAJaXKKtntItY9wzwjfnEplTThFp2ryzSsEoPAm4rcU6 sW8PRMRQ4VdjTFgn48k74aStfwPhvLFSV3nGUMmEzxjCKPZ4teo/277X2REtaPkC65py TnHfE0lsP2EOEo9u9VIBFazJFMFPA5JR+CBO1hQ56ak09TEZ45DWljLpwFFhwErNxSbL X7ksfrtoM2nHSbcO9PsIofYlDU5AoUmNxBd33CyEYxSCs/o6kFzgR247G8Ho7terC+Id Jo+g== X-Gm-Message-State: AOJu0YzYF2Yl+jXO2t9pgUkcBZQs5jbyrBmvfoAq8uCLHQOxcWdJup8r fRuQp/XEBrL4LkVyn33/t9h49nyxn0HXN1AoLQcC8ZBSG5/3Q3J6qsI= X-Google-Smtp-Source: AGHT+IH6zSYQv6H5+daENU3VLJVWSlZub00cgem/jbbdVhDapejlE0oPrziKypJoonuw7sLYgSFBYcrdiTwnef0D8bU= X-Received: by 2002:a25:ae53:0:b0:d63:44e:cbcf with SMTP id g19-20020a25ae53000000b00d63044ecbcfmr6402261ybe.7.1692635206940; Mon, 21 Aug 2023 09:26:46 -0700 (PDT) List-Id: Packaging the FreeBSD base system List-Archive: https://lists.freebsd.org/archives/freebsd-pkgbase List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pkgbase@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Doug Rabson Date: Mon, 21 Aug 2023 17:26:35 +0100 Message-ID: Subject: Re: Repeatable builds using pkgbase To: Baptiste Daroussin Cc: freebsd-pkgbase@freebsd.org Content-Type: multipart/alternative; boundary="0000000000003e97df06037158db" X-Rspamd-Queue-Id: 4RTyY371Ndz4Q8Y X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --0000000000003e97df06037158db Content-Type: text/plain; charset="UTF-8" On Mon, 21 Aug 2023 at 17:23, Baptiste Daroussin wrote: > On Mon, Aug 21, 2023 at 02:33:24PM +0100, Doug Rabson wrote: > > While working on build scripts for FreeBSD container images, I wanted to > > get to the point where my builds are repeatable, i.e. if I create two > > images with the same set of packages installed in the same order, they > > should be identical. > > > > The main stumbling block is timestamps. I can force all the file > timestamps > > to a fixed value with buildah using the '--timestamp' argument to either > > 'buildah commit' or 'buildah build' but even then, the two images have > > different hashes. Looking deeper, the difference is in > > /var/db/pkg/local.sqlite. If I compare SQL dumps of the databases from > each > > image, I can see a timestamp embedded in the sqlite file: > > > > diff dump1 dump2 > > > > > > 4c4 > > < INSERT INTO packages > > VALUES(1,'base','FreeBSD-zoneinfo','13.2p2','zoneinfo package','zoneinfo > > package',NULL,NULL,'FreeBSD:13:amd64','re@FreeBSD.org',' > > https://www.FreeBSD.org > > > ','/',731014,0,0,1,1692446701,'2$2$c9w95oqai9bwhny1k4pcg8mji77xgk43zjxxb69j1duzq5jao18wak4deer85epmfpc8ngyysyt9wu74pg7sczkqc3ekyawkfgwzi8d',NULL,NULL,0); > > --- > > > INSERT INTO packages > > VALUES(1,'base','FreeBSD-zoneinfo','13.2p2','zoneinfo package','zoneinfo > > package',NULL,NULL,'FreeBSD:13:amd64','re@FreeBSD.org',' > > https://www.FreeBSD.org > > > ','/',731014,0,0,1,1692622924,'2$2$c9w95oqai9bwhny1k4pcg8mji77xgk43zjxxb69j1duzq5jao18wak4deer85epmfpc8ngyysyt9wu74pg7sczkqc3ekyawkfgwzi8d',NULL,NULL,0); > > > > > > Looking at the pkg source, I can see that the prepared statement for > > inserting into the packages table explicitly uses NOW() for this column. > > Would it be reasonable to allow changing this, e.g. by adding a command > > line argument to pkg to override the default? I haven't tried this to see > > if that makes the two databases identical - if not, I guess I'll just > > remove pkg metadata altogether. > > yes this would be reasonable, if you use en env var, please respect > SOURCE_DATE_EPOCH. > > I'll try this out, probably using an env var as you suggest. Hopefully there is nothing non-deterministic in sqlite which would stop this from being reproducible. Doug. --0000000000003e97df06037158db Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, 21 Aug 2023 at 17:23, Baptist= e Daroussin <bapt@freebsd.org>= ; wrote:
On Mon, Aug 21, 2023 at 02:33:24PM +0100= , Doug Rabson wrote:
> While working on build scripts for FreeBSD container images, I wanted = to
> get to the point where my builds are repeatable, i.e. if I create two<= br> > images with the same set of packages installed in the same order, they=
> should be identical.
>
> The main stumbling block is timestamps. I can force all the file times= tamps
> to a fixed value with buildah using the '--timestamp' argument= to either
> 'buildah commit' or 'buildah build' but even then, the= two images have
> different hashes. Looking deeper, the difference is in
> /var/db/pkg/local.sqlite. If I compare SQL dumps of the databases from= each
> image, I can see a timestamp embedded in the sqlite file:
>
> diff dump1 dump2
>
>
> 4c4
> < INSERT INTO packages
> VALUES(1,'base','FreeBSD-zoneinfo','13.2p2',&#= 39;zoneinfo package','zoneinfo
> package',NULL,NULL,'FreeBSD:13:amd64','re@FreeBSD.org&= #39;,'
> https://www.FreeBSD.org
> ','/',731014,0,0,1,1692446701,'2$2$c9w95oqai9bwhny1k4p= cg8mji77xgk43zjxxb69j1duzq5jao18wak4deer85epmfpc8ngyysyt9wu74pg7sczkqc3ekya= wkfgwzi8d',NULL,NULL,0);
> ---
> > INSERT INTO packages
> VALUES(1,'base','FreeBSD-zoneinfo','13.2p2',&#= 39;zoneinfo package','zoneinfo
> package',NULL,NULL,'FreeBSD:13:amd64','re@FreeBSD.org&= #39;,'
> https://www.FreeBSD.org
> ','/',731014,0,0,1,1692622924,'2$2$c9w95oqai9bwhny1k4p= cg8mji77xgk43zjxxb69j1duzq5jao18wak4deer85epmfpc8ngyysyt9wu74pg7sczkqc3ekya= wkfgwzi8d',NULL,NULL,0);
>
>
> Looking at the pkg source, I can see that the prepared statement for > inserting into the packages table explicitly uses NOW() for this colum= n.
> Would it be reasonable to allow changing this, e.g. by adding a comman= d
> line argument to pkg to override the default? I haven't tried this= to see
> if that makes the two databases identical - if not, I guess I'll j= ust
> remove pkg metadata altogether.

yes this would be reasonable, if you use en env var, please respect
SOURCE_DATE_EPOCH.

I'll try this out, probably using an env var as y= ou suggest. Hopefully there is nothing non-deterministic in sqlite which wo= uld stop this from being reproducible.

Doug.
=

--0000000000003e97df06037158db--