From nobody Sat Jan 21 16:59:02 2023 X-Original-To: ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NzjJP4Mjxz32Kxm for ; Sat, 21 Jan 2023 16:59:17 +0000 (UTC) (envelope-from hiroo.ono@gmail.com) Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NzjJM6fWrz4MsT for ; Sat, 21 Jan 2023 16:59:15 +0000 (UTC) (envelope-from hiroo.ono@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=G3ACOE4P; spf=pass (mx1.freebsd.org: domain of hiroo.ono@gmail.com designates 2607:f8b0:4864:20::102e as permitted sender) smtp.mailfrom=hiroo.ono@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x102e.google.com with SMTP id lp10so4867556pjb.4 for ; Sat, 21 Jan 2023 08:59:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :reply-to:in-reply-to:references:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=PsXXdXwGRKAhbhsKYEUsk8dCllbp9LD28wDOGvVrHeM=; b=G3ACOE4PBioGwUqxOsyNky6MCd5ApwZFCG1NtjQK2X4N100WVvOtYgIebM39o7uChF CAsf44GuiprflUZshKGwAXW7uQRu6kWYCP6Uvqs+Ryr71qCpnoBmgVLn2fgrw95whAu3 ndrdJeTicGXMEcIqEWRMzSBCHe0Eai+f90W3wBIcVBpc0mHH7e8PkMw0D26GB//NAaJr PQCx+ZAMEnBaUMf8nzhrZf6BPElCcuRHyGGCkVnUVpKwzOPFtMdi7dmKSQhZtdDA1Y6s Rybf1e9voAKYnYnSsvBbmNiVZ2oyWYOK+XiCDtbsN/N/JKzKgh48GUY/oc12src7vLFU gz2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :reply-to:in-reply-to:references:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=PsXXdXwGRKAhbhsKYEUsk8dCllbp9LD28wDOGvVrHeM=; b=ZHWZkBe4VEg3QKeLsjcD1JQrhiE7lZdVbYMv65ckWuVK34uDUPmPDXlyoum2Qc+C9C /2gu9rHjy6pLrKl7gqTXtCZ6rag3yCG8qnn570z9JXAVYsXXqXjjrgyV+yRPw+V5hLcg qR5TZxrmXnBHWRt3GqqyCV8FRjh1+85rUsxhIwyyx9iJuGoHbOPW9LnGpvb0mY+jCJQ2 LviD0XHDZMVXtlaEXMFA8dqWiAEVLDF2XpHI4YYt4r31/4T10QEpZ1zBs9lPq1o6ehkE viYs/NXzxNUUIBMP4+BJJSeOestjJgWIeWtCt5jTKQ40zkjcJzD2ouJJqsIU7EHywwIE yOag== X-Gm-Message-State: AFqh2kopD+dUF9BDoKqFstxFsGqyKM1IOsPOrB4uReik+nS9pyrmEx7+ 2K8Utq+JCppHKaBEqx775aIsbjw21WfkZxkQQA41H/N5JJQ= X-Google-Smtp-Source: AMrXdXsOBNCbOMdhathFmRAHl2hutvwNxP5ArG07Y+koZxM+hndUtfI8003VvGpFiCDQV7r8r2QyPnGbGCmdcwv2PX0= X-Received: by 2002:a17:90a:8a86:b0:229:3efe:7999 with SMTP id x6-20020a17090a8a8600b002293efe7999mr2053499pjn.133.1674320353770; Sat, 21 Jan 2023 08:59:13 -0800 (PST) List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 References: <20230113181413.d38e665bfe1f9e8f155a7006@dec.sakura.ne.jp> <20230114105534.5e95a3fa5aa88f4194e2b9a6@dec.sakura.ne.jp> In-Reply-To: <20230114105534.5e95a3fa5aa88f4194e2b9a6@dec.sakura.ne.jp> Reply-To: hiroo.ono+freebsd@gmail.com From: =?UTF-8?B?SGlyb28gT25vICjlsI/ph47lr5vnlJ8p?= Date: Sun, 22 Jan 2023 01:59:02 +0900 Message-ID: Subject: Re: npm install, ports and poudriere To: Tomoaki AOKI Cc: ports@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.25 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_SPAM_SHORT(0.75)[0.751]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; HAS_REPLYTO(0.00)[hiroo.ono+freebsd@gmail.com]; FREEMAIL_REPLYTO(0.00)[gmail.com]; MLMMJ_DEST(0.00)[ports@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[ports@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102e:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TAGGED_FROM(0.00)[freebsd]; REPLYTO_ADDR_EQ_FROM(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4NzjJM6fWrz4MsT X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N 2023=E5=B9=B41=E6=9C=8814=E6=97=A5(=E5=9C=9F) 10:55 Tomoaki AOKI : > > On Fri, 13 Jan 2023 21:00:50 +0900 > Hiroo Ono (=E5=B0=8F=E9=87=8E=E5=AF=9B=E7=94=9F) wrote: > > > Thank you. > > > > 2023=E5=B9=B41=E6=9C=8813=E6=97=A5(=E9=87=91) 18:14 Tomoaki AOKI : > > > > > > On Fri, 13 Jan 2023 11:30:09 +0900 > > > Hiroo Ono (=E5=B0=8F=E9=87=8E=E5=AF=9B=E7=94=9F) wrote: > > > > > > > Hello, > > > > > > > > I made a port of MongoDB Compass https://www.mongodb.com/docs/compa= ss/current/ . > > > > As it is an electron app, I need to use "npm install" to build it. > > > > > > > > npm install does not work without network connection, while poudrie= re > > > > allow network connection only at fetch time. > > > > As a result, I need to do bunch of things in fetch mode, and the > > > > Makefile became as attached. Is this acceptable as a port? > > > > If not, I need to rework the Makefile to do npm install at build ti= me, > > > > but that makes the port not officially buildable I think. (though t= o > > > > each of us, there is ALLOW_NETWORKING_PACKAGES option in > > > > poudriere.conf) > > > > > > Can editors/vscode be any hint? It's electron app, too. > > > > > > https://cgit.freebsd.org/ports/tree/editors/vscode > > > > editors/vscode helped me very much when I wrote the Makefile, but it > > uses yarn which has the cache mechanism. > > The problem is that "npm install" always try to connect to network > > even if the package's tgz file is cached. > > cf. https://github.com/npm/npm/issues/7341 > > So I have to eliminate all the "npm install" after fetch target. > > What I am doing is similar to what Emanuel Haupt suggested to me. > > Doing "npm install" at post-fetch time, and backup node_modules directo= ries, > > and extract them at post-extract target. > > (Else, WRKDIR is wiped away at the beginning of extract target.) > > > > Every npm package is run "npm run install" with "npm install" and some = packages > > require cc, perl, python, rust, go and other things. > > Thus, FETCH_DEPENDS became like this: > > > > FETCH_DEPENDS=3D git:devel/git \ > > npm-node${NODE_VER_MAJOR}>0:www/npm-node${NODE_VER_MAJO= R} \ > > cmake:devel/cmake-core \ > > pkg-config:devel/pkgconf \ > > ${PYTHON_VERSION}:lang/python${PYTHON_SUFFIX} \ > > cargo:lang/rust \ > > # yarn:www/yarn-node${NODE_VER_MAJOR} \ > > ${PREFIX}/lib/libmongocrypt.so:databases/libmongocrypt = \ > > ${PREFIX}/lib/libsecret-1.so:security/libsecret > > . if !empty(PORT_OPTIONS:MGSSAPI_MIT) > > FETCH_DEPENDS+=3D ${_MITKRB5_DEPENDS} > > . else > > FETCH_DEPENDS+=3D ${_HEIMDAL_DEPENDS} > > . endif > > > > It requires not only npm, but also cmake, pkgconf, git, python, rust > > to build and some libraries to be linked > > in "npm run install". > > (The dependency to yarn is just a residue of the trial to use yarn's > > cache feature.) > > Is it ok or running cc and rust should be avoided at fetch time? is > > what I wanted to ask. > > It can be avoided with "npm install --ignore-scripts", but I have to > > run "npm run install" with each > > npm package myself solving whole the dependency tree. > > There are still things to do in do-build target. Not all the build > > things are done in post-fetch. > > > > So, I want to do "npm install" without "--ignore-scripts" in > > post-fetch target. If I have to do > > "npm run installs" in pre-build or do-build, in the order I have to > > determine digging all the package.json's. > > Google'd with "npm install offline from cache", found this. > > https://zameermanji.com/blog/2023/1/10/building-and-installing-an-npm-p= ackage-offline/ > > Does infos here help? > Maybe maintainers seems to be forced to `npm install` locally to get > package-lock.json file, IIUC. I tried, but did not succeed so far. 1. as HOME is ${WRKDIR} (at least with poudriere), the cache directory .npm/_cacache will be wiped out at the extract target. (see _EXTRACT_SEQ definition in bsd.port.mk) I can back up .npm directory in DISTDIR (or /tmp if it works) by tar, but it is nearly equal to backing up node_modules in DISTDIR. Is it OK to do so? It may be solvable like editors/vscode which downloads yarn cache from the maintainer's github repository, but I'd rather like to back up at fetch stage and later restore at extract stage. 2. I thought listing all npm package files (*.tgz) in DISTFILES and do 'npm install ${DISTDIR}/${DIST_SUBDIR}/*.tgz --offline' might work, but there is two problems I cannot solve. 2a. with npm, the two files below are different, but with ports, both fall into the same file in DISTDIR. https://registry.npmjs.org/lodash.isplainobject/-/lodash.isplainobject-4.0= .6.tgz https://registry.npmjs.org/@types/lodash.isplainobject/-/lodash.isplainobj= ect-4.0.6.tgz 2b. some npm packages are described to fetch by git from github. like: "node_modules/flat": { "version": "5.0.0", "resolved": "git+ssh://git@github.com/cipacda/flat.git#0453680d406afc82a88dbe1fb9816baa= d87c92af", it is not cached in .npm folder and even with --offline, git is kicked and fail as it cannot access to github. > If you prefer bundling all deps, could this help? > > https://stackoverflow.com/questions/43064107/how-to-install-npm-package= -while-offline packing after 'npm install' is a bad idea, but I will try similar way by packing after 'npm install --ignore-scripts' and do in npm run install in each folder in the dependency order (which I have to determine in some way). > > > I'm not at all familiar with electron nor Node.js, but is there any > > > cache mechanism for npm that can store required files into before > > > install (at `make fetch` phase)? Does below be of any help? > > > > > > https://docs.npmjs.com/cli/v7/commands/npm-cache > > > > > > > > > -- > > > Tomoaki AOKI > > > > > > -- > Tomoaki AOKI >