Re: npm install, ports and poudriere

2023年1月14日(土) 10:55 Tomoaki AOKI <junchoon@dec.sakura.ne.jp>:
>
> On Fri, 13 Jan 2023 21:00:50 +0900
> Hiroo Ono (小野寛生) <hiroo.ono+freebsd@gmail.com> wrote:
>
> > Thank you.
> >
> > 2023年1月13日(金) 18:14 Tomoaki AOKI <junchoon@dec.sakura.ne.jp>:
> > >
> > > On Fri, 13 Jan 2023 11:30:09 +0900
> > > Hiroo Ono (小野寛生) <hiroo.ono+freebsd@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I made a port of MongoDB Compass https://www.mongodb.com/docs/compass/current/ .
> > > > As it is an electron app, I need to use "npm install" to build it.
> > > >
> > > > npm install does not work without network connection, while poudriere
> > > > allow network connection only at fetch time.
> > > > As a result, I need to do bunch of things in fetch mode, and the
> > > > Makefile became as attached. Is this acceptable as a port?
> > > > If not, I need to rework the Makefile to do npm install at build time,
> > > > but that makes the port not officially buildable I think. (though to
> > > > each of us, there is ALLOW_NETWORKING_PACKAGES option in
> > > > poudriere.conf)
> > >
> > > Can editors/vscode be any hint? It's electron app, too.
> > >
> > >  https://cgit.freebsd.org/ports/tree/editors/vscode
> >
> > editors/vscode helped me very much when I wrote the Makefile, but it
> > uses yarn which has the cache mechanism.
> > The problem is that "npm install" always try to connect to network
> > even if the package's tgz file is cached.
> > cf. https://github.com/npm/npm/issues/7341
> > So I have to eliminate all the "npm install" after fetch target.
> > What I am doing is similar to what Emanuel Haupt suggested to me.
> > Doing "npm install" at post-fetch time, and backup node_modules directories,
> > and extract them at post-extract target.
> > (Else, WRKDIR is wiped away at the beginning of extract target.)
> >
> > Every npm package is run "npm run install" with "npm install" and some packages
> > require cc, perl, python, rust, go and other things.
> > Thus, FETCH_DEPENDS became like this:
> >
> > FETCH_DEPENDS=  git:devel/git \
> >                 npm-node${NODE_VER_MAJOR}>0:www/npm-node${NODE_VER_MAJOR} \
> >                 cmake:devel/cmake-core \
> >                 pkg-config:devel/pkgconf \
> >                 ${PYTHON_VERSION}:lang/python${PYTHON_SUFFIX} \
> >                 cargo:lang/rust \
> >                 # yarn:www/yarn-node${NODE_VER_MAJOR} \
> >                 ${PREFIX}/lib/libmongocrypt.so:databases/libmongocrypt \
> >                 ${PREFIX}/lib/libsecret-1.so:security/libsecret
> > .  if !empty(PORT_OPTIONS:MGSSAPI_MIT)
> > FETCH_DEPENDS+= ${_MITKRB5_DEPENDS}
> > .  else
> > FETCH_DEPENDS+= ${_HEIMDAL_DEPENDS}
> > .  endif
> >
> > It requires not only npm, but also cmake, pkgconf, git, python, rust
> > to build and some libraries to be linked
> > in "npm run install".
> > (The dependency to yarn is just a residue of the trial to use yarn's
> > cache feature.)
> > Is it ok or running cc and rust should be avoided at fetch time? is
> > what I wanted to ask.
> > It can be avoided with "npm install --ignore-scripts", but I have to
> > run "npm run install" with each
> > npm package myself solving whole the dependency tree.
> > There are still things to do in do-build target. Not all the build
> > things are done in post-fetch.
> >
> > So, I want to do "npm install" without "--ignore-scripts" in
> > post-fetch target. If I have to do
> > "npm run installs" in pre-build or do-build,  in the order I have to
> > determine digging all the package.json's.
>
> Google'd with "npm install offline from cache", found this.
>
>   https://zameermanji.com/blog/2023/1/10/building-and-installing-an-npm-package-offline/
>
> Does infos here help?
> Maybe maintainers seems to be forced to `npm install` locally to get
> package-lock.json file, IIUC.

I tried, but did not succeed so far.

1. as HOME is ${WRKDIR} (at least with poudriere), the cache directory
.npm/_cacache will be wiped out at the extract target. (see
_EXTRACT_SEQ definition in bsd.port.mk)
   I can back up .npm directory in DISTDIR (or /tmp if it works) by
tar, but it is nearly equal to backing up node_modules in DISTDIR. Is
it OK to do so?
  It may be solvable like editors/vscode which downloads yarn cache
from the maintainer's github repository, but I'd rather like to back
up at fetch stage and later restore at extract stage.
2. I thought listing all npm package files (*.tgz) in DISTFILES and do
'npm install ${DISTDIR}/${DIST_SUBDIR}/*.tgz --offline' might work,
but there is two problems I cannot solve.
2a. with npm, the two files below are different, but with ports, both
fall into the same file in DISTDIR.
 https://registry.npmjs.org/lodash.isplainobject/-/lodash.isplainobject-4.0.6.tgz
 https://registry.npmjs.org/@types/lodash.isplainobject/-/lodash.isplainobject-4.0.6.tgz
2b. some npm packages are described to fetch by git from github. like:
    "node_modules/flat": {
      "version": "5.0.0",
      "resolved":
"git+ssh://git@github.com/cipacda/flat.git#0453680d406afc82a88dbe1fb9816baad87c92af",
  it is not cached in .npm folder and even with --offline, git is
kicked and fail as it cannot access to github.

> If you prefer bundling all deps, could this help?
>
>   https://stackoverflow.com/questions/43064107/how-to-install-npm-package-while-offline

packing after 'npm install' is a bad idea, but I will try similar way
by packing after 'npm install --ignore-scripts' and do in npm run
install in each folder in the dependency order (which I have to
determine in some way).

> > > I'm not at all familiar with electron nor Node.js, but is there any
> > > cache mechanism for npm that can store required files into before
> > > install (at `make fetch` phase)? Does below be of any help?
> > >
> > >  https://docs.npmjs.com/cli/v7/commands/npm-cache
> > >
> > >
> > > --
> > > Tomoaki AOKI    <junchoon@dec.sakura.ne.jp>
> > >
>
>
> --
> Tomoaki AOKI    <junchoon@dec.sakura.ne.jp>
>