Understand npm concepts

Dec 14 2022

Last updated on Nov 27 2024

package.json and package-lock.json

package-lock.json (called package locks, or lockfiles) is automatically generated for any operations where npm modifies either the node_modules tree or package.json. This file is intended to be committed into source repositories. The purpose of the package-lock.json is to avoid the situation where installing modules from the same package.json results in two different installs. package-lock.json is a large list of each dependency listed in your package.json, the specific version that should be installed, the location (URI) of the module, a hash that verifies the integrity of the module, the list of packages it requires.

If you run npm i against the package.json and package-lock.json, the latter will never be updated, even if the package.json would be happy with newer versions.
If you manually edit your package.json to have different ranges and run npm i and those ranges aren’t compatible with your package-lock.json, then the latter will be updated with version that are compatible with your package.json.
Listed dependencies in package-lock.json file have mixed (sha1/sha512) integrity checksum. npm changed the integrity checksum from sha1 to sha512. Only packages published with npm@5 or later will include a sha512 integrity hash.

Two fields are mandatory in package.json:

name, can be scoped

version, has to be a valid SemVer number

Package code entry points:

main, default entry point (CJS or ESM)

module, ESM-specific entry point

exports, modern entry points, more flexible

Benefits of `exports` field

// package.json
{
  "name": "my-package",
  "type": "module",
  "exports": {
    ".": {
      // Entry-point for `import "my-package"` in ESM
      "import": {
        // Where TypeScript will look
        "types": "./types/esm/index.d.ts",
        // Where Node.js will look
        "default": "./esm/index.js"
      },
      // Entry-point for `require("my-package") in CJS
      "require": {
        "types": "./types/commonjs/index.d.cts",
        "default": "./commonjs/index.cjs"
      },
    }
  },
  // Fall-back for older versions of TypeScript
  "types": "./types/index.d.ts",
  // CJS fall-back for older versions of Node.js
  "main": "./commonjs/index.cjs"
}

By default, TypeScript overlays the same rules with import conditions – if you write an import from an ES module, it will look up the import field, and from a CommonJS module, it will look at the require field. If it finds them, it will look for a corresponding declaration file. If you need to point to a different location for your type declarations, you can add a "types" import condition. Note that the "types" condition should always come first in "exports".

Protecting internal files: Previously, consumers could import any file in a package, even internal ones. With exports, maintainers can explicitly define which files are accessible, establishing a clear public API and preventing unintended imports of internal files.
Mapping subpaths to dist directory: Package authors may prefer not to have dist in the import path for a simpler API. With exports, package subpaths can map directly inside the dist directory, allowing consumers to use cleaner imports like import foo from 'pkg-a/util' without complex publishing scripts for maintainers.
Multi-format packages: Packages can toggle entry points to resolve to different files for different environments (e.g., Node.js vs. browsers) and module types (e.g., CJS vs. ESM).

Read How To Create An NPM Package by Total TypeScript

Create a package.json with:

files is an array of files that should be included when people install your package. In this case, we’re including the dist folder. README.md, package.json and LICENSE are included by default.
type is set to module to indicate that your package uses ECMAScript modules, not CommonJS modules.

@arethetypeswrong/cli is a tool that checks if your package exports are correct. Add a script "check-exports": "attw --pack ." to check if all exports from your package are correct.

Add a main field to your package.json with "main": "dist/index.js", and our package is compatible with systems running ESM.

npm run check-exports
┌───────────────────┬──────────────────────────────┐
│                   │ "tt-package-demo"            │
├───────────────────┼──────────────────────────────┤
│ node10            │ 🟢                           │
├───────────────────┼──────────────────────────────┤
│ node16 (from CJS) │ ⚠️ ESM (dynamic import only) │
├───────────────────┼──────────────────────────────┤
│ node16 (from ESM) │ 🟢 (ESM)                     │
├───────────────────┼──────────────────────────────┤
│ bundler           │ 🟢                           │
└───────────────────┴──────────────────────────────┘

If you want to publish both CJS and ESM code, you can use tsup. This is a tool built on top of esbuild that compiles your TypeScript code into both formats. We’ll now be running tsup to compile our code instead of tsc.

// tsup.config.ts
import { defineConfig } from "tsup";

export default defineConfig({
  entryPoints: ["src/index.ts"],
  format: ["cjs", "esm"],
  dts: true,
  outDir: "dist",
  clean: true,
});

This will create a dist/index.js (for ESM) and a dist/index.cjs (for CJS). Add an exports field to your package.json, which tells programs consuming your package how to find the CJS and ESM versions of your package. In this case, we’re pointing folks using import to dist/index.js and folks using require to dist/index.cjs. Run check-exports again, everything is green.

{
  "exports": {
    ".": {
      "import": "./dist/index.js",
      "require": "./dist/index.cjs"
    }
  }
}

tsup also creates declaration files for each of your outputs. index.d.ts for ESM and index.d.cts for CJS. This means you don’t need to specify types in your package.json. TypeScript can automatically find the declaration file it needs.

npm install and npm ci

npm install reads package.json to create a list of dependencies and uses package-lock.json to inform which versions of these dependencies to install. If a dependency is not in package-lock.json it will be added by npm install.

npm ci (named after Continuous Integration) installs dependencies directly from package-lock.json and uses package.json only to validate that there are no mismatched versions. If any dependencies are missing or have incompatible versions, it will throw an error. It will delete any existing node_modules folder to ensure a clean state. It never writes to package.json or package-lock.json. It does however expect a package-lock.json file in your project — if you do not have this file, npm ci will not work and you have to use npm install instead.

npm audit automatically runs when you install a package with npm install. It checks direct dependencies and devDependencies, but does not check peerDependencies. Read more about npm audit: Broken by Design by Dan Abramov.

npm outdated, a built-in npm command, will check the registry to see if any installed packages are currently outdated. By default, only the direct dependencies of the root project are shown. Use --all to find all outdated meta-dependencies as well.

depcheck check your npm module for unused dependencies.

Taze is a modern cli tool that keeps your deps fresh. No installation required — npx taze. -g for global and -I for interactive.

npm-check-updates upgrades your package.json dependencies to the latest versions, ignoring specified versions.

Npm Burst tracks statistics of released npm packages and installed version counts.

Version Lens VS Code extension shows the latest version for each package.

npm ls

npm ls (aliases: list, la, ll) list dependencies that have been installed to node_modules. It throws an error for discrepancies between package.json and its lock file.

If depth is not set (default is 1), npm ls will show only the immediate dependencies of the root project.
npm ls <package> to check a specific package.

const cp = require("child_process");
const verify = () => cp.exec("npm ls", error => {
  if (error) {
    console.error("Dependency mismatch between package.json and lock. Run: npm install");
    throw error;
  }
  console.log("Dependencies verified =)");
});

verify();

What do “idealTree” and “reify” mean in the context of npm?
An idealTree is the tree of package data that we intend to install. actualTree is the representation of the actual packages on disk.

During lockfile validation, npm compares the inventory of package items in the tree that is about to be installed (idealTree) with the inventory of items stored in the package-lock file (virtualTree).

During reification, the idealTree is diffed against the actual tree, and then the nodes from the ideal tree are extracted onto disk. At the end of reify(), the ideal tree is copied to actualTree, since then it reflects the actual state of the node_modules folder.

dependencies, devDependencies and peerDependencies

Dependencies are required at runtime, like a library that provides functions that you call from your code. If you are deploying your application, dependencies has to be installed, or your app will not work. They are installed transitively (if A depends on B depends on C, npm install on A will install B and C). Example: lodash, your project calls some lodash functions.

devDependencies are dependencies you only need during development, like compilers that take your code and compile it into javascript, test frameworks or documentation generators. They are not installed transitively (if A depends on B dev-depends on C, npm install on A will install B only). Example: grunt, your project uses grunt to build itself.

The npm install command will install both devDependencies and dependencies. With the --production flag or when the NODE_ENV environment variable is set to production NODE_ENV=production npm install, npm will not install modules listed in devDependencies.
Many applications use different configuration settings when NODE_ENV is set to production. This also makes the Node.js process more efficient. If you set NODE_ENV=testing which means the devDependencies will be installed and it is more like development than it is like production.
Using the npm uninstall --no-save will tell npm not to remove the package from your package.json or package-lock.json files.

peerDependencies are dependencies that your project hooks into, or modifies, in the parent project, usually a plugin for some other library. It is just intended to be a check, making sure that the project that will depend on your project has a dependency on the project you hook into. So if you make a plugin C that adds functionality to library B, then someone making a project A will need to have a dependency on B if they have a dependency on C. Example: your project adds functionality to grunt and can only be used on projects that use grunt.

In npm versions 3 through 6, peerDependencies were not automatically installed, and would raise a warning if an invalid version of the peer dependency was found in the tree. As of npm v7, peerDependencies are installed by default. (npm has a shortcut where it automatically install mandatory peer dependencies even if the parent package does not depend on them.) If your dependency contains some peerDependencies that conflict with the root project’s dependency, run npm install --legacy-peer-deps to skips strict peer dependency checks, allowing installation of packages with unmet peer dependencies to avoid errors. (--force flag will ignore and override any dependency conflicts, forcing the installation of packages.)

optionalDependencies are dependencies that are not essential for the primary functionality of a package but are beneficial for providing additional features. Let’s say you have a dependency that may be used, but you would like the package manager to proceed if it cannot be found or fails to install. In that case, you can add those dependencies in the optionalDependencies object. A good use case for optionalDependencies is if you have a dependency that won’t necessarily work on every machine. But you should have a fallback plan in case the installation fails.

@npmcli/arborist is the library that calculates dependency trees and manages the node_modules folder hierarchy for the npm command line interface. It’s used in some tools like npm-why to help identify why a package has been installed.

Arborist - the npm tree doctor: npx @npmcli/arborist --help

URLs as dependencies

See details at https://docs.npmjs.com/cli/v8/configuring-npm/package-json#urls-as-dependencies

Git URLs as dependencies
- git+ssh://git@github.com:myaccount/myprivate.git
- git+ssh://git@github.com:myaccount/myprivate.git#develop
- git+https://[username]:[password]@github.com/myaccount/myprivate.git
GitHub URLs: refer to GitHub urls as "foo": "user/foo-project"
Local Paths: You can provide a path to a local directory that contains a package "bar": "file:../foo/bar"

You can configure npm to resolve your dependencies across multiple registries.

# .npmrc

# Fetch `@lihbr` packages from GitHub registry
@lihbr:registry=https://npm.pkg.github.com

# Fetch `@my-company` packages from My Company registry
@my-company:registry=https://npm.pkg.my-company.com

fix broken node modules instantly

patch-package lets app authors instantly make and keep fixes to npm dependencies. Patches created are automatically and gracefully applied when you use npm or yarn.

# fix a bug in one of your dependencies
vim node_modules/some-package/brokenFile.js

# it will create a folder called `patches` in the root dir of your app. 
# Inside will be a `.patch` file, which is a diff between normal old package and your fixed version
npx patch-package some-package

# commit the patch file to share the fix with your team
git add patches/some-package+3.14.15.patch
git commit -m "fix brokenFile.js in some-package"

// package.json
"scripts": {
  "postinstall": "patch-package"
}

npm and npx

One might install a package locally on a certain project using npm install some-package, then we want to execute that package from the command line. Only globally installed packages can be executed by typing their name only. To fix this, you must type the local path ./node_modules/.bin/some-package.

npx comes bundled with npm version 5.2+. It will check whether the command exists in $PATH or in the local project binaries and then execute it. So if you wish to execute the locally installed package, all you need to do is type npx some-package.

Have you ever run into a situation where you want to try some CLI tool, but it’s annoying to have to install a global just to run it once? npx is great for that. It will automatically install a package with that name from the npm registry and invoke it. When it’s done, the installed package won’t be anywhere in the global, so you won’t have to worry about pollution in the long-term. For example, npx create-react-app my-app will generate a react app boilerplate within the path the command had run in, and ensures that you always use the latest version of the package without having to upgrade each time you’re about to use it. There’s an awesome-npx repo with examples of things that work great with npx.

npm will cache the packages in the directory ~/.npm/_npx. The whole point of npx is that you can run the packages without installing them somewhere permanent. So I wouldn’t use that cache location for anything. I wouldn’t be surprised if cache entries were cleared from time to time. I don’t know what algorithm, if any, npx uses for time-based cache invalidation.

You can find the npm-debug.log file in your .npm directory. To find your .npm directory, use npm config get cache. (It is located in ~/.npm so shared accross nodejs versions that nvm installed.) The default location of the logs directory is a directory named _logs inside the npm cache.

npm init and exec

npm init <initializer> can be used to set up a npm package. initializer in this case is an npm package named create-<initializer>, which will be installed by npm exec. The init command is transformed to a corresponding npm exec operation like npm init foo -> npm exec create-foo. Another example is npm init react-app myapp, which is same as npx create-react-app myapp. If the initializer is omitted (by just calling npm init), init will fall back to legacy init behavior. It will ask you a bunch of questions, and then write a package.json for you. You can also use -y/--yes to skip the questionnaire altogether.

npm create is an alias for npm init. Check more about npm init --help.

npm 7 introduced the new npm exec command which, like npx, provided an easy way to run npm scripts on the fly. If the package is not present in the local project dependencies, npm exec installs the required package and its dependencies to a folder in the npm cache. With the introduction of npm exec, npx had been rewritten to use npm exec under the hood in a backwards compatible way, and the standalone npx package deprecated at that time.

To prevent security and user-experience problems from mistyping package names, npx prompts before installing anything. Suppress this prompt with the -y or --yes option.

npm link

Run npm link from your MyModule directory: this will create a global package {prefix}/node/{version}/lib/node_modules/<package> symlinked to the MyModule directory.
Run npm link MyModule from your MyApp directory: this will create a MyModule folder in node_modules symlinked to the globally-installed package and thus to the real location of MyModule. Note that <package-name> is taken from package.json, not from the directory name.
Now any changes to MyModule will be reflected in MyApp/node_modules/MyModule/. Use npm ls -g --depth=0 --link to list all the globally linked modules.
Run npm unlink --no-save <package> on your project’s directory to remove the local symlink.

publish npm packages

Learn how to create a new npm package and publish the code to npm by the demo Building a business card CLI tool. Once your package is published to npm, you can run npx {your-command} to execute your script whenever you like.

Most popular npm packages: https://socket.dev/npm/category/popular

npm and pnpm

The very first package manager ever released was npm, back in January 2010. In 2020, GitHub acquired npm, so in principle, npm is now under the stewardship of Microsoft. (npm should never be capitalized unless it is being displayed in a location that is customarily all-capitals.)

npm handles the dependencies by splitting the installation process into three phases: Resolving -> Fetching -> Linking. Each phase needs to end for the next one to begin.

pnpm was released in 2017. It is a drop-in replacement for npm, so if you have an npm project, you can use pnpm right away. The main problem the creators of pnpm had with npm was the redundant storage of dependencies that were used across projects. 1) The way npm manages the disc space is not efficient. 2) pnpm doesn’t have the blocking stages of installation - the processes run for each of the packages independently.

Traditionally, npm installed dependencies in a flat node_modules folder. On the other hand, pnpm manages node_modules by using hard linking and symbolic linking to a global on-disk content-addressable store. It results in a nested node_modules folder that stores packages in a global store on your home folder (~/.pnpm-store/). Every version of a dependency is physically stored in that folder only once, constituting a single source of truth. pnpm identifies the files by a hash id (also called “content integrity” or “checksum”) and not by the filename, which means that two same files will have identical hash id and pnpm will determine that there’s no reason for duplication.

pnpm `shamefully-hoist=true` configuration

pnpm organizes node_modules differently from npm, exposing only the dependencies explicitly declared in package.json. Transitive dependencies are installed in node_modules/.pnpm/registry.npmjs.org/, rather than the flat structure for node_modules as npm.

In simple terms, if there is a module A that depends on module B, and module A is depended on in the project’s package.json, module A can access module B, but the project cannot. When shamefully-hoist=true is set, module B will be hoisted, making it accessible in the project.

npm scripts

npm scripts are a set of built-in and custom scripts defined in the package.json file. Their goal is to provide a simple way to execute repetitive tasks.

npm makes all your dependencies’ binaries available in the scripts. So you can access them directly as if they were referenced in your PATH. For example, instead of doing ./node_modules/.bin/eslint ., you can use eslint . as the lint script.
npm run is an alias for npm run-script, meaning you could also use npm run-script lint.
Built-in scripts can be executed using aliases, making the complete command shorter and easier to remember. For example, npm run-script test, npm run test, npm test, and npm t are same to run the test script. npm run-script start, npm run start, and npm start are also same.
Run npm run if you forget what npm scripts are available. This produces a list of scripts, and displays the code that each script runs.
We can use && to run multiple scripts sequentially. If the first script fails, the second script is never executed. Another option is using the library npm-run-all to run multiple npm-scripts in parallel or sequential, which is simplified and cross platform.

concurrently can run multiple commands concurrently. Say you have both backend and frontend folder in the project directroy containing a package.json file:

{
  "scripts": {
    "server": "nodemon backend/server.js",
    "client": "npm run dev --prefix frontend",
    "dev": "concurrently \"npm run server\" \"npm run client\""
  }
}

When a script finishes with a non-zero exit code, it means an error occurred while running the script, and the execution is terminated.
Use npm run <script> --silent to reduce logs and to prevent the script from throwing an error. This can be helpful when you want to run a script that you know may fail, but you don’t want it to throw an error. Maybe in a CI pipeline, you want your whole pipeline to keep running even when the test command fails.

We can create “pre” and “post” scripts for any of our scripts, and npm will automatically run them in order.

{
  "scripts": {
    "prefoo": "echo prefoo",
    "foo": "echo foo",
    "postfoo": "echo postfoo"
  }
}

You can run npm config ls -l to get a list of the configuration parameters, and you can use $npm_config_ prefix (like $npm_config_editor) to access them in the scripts. Any key-value pairs we add to our script will be translated into an environment variable with the npm_config prefix.
```
{
  "scripts": {
    "hello": "echo \"Hello $npm_config_firstname\""
  }
}

// Output: "Hello Paula"
npm run hello --firstname=Paula
```

package.json vars are available via process.env (with npm_package_ prefix) in Node scripts by default.

{
  "name": "foo",
  "version":"1.2.5",
}

// When you run Node.js files via npm scripts
// Output: 'foo', '1.2.5'
console.log(process.env.npm_package_name, process.env.npm_package_version);

Passing arguments to other npm scripts, we can leverage the -- separator. e.g. "pass-flags-to-other-script": "npm run my-script -- --watch" will pass the --watch flag to the my-script command.
One convention that you may have seen is using a prefix and a colon to group scripts, for example build:dev and build:prod. This can be helpful to create groups of scripts that are easier to identify by their prefixes.
shx is a wrapper around ShellJS Unix commands, providing an easy solution for simple Unix-like, cross-platform commands in npm package scripts. ShellJS is a portable (Windows/Linux/macOS) implementation of Unix shell commands on top of the Node.js API. shx is good for writing one-off commands in npm package scripts (e.g. "clean": "shx rm -rf out/"). Run npm install shx --save-dev to install it, and run command in either a Unix or Windows command line.

Despite “npm scripts” high usage they are not particularly well optimized.

By running cat $(which npm), you will find npm CLI is a standard JavaScript file. The only special thing is the first line #!/usr/bin/env node which tells your shell the current file can be executed with node.
Because it’s just a js file, we can rely on all the usual ways to generate a profile. My favorite one is node’s --cpu-prof argument. Combine that knowledge together and we can generate a profile from an npm script via node --cpu-prof $(which npm) run myscript. Loading that profile into speedscope reveals quite a bit about how npm is structured. The majority of time is spent on loading all the modules that compose the npm cli. The time of the script that we’re running pales in comparison.

Kexi's blog

blog/notes/digital garden

Kexi's blog

Congradulations,
you find the Easter egg

Understand npm concepts

package.json and package-lock.json

Benefits of `exports` field

Read How To Create An NPM Package by Total TypeScript

npm install and npm ci

npm ls

dependencies, devDependencies and peerDependencies

URLs as dependencies

fix broken node modules instantly

npm and npx

npm init and exec

npm link

publish npm packages

npm and pnpm

pnpm `shamefully-hoist=true` configuration

npm scripts

Understand npm concepts

package.json and package-lock.json

Benefits of exports field

Read How To Create An NPM Package by Total TypeScript

npm install and npm ci

npm ls

dependencies, devDependencies and peerDependencies

URLs as dependencies

fix broken node modules instantly

npm and npx

npm init and exec

npm link

publish npm packages

npm and pnpm

pnpm shamefully-hoist=true configuration

npm scripts

Benefits of `exports` field

pnpm `shamefully-hoist=true` configuration