Running Yarn offline

Posted Nov 24, 2016 by Konstantin Raev

Repeatable and reliable builds for large JavaScript projects are vital. If your builds depend on dependencies being downloaded from network, this build system is neither repeatable nor reliable.

One of the main advantages of Yarn is that it can install node_modules from files located in file system. We call it “Offline Mirror” because it mirrors the files downloaded from registry during the first build and stores them locally for future builds.

“Offline mirror” is different from cache that both npm CLI and Yarn have. Caches store already unzipped tarballs downloaded from registry, they also can be implementation specific and may be invalid between multiple versions of CLI tools. The tarballs in “Offline mirror” can be consumed by any version Yarn that will build cache based on them. It is also easier to store files when they are compressed.

Let’s set up “Offline mirror” for a simple JS project

{
  "name": "yarn-offline",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "is-array": "^1.0.1",
    "left-pad": "^1.1.3",
    "mime-types": "^2.1.13"
  }
}

When you run yarn install, the generated yarn.lock file has sections for every dependency:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1

is-array@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/is-array/-/is-array-1.0.1.tgz#e9850cc2cc860c3bc0977e84ccf0dd464584279a"

left-pad@^1.1.3:
  version "1.1.3"
  resolved "https://registry.yarnpkg.com/left-pad/-/left-pad-1.1.3.tgz#612f61c033f3a9e08e939f1caebeea41b6f3199a"

mime-db@~1.25.0:
  version "1.25.0"
  resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.25.0.tgz#c18dbd7c73a5dbf6f44a024dc0d165a1e7b1c392"

mime-types@^2.1.13:
  version "2.1.13"
  resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.13.tgz#e07aaa9c6c6b9a7ca3012c69003ad25a39e92a88"
  dependencies:
    mime-db "~1.25.0"

Each of these dependencies have a resolved field with a remote URL. If you delete your node_modules folder and run yarn install again, Yarn will download the same resolved dependencies specified in this lockfile. It will even guarantee that no one modified the files since your first installs by verifying checksum for each of them.

However, if for some reason these urls are unreachable during your build, it will fail. To solve this, we’ll need an “Offline mirror”.

Set up .yarnrc

First we need to setup a directory to be our “Offline mirror” storage, we can do that with yarn config command:

$ yarn config set yarn-offline-mirror ./npm-packages-offline-cache
yarn config v0.23.2
success Set "yarn-offline-mirror" to "./npm-packages-offline-cache".
✨  Done in 0.06s.

./npm-packages-offline-cache is an example location relative to home folder where all the source.tar.gz files will be downloaded to from the registry.

Offline mirror does not come with removing tarballs. In order to keep the cache folder up to date, you need to add the following to the config file:

This feature is only available in version 0.23.0 and above.

$ yarn config set yarn-offline-mirror-pruning true
yarn config v0.23.2
success Set "yarn-offline-mirror-pruning" to "true".
✨  Done in 0.06s.

This will create a .yarnrc file in your HOME directory. Let’s move this file to the project root so that offline mirror would be used only for this project.

$ mv ~/.yarnrc ./

(In the unlikely event that yarn config didn’t update the file in your home folder, e.g. if you are running yarn config as root inside a Docker container, the file being updated might be a different one. Use yarn config list --verbose to locate the proper file.)

Initialize the new lockfile

Remove the node_modules that got generated previously and run yarn install again:

$ rm -rf node_modules/ yarn.lock
$ yarn install
yarn install v0.17.8
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 0.57s.

The dependency resolutions in your yarn.lock should look the same as the original (you shouldn’t have any diff):

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1

is-array@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/is-array/-/is-array-1.0.1.tgz#e9850cc2cc860c3bc0977e84ccf0dd464584279a"

left-pad@^1.1.3:
  version "1.1.3"
  resolved "https://registry.yarnpkg.com/left-pad/-/left-pad-1.1.3.tgz#612f61c033f3a9e08e939f1caebeea41b6f3199a"

mime-db@~1.25.0:
  version "1.25.0"
  resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.25.0.tgz#c18dbd7c73a5dbf6f44a024dc0d165a1e7b1c392"

mime-types@^2.1.13:
  version "2.1.13"
  resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.13.tgz#e07aaa9c6c6b9a7ca3012c69003ad25a39e92a88"
  dependencies:
    mime-db "~1.25.0"

The offline cache file will be stored in the npm-packages-offline-cache folder that was configured earlier. Each resolved dependency also contains a checksum after the file name to ensure that no one mangles with the downloaded files.

If we look inside the “Offline mirror” folder we’ll see the .tgz files that yarn will use for the following builds without reaching out to network:

$ ls npm-packages-offline-cache/
is-array-1.0.1.tgz    left-pad-1.1.3.tgz    mime-db-1.25.0.tgz    mime-types-2.1.13.tgz

How can you test to make sure it is offline?

Clear your global cache with “yarn cache clean”
Turn off wifi
Run “yarn install –offline”. The offline flag will make sure yarn does not reach out to the network

In a nutshell, to enable “Offline mirror” for your project you need:

add “yarn-offline-mirror” configuration to .yarnrc file
generate a new yarn.lock with “yarn install” command

A few tips and tricks

Updating your package

If you want to make sure you have a clean cached modules, here are few of the steps you can take:

Remove the package first. Make sure you have “yarn-offline-mirror-pruning” set to true in your .yarnrc file
Clear the yarn cache with “yarn cache clean” before adding the updated version of the package
Add your package

The “yarn-offline-mirror-pruning” will help clean up any unlinked dependencies. When you add the updated package, it will check the yarn cache first and pull any missing dependencies from there. This will prevent yarn adding new tarball back with the updated package. You want to make sure the yarn cache is all clean before you do any adding for cache module.

You can check in “Offline mirror” into git or mercurial repository

The “Offline Mirror” can be shared between build servers or development machines in any way that is convenient: a Box / Dropbox folder, stored in source control or on a network drive. At Facebook the offline mirror lives inside of our big Mercurial “monorepo”.

Whether to commit binary files into a repository or not depends on the number and size of your project’s dependencies. For example, out of 849 React Native dependencies totaling 23MB, only 10% are larger than 30KB. list of React Native .tgz files sorted by size descending Size of all .tgz files used in React Native

Many Facebook teams, including the React Native team, decided to check in their “Offline mirror”. They all share the same “Offline mirror” which means that most dependencies for new projects are often already checked into that folder, so the cost of storing the packages in source control gets lower the more projects use it.

Let’s compare checking in node_modules to checking in “Offline mirror”

The React Native team used to check in the node_modules folder but they hit several limits:

node_modules contains more than 37,000 files (and more than 100,000 files back when we were using npm2). This had a bad performance impact on our Mercurial repository.
Reviewing Pull Requests that changed a dependency was quite hard as all the files in node_modules that were added and removed created a ton of noise, making code reviews unpleasant

In comparison, updating a third-party dependency with the Offline Mirror adds just a few files that are very easy to review:

$ yarn add shelljs@0.7.0 --dev
yarn add v0.23.2
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
success Saved 4 new dependencies.
├─ interpret@1.0.1
├─ rechoir@0.6.2
└─ shelljs@0.7.0
│  └─ glob@7.1.1
✨  Done in 8.15s.

$ git diff
diff --git a/package.json b/package.json
index 4619f16..7acb42f 100644
--- a/package.json
+++ b/package.json
@@ -220,7 +220,7 @@
     "mock-fs": "^3.11.0",
     "portfinder": "0.4.0",
     "react": "~15.3.1",
-    "shelljs": "0.6.0",
+    "shelljs": "0.7.0",
     "sinon": "^2.0.0-pre.2"
   }
 }
diff --git a/yarn.lock b/yarn.lock
index 11ce116..f5d81ba 100644
--- a/yarn.lock
+++ b/yarn.lock
...
-shelljs@0.6.0:
-  version "0.6.0"
-  resolved https://registry.yarnpkg.com/shelljs/-/shelljs-0.6.0.tgz#ce1ed837b4b0e55b5ec3dab84251ab9dbdc0c7ec
+shelljs@0.7.0:
+  version "0.7.0"
+  resolved https://registry.yarnpkg.com/shelljs/-/shelljs-0.7.0.tgz#3f6f2e4965cec565f65ff3861d644f879281a576
+  dependencies:
+    glob "^7.0.0"
+    interpret "^1.0.0"
+    rechoir "^0.6.2"

 shellwords@^0.1.0:
   version "0.1.0"

$ git status
On branch testing-yarn
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   package.json
    modified:   yarn.lock

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    yarn-offline-mirror/interpret-1.0.1.tgz
    yarn-offline-mirror/rechoir-0.6.2.tgz
    yarn-offline-mirror/shelljs-0.7.0.tgz

no changes added to commit (use "git add" and/or "git commit -a")

Did you know that Yarn is also distributed as a single bundle JS file in releases that can be used on CI systems without internet access?

Files distributed with Yarn releases **yarn-.js** (for Node 5+) and **yarn-legacy-.js** (for Node 4) can be used stand-alone in CI systems without a need to install it.

Just check it into your project’s repository and use it in the build script:

node ./yarn-0.23.2.js install

This is quite convenient for teams that use multiple operating systems and want to have atomic updates for Yarn.

This is going to get better

The “Offline mirror” was implemented early in Yarn’s development cycle and we are working on improving it in a backwards compatible way:

The resolved field is used both for offline mirror paths and registry URIs. This means that the yarn.lock file that React Native team uses internally can’t be shared with the open source community because the React Native team does not sync the offline mirror with the open source version of React Native. Issue.
There is an improved workflow being considered for future versions of Yarn. It is not drastically different but some settings and lock files may change.