Down the NPM Rabbit Hole
Author’s note: this post is a collaborative effort by Grady Johnson and Mustafa Haddara.
The following is a story of trials, tribulations, and triumphs, and stands as a testament to two young men’s ability to say “package” over a hundred times without giggling.
At Vena, we have an internal tool for running API tests against our application servers. This tool was written in JavaScript, back in 2015 and 2016, on Node version 0.12. This tool worked, and, as things go, no one was maintaining the tool because it kept working. Devs spent their time working on features for the product, instead of keeping this tool up to date with the latest versions of node.
Earlier this week, Grady attempted to install this package and EVERYTHING BROKE.
On Mustafa’s machine:
$ node --version
V0.12.10
$ npm --version
2.14.9
$ vena_tool --version
1.2.17
BUT ON GRADY’S MACHINE:
$ node --version
V0.12.10
$ npm --version
2.14.9
$ vena_tool --version
SyntaxError: Use of const in strict mode.
“Alright,” we thought, “time for the nuclear option.” We reinstalled everything from scratch.
Let’s try that again.
$ node --version
V0.12.10
Yep.
$ npm --version
2.14.9
Story checks out.
$ vena_tool --version
SyntaxError: Use of const in strict mode.
Go home Node, you’re drunk.
Now here’s where things get interesting. Notice this error message:
/root/vena_tool/node_modules/unirest/node_modules/request/node_modules/tough-cookie/node_modules/ip-regex/index.js:3
One of our dependencies, unirest, was having problems, so we checked the repo. The latest commit was almost two years ago…
H’okay. So we checked its dependency, request. Still no recent changes.
Descending further into the madness, we looked at its dependency, tough-cookie. Lots of recent changes.
Looking at its dependencies, we see that one of those dependencies ip-regex, recently stopped supporting versions of Node lower than 8.0.
Now, the current stable version of Node is 11.7.0.
We, however, are tied to Node version 0.12.10.
In mathematical terms, that would mean we were behind by two orders of magnitude.
So, to recap:
Our tool requires unirest at version 0.4.0.
unirest requires request at version ~2.51.0.
request requires tough-cookie at version >=0.12.0
A few weeks ago, tough-cookie released v.3.0.0. As part of this release, they began using the ip-regex package.
In the ip-regex README, they say:
This module targets Node.js 8 or later…
Now normally when adding a dependency you would use the ~ operator, which means use “approximately” this version. That’s just good manners.
The reason for this is simple:
In version numbering, standard practice is to mark incremental patches by changing the trailing digits (eg. 2.5.1 becomes 2.5.2).
More substantial changes (known as “minor” changes) are generally reflected in the second set of digits (eg. 2.5.2 becomes 2.6.0).
Finally, major changes to the API (also known as “breaking” changes) are indicated by the first set of digits (eg. 2.6.0 becomes 3.0.0).
Systems like this are what separate us from the animals. They are civilization. Without them we are just… monsters.
So when you see something like this:
… it tends to angry up the blood.
In human speak, “>=0.12.0” means “use any version of tough-cookie 0.12.0 or greater.”
Well, version 3.0.0 is definitely greater than 0.12.0.
What’s not particularly great is that because this happens three layers deep in our dependencies, fixing it is not a simple matter of specifying exactly the version we want. Because we were already doing that.
Our initial approach to solving this was to take the node_modules folder off of Mustafa’s computer, put it on Grady’s computer, and Grady’s problem was solved. He could continue to use our internal tool, no problem.
But… that left this problem open for anyone who would try to install this tool in the future. So our “future proof” solution was simply to commit the entire contents of node_modules into the git repo for this tool. The problem with that is, well, node_modules contains all of the JavaScript libraries used by our tool, and, well, there are a lot of JavaScript libraries:
This is because node_modules contains all of our dependencies’ source code. Which includes their node_modules folder. Which contains all of their dependencies. Essentially, we’re capturing all of the JavaScript libraries we would ever need to run our tool.
Our node_modules folder was HUGE. This is what the initial pull request looked like:
Yup, you read that right. 1.5 MILLION lines of code changed, and almost 11 THOUSAND files added.
And a single line removed, in the first commit:
This PR got a few… choice… reactions from our fellow coworkers.
Yep, it’s absolute megaton of a PR.
But then, other comments came in that pointed us to the right spot.
Of course, we have to be careful:
As it turns out, the solution is to use a little piece of genius called npm shrinkwrap to create a “publishable lockfile” (which it stores at npm-shrinkwrap.json). This lockfile “describes an exact, and more importantly reproducible” dependency tree, which means that future installations will reproduce the exact same module structure whenever possible.
This means that we don’t need to commit the ENTIRE node_modules folder, because npm-shrinkwrap.json describes that folder structure exactly. When someone does an npm install, npm will use the exact version numbers in npm-shrinkwrap.json instead of doing the default version resolution, so we can specify that we need to download, say, version 2.3.3 of tough-cookie instead of npm fetching version 3.0.0.
To the author of this beautiful little tool, we would like to buy you a drink. Until then, we’ll just be here googling .js.
