aboutsummaryrefslogtreecommitdiff
path: root/_posts/2018-08-01-verifying-npm-ci-reproducibility.md
diff options
context:
space:
mode:
authorEuAndreh <eu@euandre.org>2020-02-06 00:19:36 -0300
committerEuAndreh <eu@euandre.org>2020-02-06 00:19:36 -0300
commitc097bdb3ba09b0292d944fcc518a1cad6ab03ff2 (patch)
treec7a105ac4315162bd1390a04a8b9a74df05a2cf8 /_posts/2018-08-01-verifying-npm-ci-reproducibility.md
parentUse custom date format for pt (diff)
downloadeuandre.org-c097bdb3ba09b0292d944fcc518a1cad6ab03ff2.tar.gz
euandre.org-c097bdb3ba09b0292d944fcc518a1cad6ab03ff2.tar.xz
Sanitize markdown content
Diffstat (limited to '')
-rw-r--r--_posts/2018-08-01-verifying-npm-ci-reproducibility.md104
1 files changed, 47 insertions, 57 deletions
diff --git a/_posts/2018-08-01-verifying-npm-ci-reproducibility.md b/_posts/2018-08-01-verifying-npm-ci-reproducibility.md
index dcfdd75..f1fd1dd 100644
--- a/_posts/2018-08-01-verifying-npm-ci-reproducibility.md
+++ b/_posts/2018-08-01-verifying-npm-ci-reproducibility.md
@@ -6,33 +6,29 @@ lang: en
ref: veryfing-npm-ci-reproducibility
updated_at: 2019-05-22
---
-When
-[npm\@5](https://blog.npmjs.org/post/161081169345/v500) came bringing
+When [npm@5](https://blog.npmjs.org/post/161081169345/v500) came bringing
[package-locks](https://docs.npmjs.com/files/package-locks) with it, I
was confused about the benefits it provided, since running `npm install`
more than once could resolve all the dependencies again and yield yet
-another fresh `package-lock.json` file. The message saying \"you should
-add this file to version control\" left me hesitant on what to do[^1].
-
-However the [addition of
-`npm ci`](https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable)
-filled this gap: it\'s a stricter variation of `npm install` which
-guarantees that \"[subsequent installs are able to generate identical
-trees](https://docs.npmjs.com/files/package-lock.json)\". But are they
-really identical? I could see that I didn\'t have the same problems of
-different installation outputs, but I didn\'t know for **sure** if it
+another fresh `package-lock.json` file. The message saying "you should
+add this file to version control" left me hesitant on what to do[^1].
+
+However the [addition of `npm ci`](https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable)
+filled this gap: it's a stricter variation of `npm install` which
+guarantees that "[subsequent installs are able to generate identical trees](https://docs.npmjs.com/files/package-lock.json)". But are they
+really identical? I could see that I didn't have the same problems of
+different installation outputs, but I didn't know for **sure** if it
was really identical.
-Computing the hash of a directory\'s content
---------------------------------------------
+## Computing the hash of a directory's content
I quickly searched for a way to check for the hash signature of an
-entire directory tree, but I couldn\'t find one. I\'ve made a poor
-man\'s [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree)
+entire directory tree, but I couldn't find one. I've made a poor
+man's [Merkle tree](https://en.wikipedia.org/wiki/Merkle_tree)
implementation using `sha256sum` and a few piped commands at the
terminal:
-``` {.bash .numberLines startFrom=""}
+```bash
merkle-tree () {
dirname="${1-.}"
pushd "$dirname"
@@ -47,25 +43,25 @@ merkle-tree () {
Going through it line by line:
-- \#1 we define a Bash function called `merkle-tree`;
-- \#2 it accepts a single argument: the directory to compute the
- merkle tree from. If nothing is given, it runs on the current
- directory (`.`);
-- \#3 we go to the directory, so we don\'t get different prefixes in
- `find`\'s output (like `../a/b`);
-- \#4 we get all files from the directory tree. Since we\'re using
- `sha256sum` to compute the hash of the file contents, we need to
- filter out folders from it;
-- \#5 we need to sort the output, since different file systems and
- `find` implementations may return files in different orders;
-- \#6 we use `xargs` to compute the hash of each file individually
- through `sha256sum`. Since a file may contain spaces we need to
- escape it with quotes;
-- \#7 we compute the hash of the combined hashes. Since `sha256sum`
- output is formatted like `<hash> <filename>`, it produces a
- different final hash if a file ever changes name without changing
- it\'s content;
-- \#8 we get the final hash output, excluding the `<filename>` (which
+- #1 we define a Bash function called `merkle-tree`;
+- #2 it accepts a single argument: the directory to compute the
+ merkle tree from. If nothing is given, it runs on the current
+ directory (`.`);
+- #3 we go to the directory, so we don't get different prefixes in
+ `find`'s output (like `../a/b`);
+- #4 we get all files from the directory tree. Since we're using
+ `sha256sum` to compute the hash of the file contents, we need to
+ filter out folders from it;
+- #5 we need to sort the output, since different file systems and
+ `find` implementations may return files in different orders;
+- #6 we use `xargs` to compute the hash of each file individually
+ through `sha256sum`. Since a file may contain spaces we need to
+ escape it with quotes;
+- #7 we compute the hash of the combined hashes. Since `sha256sum`
+ output is formatted like `<hash> <filename>`, it produces a
+ different final hash if a file ever changes name without changing
+ it's content;
+- #8 we get the final hash output, excluding the `<filename>` (which
is `-` in this case, aka `stdin`).
### Positive points:
@@ -77,13 +73,13 @@ Going through it line by line:
### Limitations:
1. it ignores empty folders from the hash computation;
-2. the implementation\'s only goal is to represent using a digest
+2. the implementation's only goal is to represent using a digest
whether the content of a given directory is the same or not. Leaf
presence checking is obviously missing from it.
### Testing locally with sample data
-``` {.bash .numberLines startFrom=""}
+```bash
mkdir /tmp/merkle-tree-test/
cd /tmp/merkle-tree-test/
mkdir -p a/b/ a/c/ d/
@@ -104,15 +100,14 @@ It seems to work for this simple test case.
You can try copying and pasting it to verify the hash signatures.
-Using `merkle-tree` to check the output of `npm ci`
----------------------------------------------------
+## Using `merkle-tree` to check the output of `npm ci`
-*I\'ve done all of the following using Node.js v8.11.3 and npm\@6.1.0.*
+*I've done all of the following using Node.js v8.11.3 and npm@6.1.0.*
-In this test case I\'ll take the main repo of
+In this test case I'll take the main repo of
[Lerna](https://lernajs.io/)[^2]:
-``` {.bash .numberLines startFrom=""}
+```bash
cd /tmp/
git clone https://github.com/lerna/lerna.git
cd lerna/
@@ -128,29 +123,24 @@ merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c9982116
Good job `npm ci` :)
-\#6 and \#9 take some time to run (21 seconds in my machine), but this
-specific use case isn\'t performance sensitive. The slowest step is
+#6 and #9 take some time to run (21 seconds in my machine), but this
+specific use case isn't performance sensitive. The slowest step is
computing the hash of each individual file.
-Conclusion
-----------
+## Conclusion
-`npm ci` really \"generates identical trees\".
+`npm ci` really "generates identical trees".
-I\'m not aware of any other existing solution for verifying the hash
-signature of a directory. If you know any I\'d [like to
-know](mailto:eu@euandre.org).
+I'm not aware of any other existing solution for verifying the hash
+signature of a directory. If you know any I'd [like to know](mailto:eu@euandre.org).
-*Edit*
-------
+## *Edit*
2019/05/22: Fix spelling.
-[^1]: The
- [documentation](https://docs.npmjs.com/cli/install#description)
+[^1]: The [documentation](https://docs.npmjs.com/cli/install#description)
claims `npm install` is driven by the existing `package-lock.json`,
- but that\' actually [a little bit
- tricky](https://github.com/npm/npm/issues/17979#issuecomment-332701215).
+ but that's actually [a little bit tricky](https://github.com/npm/npm/issues/17979#issuecomment-332701215).
[^2]: Finding a big known repo that actually committed the
`package-lock.json` file was harder than I expected.