git mv src/content/* src/content/en/

1 files changed, 147 insertions, 0 deletions

diff --git a/src/content/en/blog/2018/08/01/npm-ci-reproducibility.adoc b/src/content/en/blog/2018/08/01/npm-ci-reproducibility.adoc
new file mode 100644
index 0000000..76bd8e6
--- /dev/null
+++ b/src/content/en/blog/2018/08/01/npm-ci-reproducibility.adoc
@@ -0,0 +1,147 @@
+= Verifying "npm ci" reproducibility
+:updatedat: 2019-05-22
+
+:empty:
+:npm-5: https://blog.npmjs.org/post/161081169345/v500
+:package-locks-old: https://docs.npmjs.com/files/package-locks
+:package-lock: https://docs.npmjs.com/files/package-lock.json
+:add-npm-ci: https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable
+:cli-docs: https://docs.npmjs.com/cli/install#description
+:tricky-issue: https://github.com/npm/npm/issues/17979#issuecomment-332701215
+
+When {npm-5}[npm@5] came bringing {package-locks-old}[package-locks] with it, I
+was confused about the benefits it provided, since running `npm install` more
+than once could resolve all the dependencies again and yield yet another fresh
+`package-lock.json` file.  The message saying "you should add this file to
+version control" left me hesitant on what to
+do{empty}footnote:package-lock-message[
+  {cli-docs}[documentation] claims `npm install` is driven by the existing
+  `package-lock.json`, but that's actually {tricky-issue}[a little bit tricky].
+].
+
+However the {add-npm-ci}[addition of `npm ci`] filled this gap: it's a stricter
+variation of `npm install` which guarantees that "{package-lock}[subsequent
+installs are able to generate identical trees]".  But are they really identical?
+I could see that I didn't have the same problems of different installation
+outputs, but I didn't know for *sure* if it was really identical.
+
+== Computing the hash of a directory's content
+
+:merkle-tree: https://en.wikipedia.org/wiki/Merkle_tree
+
+I quickly searched for a way to check for the hash signature of an entire
+directory tree, but I couldn't find one.  I've made a poor man's
+{merkle-tree}[Merkle tree] implementation using `sha256sum` and a few piped
+commands at the terminal:
+
+[source,sh]
+----
+merkle-tree () {
+	dirname="${1-.}"
+	pushd "$dirname"
+	find . -type f |
+		sort                      |
+		xargs -I{} sha256sum "{}" |
+		sha256sum                 |
+		awk '{print $1}'
+	popd
+}
+----
+
+Going through it line by line:
+
+* #1 we define a Bash function called `merkle-tree`;
+* #2 it accepts a single argument: the directory to compute the merkle tree from
+  If nothing is given, it runs on the current directory (`.`);
+* #3 we go to the directory, so we don't get different prefixes in `find`'s
+  output (like `../a/b`);
+* #4 we get all files from the directory tree.  Since we're using `sha256sum` to
+  compute the hash of the file contents, we need to filter out folders from it;
+* #5 we need to sort the output, since different file systems and `find`
+  implementations may return files in different orders;
+* #6 we use `xargs` to compute the hash of each file individually through
+  `sha256sum`.  Since a file may contain spaces we need to escape it with
+  quotes;
+* #7 we compute the hash of the combined hashes.  Since `sha256sum` output is
+  formatted like `<hash> <filename>`, it produces a different final hash if a
+  file ever changes name without changing it's content;
+* #8 we get the final hash output, excluding the `<filename>` (which is `-` in
+  this case, aka `stdin`).
+
+=== Positive points:
+
+. ignore timestamp: running more than once on different installation yields the
+  same hash;
+. the name of the file is included in the final hash computation.
+
+=== Limitations:
+
+. it ignores empty folders from the hash computation;
+. the implementation's only goal is to represent using a digest whether the
+  content of a given directory is the same or not.  Leaf presence checking is
+  obviously missing from it.
+
+=== Testing locally with sample data
+
+[source,sh]
+----
+mkdir /tmp/merkle-tree-test/
+cd /tmp/merkle-tree-test/
+mkdir -p a/b/ a/c/ d/
+echo "one"   > a/b/one.txt
+echo "two"   > a/c/two.txt
+echo "three" > d/three.txt
+merkle-tree . # output is       be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
+merkle-tree . # output still is be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
+echo "four"  > d/four.txt
+merkle-tree . # output is now   b5464b958969ed81815641ace96b33f7fd52c20db71a7fccc45a36b3a2ae4d4c
+rm d/four.txt
+merkle-tree . # output back to  be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
+echo "hidden-five" > a/b/one.txt
+merkle-tree . # output changed  471fae0d074947e4955e9ac53e95b56e4bc08d263d89d82003fb58a0ffba66f5
+----
+
+It seems to work for this simple test case.
+
+You can try copying and pasting it to verify the hash signatures.
+
+== Using `merkle-tree` to check the output of `npm ci`
+
+_I've done all of the following using Node.js v8.11.3 and npm@6.1.0_.
+
+In this test case I'll take the main repo of
+https://lernajs.io/[Lerna]footnote:lerna-package-lock[
+  Finding a big known repo that actually committed the `package-lock.json` file
+  was harder than I expected.
+]:
+
+```bash
+cd /tmp/
+git clone https://github.com/lerna/lerna.git
+cd lerna/
+git checkout 57ff865c0839df75dbe1974971d7310f235e1109
+npm ci
+merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
+rm -rf node_modules/
+npm ci
+merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
+npm ci      # test if it also works with an existing node_modules/ folder
+merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
+```
+
+Good job `npm ci` :)
+
+#6 and #9 take some time to run (21 seconds in my machine), but this specific
+use case isn't performance sensitive.  The slowest step is computing the hash of
+each individual file.
+
+== Conclusion
+
+`npm ci` really "generates identical trees".
+
+I'm not aware of any other existing solution for verifying the hash signature of
+a directory.  If you know any, shoot me an email, as I'd like to know it.
+
+== *Edit*
+
+2019-05-22: Fix spelling.