summaryrefslogtreecommitdiff
path: root/src/content/blog/2018/08/01
diff options
context:
space:
mode:
authorEuAndreh <eu@euandre.org>2025-04-18 02:17:12 -0300
committerEuAndreh <eu@euandre.org>2025-04-18 02:48:42 -0300
commit020c1e77489b772f854bb3288b9c8d2818a6bf9d (patch)
tree142aec725a52162a446ea7d947cb4347c9d573c9 /src/content/blog/2018/08/01
parentMakefile: Remove security.txt.gz (diff)
downloadeuandre.org-020c1e77489b772f854bb3288b9c8d2818a6bf9d.tar.gz
euandre.org-020c1e77489b772f854bb3288b9c8d2818a6bf9d.tar.xz
git mv src/content/* src/content/en/
Diffstat (limited to 'src/content/blog/2018/08/01')
-rw-r--r--src/content/blog/2018/08/01/npm-ci-reproducibility.adoc147
1 files changed, 0 insertions, 147 deletions
diff --git a/src/content/blog/2018/08/01/npm-ci-reproducibility.adoc b/src/content/blog/2018/08/01/npm-ci-reproducibility.adoc
deleted file mode 100644
index 76bd8e6..0000000
--- a/src/content/blog/2018/08/01/npm-ci-reproducibility.adoc
+++ /dev/null
@@ -1,147 +0,0 @@
-= Verifying "npm ci" reproducibility
-:updatedat: 2019-05-22
-
-:empty:
-:npm-5: https://blog.npmjs.org/post/161081169345/v500
-:package-locks-old: https://docs.npmjs.com/files/package-locks
-:package-lock: https://docs.npmjs.com/files/package-lock.json
-:add-npm-ci: https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable
-:cli-docs: https://docs.npmjs.com/cli/install#description
-:tricky-issue: https://github.com/npm/npm/issues/17979#issuecomment-332701215
-
-When {npm-5}[npm@5] came bringing {package-locks-old}[package-locks] with it, I
-was confused about the benefits it provided, since running `npm install` more
-than once could resolve all the dependencies again and yield yet another fresh
-`package-lock.json` file. The message saying "you should add this file to
-version control" left me hesitant on what to
-do{empty}footnote:package-lock-message[
- {cli-docs}[documentation] claims `npm install` is driven by the existing
- `package-lock.json`, but that's actually {tricky-issue}[a little bit tricky].
-].
-
-However the {add-npm-ci}[addition of `npm ci`] filled this gap: it's a stricter
-variation of `npm install` which guarantees that "{package-lock}[subsequent
-installs are able to generate identical trees]". But are they really identical?
-I could see that I didn't have the same problems of different installation
-outputs, but I didn't know for *sure* if it was really identical.
-
-== Computing the hash of a directory's content
-
-:merkle-tree: https://en.wikipedia.org/wiki/Merkle_tree
-
-I quickly searched for a way to check for the hash signature of an entire
-directory tree, but I couldn't find one. I've made a poor man's
-{merkle-tree}[Merkle tree] implementation using `sha256sum` and a few piped
-commands at the terminal:
-
-[source,sh]
-----
-merkle-tree () {
- dirname="${1-.}"
- pushd "$dirname"
- find . -type f |
- sort |
- xargs -I{} sha256sum "{}" |
- sha256sum |
- awk '{print $1}'
- popd
-}
-----
-
-Going through it line by line:
-
-* #1 we define a Bash function called `merkle-tree`;
-* #2 it accepts a single argument: the directory to compute the merkle tree from
- If nothing is given, it runs on the current directory (`.`);
-* #3 we go to the directory, so we don't get different prefixes in `find`'s
- output (like `../a/b`);
-* #4 we get all files from the directory tree. Since we're using `sha256sum` to
- compute the hash of the file contents, we need to filter out folders from it;
-* #5 we need to sort the output, since different file systems and `find`
- implementations may return files in different orders;
-* #6 we use `xargs` to compute the hash of each file individually through
- `sha256sum`. Since a file may contain spaces we need to escape it with
- quotes;
-* #7 we compute the hash of the combined hashes. Since `sha256sum` output is
- formatted like `<hash> <filename>`, it produces a different final hash if a
- file ever changes name without changing it's content;
-* #8 we get the final hash output, excluding the `<filename>` (which is `-` in
- this case, aka `stdin`).
-
-=== Positive points:
-
-. ignore timestamp: running more than once on different installation yields the
- same hash;
-. the name of the file is included in the final hash computation.
-
-=== Limitations:
-
-. it ignores empty folders from the hash computation;
-. the implementation's only goal is to represent using a digest whether the
- content of a given directory is the same or not. Leaf presence checking is
- obviously missing from it.
-
-=== Testing locally with sample data
-
-[source,sh]
-----
-mkdir /tmp/merkle-tree-test/
-cd /tmp/merkle-tree-test/
-mkdir -p a/b/ a/c/ d/
-echo "one" > a/b/one.txt
-echo "two" > a/c/two.txt
-echo "three" > d/three.txt
-merkle-tree . # output is be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
-merkle-tree . # output still is be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
-echo "four" > d/four.txt
-merkle-tree . # output is now b5464b958969ed81815641ace96b33f7fd52c20db71a7fccc45a36b3a2ae4d4c
-rm d/four.txt
-merkle-tree . # output back to be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
-echo "hidden-five" > a/b/one.txt
-merkle-tree . # output changed 471fae0d074947e4955e9ac53e95b56e4bc08d263d89d82003fb58a0ffba66f5
-----
-
-It seems to work for this simple test case.
-
-You can try copying and pasting it to verify the hash signatures.
-
-== Using `merkle-tree` to check the output of `npm ci`
-
-_I've done all of the following using Node.js v8.11.3 and npm@6.1.0_.
-
-In this test case I'll take the main repo of
-https://lernajs.io/[Lerna]footnote:lerna-package-lock[
- Finding a big known repo that actually committed the `package-lock.json` file
- was harder than I expected.
-]:
-
-```bash
-cd /tmp/
-git clone https://github.com/lerna/lerna.git
-cd lerna/
-git checkout 57ff865c0839df75dbe1974971d7310f235e1109
-npm ci
-merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
-rm -rf node_modules/
-npm ci
-merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
-npm ci # test if it also works with an existing node_modules/ folder
-merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
-```
-
-Good job `npm ci` :)
-
-#6 and #9 take some time to run (21 seconds in my machine), but this specific
-use case isn't performance sensitive. The slowest step is computing the hash of
-each individual file.
-
-== Conclusion
-
-`npm ci` really "generates identical trees".
-
-I'm not aware of any other existing solution for verifying the hash signature of
-a directory. If you know any, shoot me an email, as I'd like to know it.
-
-== *Edit*
-
-2019-05-22: Fix spelling.