diff options
Diffstat (limited to '')
-rw-r--r-- | site/posts/2018-08-01-verifying-npm-ci-reproducibility.org | 12 |
1 files changed, 7 insertions, 5 deletions
diff --git a/site/posts/2018-08-01-verifying-npm-ci-reproducibility.org b/site/posts/2018-08-01-verifying-npm-ci-reproducibility.org index 4c01a62..da3947d 100644 --- a/site/posts/2018-08-01-verifying-npm-ci-reproducibility.org +++ b/site/posts/2018-08-01-verifying-npm-ci-reproducibility.org @@ -4,9 +4,9 @@ date: 2018-08-01 --- When [[https://blog.npmjs.org/post/161081169345/v500][npm@5]] came bringing [[https://docs.npmjs.com/files/package-locks][package-locks]] with it, I was confused about the benefits it provided, since running =npm install= more than once could resolve all the dependencies again and yield yet another fresh =package-lock.json= file. The message saying "you should add this file to version control" left me hesitant on what to do[fn:npm-install]. -However the [[https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable][addition of =npm ci=]] filled this gapped: it's a stricter variation of =npm install= which guarantees that "[[https://docs.npmjs.com/files/package-lock.json][subsequent installs are able to generate identical trees]]". But are they really identical? I could see that I didn't have the same problems of different installation outputs, but I didn't know for *sure* if it was really identical. +However the [[https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable][addition of =npm ci=]] filled this gap: it's a stricter variation of =npm install= which guarantees that "[[https://docs.npmjs.com/files/package-lock.json][subsequent installs are able to generate identical trees]]". But are they really identical? I could see that I didn't have the same problems of different installation outputs, but I didn't know for *sure* if it was really identical. ** Computing the hash of a directory's content -I quickly searched for a way to check for the hash signature of an entire directory tree, but I couldn't find one. I've made a poor man's [[https://en.wikipedia.org/wiki/Merkle_tree][Merkle tree]] implementation using =sha256sum= and a few piped comands at the terminal: +I quickly searched for a way to check for the hash signature of an entire directory tree, but I couldn't find one. I've made a poor man's [[https://en.wikipedia.org/wiki/Merkle_tree][Merkle tree]] implementation using =sha256sum= and a few piped commands at the terminal: #+BEGIN_SRC bash -n merkle-tree () { dirname="${1-.}" @@ -24,8 +24,8 @@ Going through it line by line: - #2 it accepts a single argument: the directory to compute the merkle tree from. If nothing is given, it runs on the current directory (=.=); - #3 we go to the directory, so we don't get different prefixes in =find='s output (like =../a/b=); - #4 we get all files from the directory tree. Since we're using =sha256sum= to compute the hash of the file contents, we need to filter out folders from it; -- #5 we need to sort the output, since different filesystems and =find= implementations may return files in different orders; -- #6 we use =xargs= to compute the hash of each file individually through =sha256sum=. Since a file may contain spaces we need to scape it with quotes; +- #5 we need to sort the output, since different file systems and =find= implementations may return files in different orders; +- #6 we use =xargs= to compute the hash of each file individually through =sha256sum=. Since a file may contain spaces we need to escape it with quotes; - #7 we compute the hash of the combined hashes. Since =sha256sum= output is formatted like =<hash> <filename>=, it produces a different final hash if a file ever changes name without changing it's content; - #8 we get the final hash output, excluding the =<filename>= (which is =-= in this case, aka =stdin=). *** Positive points: @@ -73,11 +73,13 @@ In this test case I'll take the main repo of [[https://lernajs.io/][Lerna]][fn:j #+END_SRC Good job =npm ci= :) -#6 and #9 take some time to run (21s in my machine), but this specific use case isn't performance sensitive. The slowest step is computing the hash of each individual file. +#6 and #9 take some time to run (21 seconds in my machine), but this specific use case isn't performance sensitive. The slowest step is computing the hash of each individual file. ** Conclusion =npm ci= really "generates identical trees". I'm not aware of any other existing solution for verifying the hash signature of a directory. If you know any I'd [[mailto:eu@euandre.org][like to know]]. +** /Edit/ +2019/05/22: Fix spelling. [fn:npm-install] The [[https://docs.npmjs.com/cli/install#description][documentation]] claims =npm install= is driven by the existing =package-lock.json=, but that' actually [[https://github.com/npm/npm/issues/17979#issuecomment-332701215][a little bit tricky]]. [fn:js-repos] Finding a big known repo that actually committed the =package-lock.json= file was harder than I expected. |