1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
|
---
title: Verifying npm ci reproducibility
date: 2018-08-01
---
When [[https://blog.npmjs.org/post/161081169345/v500][npm@5]] came bringing [[https://docs.npmjs.com/files/package-locks][package-locks]] with it, I was confused about the benefits it provided, since running =npm install= more than once could resolve all the dependencies again and yield yet another fresh =package-lock.json= file. The message saying "you should add this file to version control" left me hesitant on what to do[fn:npm-install].
However the [[https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable][addition of =npm ci=]] filled this gap: it's a stricter variation of =npm install= which guarantees that "[[https://docs.npmjs.com/files/package-lock.json][subsequent installs are able to generate identical trees]]". But are they really identical? I could see that I didn't have the same problems of different installation outputs, but I didn't know for *sure* if it was really identical.
** Computing the hash of a directory's content
I quickly searched for a way to check for the hash signature of an entire directory tree, but I couldn't find one. I've made a poor man's [[https://en.wikipedia.org/wiki/Merkle_tree][Merkle tree]] implementation using =sha256sum= and a few piped commands at the terminal:
#+BEGIN_SRC bash -n
merkle-tree () {
dirname="${1-.}"
pushd "$dirname"
find . -type f | \
sort | \
xargs -I{} sha256sum "{}" | \
sha256sum | \
awk '{print $1}'
popd
}
#+END_SRC
Going through it line by line:
- #1 we define a Bash function called =merkle-tree=;
- #2 it accepts a single argument: the directory to compute the merkle tree from. If nothing is given, it runs on the current directory (=.=);
- #3 we go to the directory, so we don't get different prefixes in =find='s output (like =../a/b=);
- #4 we get all files from the directory tree. Since we're using =sha256sum= to compute the hash of the file contents, we need to filter out folders from it;
- #5 we need to sort the output, since different file systems and =find= implementations may return files in different orders;
- #6 we use =xargs= to compute the hash of each file individually through =sha256sum=. Since a file may contain spaces we need to escape it with quotes;
- #7 we compute the hash of the combined hashes. Since =sha256sum= output is formatted like =<hash> <filename>=, it produces a different final hash if a file ever changes name without changing it's content;
- #8 we get the final hash output, excluding the =<filename>= (which is =-= in this case, aka =stdin=).
*** Positive points:
1. ignore timestamp: running more than once on different installation yields the same hash;
2. the name of the file is included in the final hash computation.
*** Limitations:
1. it ignores empty folders from the hash computation;
2. the implementation's only goal is to represent using a digest whether the content of a given directory is the same or not. Leaf presence checking is obviously missing from it.
*** Testing locally with sample data
#+BEGIN_SRC bash -n
mkdir /tmp/merkle-tree-test/
cd /tmp/merkle-tree-test/
mkdir -p a/b/ a/c/ d/
echo "one" > a/b/one.txt
echo "two" > a/c/two.txt
echo "three" > d/three.txt
merkle-tree . # output is be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
merkle-tree . # output still is be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
echo "four" > d/four.txt
merkle-tree . # output is now b5464b958969ed81815641ace96b33f7fd52c20db71a7fccc45a36b3a2ae4d4c
rm d/four.txt
merkle-tree . # output back to be343bb01fe00aeb8fef14a3e16b1c3d1dccbf86d7e41b4753e6ccb7dc3a57c3
echo "hidden-five" > a/b/one.txt
merkle-tree . # output changed 471fae0d074947e4955e9ac53e95b56e4bc08d263d89d82003fb58a0ffba66f5
#+END_SRC
It seems to work for this simple test case.
You can try copying and pasting it to verify the hash signatures.
** Using =merkle-tree= to check the output of =npm ci=
/I've done all of the following using Node.js v8.11.3 and npm@6.1.0./
In this test case I'll take the main repo of [[https://lernajs.io/][Lerna]][fn:js-repos]:
#+BEGIN_SRC bash -n
cd /tmp/
git clone https://github.com/lerna/lerna.git
cd lerna/
git checkout 57ff865c0839df75dbe1974971d7310f235e1109
npm ci
merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
rm -rf node_modules/
npm ci
merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
npm ci # test if it also works with an existing node_modules/ folder
merkle-tree node_modules/ # outputs 11e218c4ac32fac8a9607a8da644fe870a25c99821167d21b607af45699afafa
#+END_SRC
Good job =npm ci= :)
#6 and #9 take some time to run (21 seconds in my machine), but this specific use case isn't performance sensitive. The slowest step is computing the hash of each individual file.
** Conclusion
=npm ci= really "generates identical trees".
I'm not aware of any other existing solution for verifying the hash signature of a directory. If you know any I'd [[mailto:eu@euandre.org][like to know]].
** /Edit/
2019/05/22: Fix spelling.
[fn:npm-install] The [[https://docs.npmjs.com/cli/install#description][documentation]] claims =npm install= is driven by the existing =package-lock.json=, but that' actually [[https://github.com/npm/npm/issues/17979#issuecomment-332701215][a little bit tricky]].
[fn:js-repos] Finding a big known repo that actually committed the =package-lock.json= file was harder than I expected.
|
