Storing CI data on Git notes

Posted on November 30, 2020

Extending the bare bones CI server I’ve talked about before, divoplade on Freenode suggested storing CI artifacts on Git notes, such as tarballs, binaries, logs, etc.

I’ve written a small script that will put log files and CI job data on Git notes, and make it visible on the porcelain log. It is a simple extension of the previous article:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env bash
set -Eeuo pipefail
set -x

PREFIX='/srv/ci/vps'
mkdir -p "$PREFIX"
read -r _ SHA _ # oldrev newrev refname
FILENAME="$(date -Is)-$SHA.log"
LOGFILE="$PREFIX/$FILENAME"
exec &> >(tee -a "$LOGFILE")

echo "Starting CI job at: $(date -Is)"

finish() {
  STATUS="$?"
  printf "\n\n>>> exit status was %s\n" "$STATUS"
  echo "Finishing CI job at: $(date -Is)"
  popd
  NOTE=$(cat <<EOF
See CI logs with:
  git notes --ref=refs/notes/ci-logs show $SHA
  git notes --ref=refs/notes/ci-data show $SHA
EOF
)
  git notes --ref=refs/notes/ci-data add -f -m "$STATUS $FILENAME"
  git notes --ref=refs/notes/ci-logs add -f -F "$LOGFILE"
  git notes add -f -m "$NOTE"
  printf "\n\n>>> CI logs added as Git note."
}
trap finish EXIT

unset GIT_DIR
CLONE="$(mktemp -d)"
git clone . "$CLONE"
pushd "$CLONE"
git config --global user.email git@euandre.org
git config --global user.name 'EuAndreh CI'

./container make check site
./container make publish

The important part is in the finish() function:

A commit now has an attached note, and shows it whenever you look at it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ git show 930ba1888f49f11e52a4a715438cd9f5f413dd9c
commit 930ba1888f49f11e52a4a715438cd9f5f413dd9c (oldvps/master)
Author: EuAndreh <eu@euandre.org>
Date:   Mon Nov 30 01:11:38 2020 -0300

    vps.scm: Uncomment mcron job time marker

Notes:
    See CI logs with:
      git notes --ref=refs/notes/ci-logs show 930ba1888f49f11e52a4a715438cd9f5f413dd9c
      git notes --ref=refs/notes/ci-data show 930ba1888f49f11e52a4a715438cd9f5f413dd9c

diff --git a/sync/vps.scm b/sync/vps.scm
index 3f6ca69..02b9cc6 100644
--- a/sync/vps.scm
+++ b/sync/vps.scm
@@ -280,7 +280,7 @@ pki " mail-domain " key  \"" (tls-priv-for mail-domain) "\"")))
              tls-prefixes)))

 (define generate-ci-index-html-job
-  #~(job "* * * * *" ;; "*/5 * * * *"
+  #~(job "*/5 * * * *"
          #$(program-file
             "generate-ci-index-html.scm"
             (with-imported-modules (modules:source-module-closure

Other tools such as cgit will also show notes on the web interface: https://git.euandreh.xyz/vps/commit?id=930ba1888f49f11e52a4a715438cd9f5f413dd9c

You can go even further: since cgit can serve raw blob directly, you can even serve such artifacts (log files, release artifacts, binaries) from cgit itself:

1
2
3
$ SHA="$(git notes --ref=refs/notes/ci-logs list 930ba1888f49f11e52a4a715438cd9f5f413dd9c)"
$ echo "https://git.euandreh.xyz/vps/blob?id=$SHA"
https://git.euandreh.xyz/vps/blob?id=b3a6438a0c7a47864c54c61359b6ef50e864dbff

And like that you’ll have cgit serving the artifacts for you: https://git.euandreh.xyz/vps/blob?id=b3a6438a0c7a47864c54c61359b6ef50e864dbff