Storing CI data on Git notes

Posted on November 30, 2020

Extending the bare bones CI server I’ve talked about before, divoplade on Freenode suggested storing CI artifacts on Git notes, such as tarballs, binaries, logs, etc.

I’ve written a small script that will put log files and CI job data on Git notes, and make it visible on the porcelain log. It is a simple extension of the previous article:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env bash
set -Eeuo pipefail
set -x

PREFIX='/srv/ci/vps'
mkdir -p "$PREFIX"
read -r _ SHA _ # oldrev newrev refname
FILENAME="$(date -Is)-$SHA.log"
LOGFILE="$PREFIX/$FILENAME"
exec &> >(tee -a "$LOGFILE")

echo "Starting CI job at: $(date -Is)"

finish() {
  STATUS="$?"
  printf "\n\n>>> exit status was %s\n" "$STATUS"
  echo "Finishing CI job at: $(date -Is)"
  popd
  NOTE=$(cat <<EOF
See CI logs with:
  git notes --ref=refs/notes/ci-logs show $SHA
  git notes --ref=refs/notes/ci-data show $SHA
EOF
)
  git notes --ref=refs/notes/ci-data add -f -m "$STATUS $FILENAME"
  git notes --ref=refs/notes/ci-logs add -f -F "$LOGFILE"
  git notes add -f -m "$NOTE"
  printf "\n\n>>> CI logs added as Git note."
}
trap finish EXIT

unset GIT_DIR
CLONE="$(mktemp -d)"
git clone . "$CLONE"
pushd "$CLONE"
git config --global user.email git@euandre.org
git config --global user.name 'EuAndreh CI'

./container make check site
./container make publish

The important part is in the finish() function:

A commit now has an attached note, and shows it whenever you look at it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ git show 87c57133abd8be5d7cc46afbf107f59b26066575
commit 87c57133abd8be5d7cc46afbf107f59b26066575
Author: EuAndreh <eu@euandre.org>
Date:   Wed Feb 24 21:58:28 2021 -0300

    vps/machines.scm: Change path to cronjob files

Notes:
    See CI logs with:
      git notes --ref=refs/notes/ci-logs show 87c57133abd8be5d7cc46afbf107f59b26066575
      git notes --ref=refs/notes/ci-data show 87c57133abd8be5d7cc46afbf107f59b26066575

diff --git a/servers/vps/machines.scm b/servers/vps/machines.scm
index d1830ca..a4ccde7 100644
--- a/servers/vps/machines.scm
+++ b/servers/vps/machines.scm
@@ -262,8 +262,8 @@ pki " mail-domain " key  \"" (tls-priv-for mail-domain) "\""))
             (service mcron-service-type
                      (mcron-configuration
                       (jobs
-                       (list #~(job "30 1 * * 1" "guix gc -d")
-                             #~(job "30 0 * * *" "/var/lib/euandreh/backup.sh")))))
+                       (list #~(job "30 1 * * 1" "/opt/bin/gc.sh")
+                             #~(job "30 0 * * *" "/opt/bin/backup.sh")))))
             (service dhcp-client-service-type)
             #;
             (service opensmtpd-service-type

Other tools such as cgit will also show notes on the web interface: https://euandreh.xyz/servers.git/commit?id=87c57133abd8be5d7cc46afbf107f59b26066575.

You can go even further: since cgit can serve raw blob directly, you can even serve such artifacts (log files, release artifacts, binaries) from cgit itself:

1
2
3
$ SHA="$(git notes --ref=refs/notes/ci-logs list 87c57133abd8be5d7cc46afbf107f59b26066575)"
$ echo "https://euandreh.xyz/servers.git/blob?id=$SHA"
https://euandreh.xyz/servers.git/blob?id=1707a97bae24e3864fe7943f8dda6d01c294fb5c

And like that you’ll have cgit serving the artifacts for you: https://euandreh.xyz/servers.git/blob?id=1707a97bae24e3864fe7943f8dda6d01c294fb5c.