Storing CI data on Git notes
Extending the bare bones CI server I’ve talked about before, divoplade on Freenode suggested storing CI artifacts on Git notes, such as tarballs, binaries, logs, etc.
I’ve written a small script that will put log files and CI job data on Git notes, and make it visible on the porcelain log. It is a simple extension of the previous article:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env bash
set -Eeuo pipefail
set -x
PREFIX='/srv/ci/vps'
mkdir -p "$PREFIX"
read -r _ SHA _ # oldrev newrev refname
FILENAME="$(date -Is)-$SHA.log"
LOGFILE="$PREFIX/$FILENAME"
exec &> >(tee -a "$LOGFILE")
echo "Starting CI job at: $(date -Is)"
finish() {
STATUS="$?"
printf "\n\n>>> exit status was %s\n" "$STATUS"
echo "Finishing CI job at: $(date -Is)"
popd
NOTE=$(cat <<EOF
See CI logs with:
git notes --ref=refs/notes/ci-logs show $SHA
git notes --ref=refs/notes/ci-data show $SHA
EOF
)
git notes --ref=refs/notes/ci-data add -f -m "$STATUS $FILENAME"
git notes --ref=refs/notes/ci-logs add -f -F "$LOGFILE"
git notes add -f -m "$NOTE"
printf "\n\n>>> CI logs added as Git note."
}
trap finish EXIT
unset GIT_DIR
CLONE="$(mktemp -d)"
git clone . "$CLONE"
pushd "$CLONE"
git config --global user.email git@euandre.org
git config --global user.name 'EuAndreh CI'
./container make check site
./container make publish
The important part is in the finish() function:
- #25 stores the exit status and the generated filename separated by spaces;
- #26 adds the log file in a note using the
refs/notes/ci-logsref; - #27 it adds a note to the commit saying how to see the logs.
A commit now has an attached note, and shows it whenever you look at it:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ git show 87c57133abd8be5d7cc46afbf107f59b26066575
commit 87c57133abd8be5d7cc46afbf107f59b26066575
Author: EuAndreh <eu@euandre.org>
Date: Wed Feb 24 21:58:28 2021 -0300
vps/machines.scm: Change path to cronjob files
Notes:
See CI logs with:
git notes --ref=refs/notes/ci-logs show 87c57133abd8be5d7cc46afbf107f59b26066575
git notes --ref=refs/notes/ci-data show 87c57133abd8be5d7cc46afbf107f59b26066575
diff --git a/servers/vps/machines.scm b/servers/vps/machines.scm
index d1830ca..a4ccde7 100644
--- a/servers/vps/machines.scm
+++ b/servers/vps/machines.scm
@@ -262,8 +262,8 @@ pki " mail-domain " key \"" (tls-priv-for mail-domain) "\""))
(service mcron-service-type
(mcron-configuration
(jobs
- (list #~(job "30 1 * * 1" "guix gc -d")
- #~(job "30 0 * * *" "/var/lib/euandreh/backup.sh")))))
+ (list #~(job "30 1 * * 1" "/opt/bin/gc.sh")
+ #~(job "30 0 * * *" "/opt/bin/backup.sh")))))
(service dhcp-client-service-type)
#;
(service opensmtpd-service-type
Other tools such as cgit will also show notes on the web interface: https://euandre.org/git/servers/commit?id=87c57133abd8be5d7cc46afbf107f59b26066575.
You can go even further: since cgit can serve raw blob directly, you can even serve such artifacts (log files, release artifacts, binaries) from cgit itself:
And like that you’ll have cgit serving the artifacts for you: https://euandre.org/git/servers/blob?id=1707a97bae24e3864fe7943f8dda6d01c294fb5c.