path: root/_tils/2020-11-30-storing-ci-data-on-git-notes.md

---

title: Storing CI data on Git notes

date: 2020-11-30

layout: post

lang: en

ref: storing-ci-data-on-git-notes

eu_categories: git,ci

---

Extending the bare bones CI server I've [talked about before][previous-article],
divoplade on Freenode suggested storing CI artifacts on [Git notes][git-notes],
such as tarballs, binaries, logs, *etc*.

I've written a small script that will put log files and CI job data on Git notes,
and make it visible on the porcelain log. It is a simple extension of the
previous article:

```shell
#!/usr/bin/env bash
set -Eeuo pipefail
set -x

PREFIX='/srv/ci/vps'
mkdir -p "$PREFIX"
read -r _ SHA _ # oldrev newrev refname
FILENAME="$(date -Is)-$SHA.log"
LOGFILE="$PREFIX/$FILENAME"
exec &> >(tee -a "$LOGFILE")

echo "Starting CI job at: $(date -Is)"

finish() {
  STATUS="$?"
  printf "\n\n>>> exit status was %s\n" "$STATUS"
  echo "Finishing CI job at: $(date -Is)"
  popd
  NOTE=$(cat <<EOF
See CI logs with:
  git notes --ref=refs/notes/ci-logs show $SHA
  git notes --ref=refs/notes/ci-data show $SHA
EOF
)
  git notes --ref=refs/notes/ci-data add -f -m "$STATUS $FILENAME"
  git notes --ref=refs/notes/ci-logs add -f -F "$LOGFILE"
  git notes add -f -m "$NOTE"
  printf "\n\n>>> CI logs added as Git note."
}
trap finish EXIT

unset GIT_DIR
CLONE="$(mktemp -d)"
git clone . "$CLONE"
pushd "$CLONE"
git config --global user.email git@euandre.org
git config --global user.name 'EuAndreh CI'

./container make check site
./container make publish
```

The important part is in the `finish()` function:
- #25 stores the exit status and the generated filename separated by spaces;
- #26 adds the log file in a note using the `refs/notes/ci-logs` ref;
- #27 it adds a note to the commit saying how to see the logs.

A commit now has an attached note, and shows it whenever you look at it:

```diff
$ git show 930ba1888f49f11e52a4a715438cd9f5f413dd9c
commit 930ba1888f49f11e52a4a715438cd9f5f413dd9c (oldvps/master)
Author: EuAndreh <eu@euandre.org>
Date:   Mon Nov 30 01:11:38 2020 -0300

    vps.scm: Uncomment mcron job time marker

Notes:
    See CI logs with:
      git notes --ref=refs/notes/ci-logs show 930ba1888f49f11e52a4a715438cd9f5f413dd9c
      git notes --ref=refs/notes/ci-data show 930ba1888f49f11e52a4a715438cd9f5f413dd9c

diff --git a/sync/vps.scm b/sync/vps.scm
index 3f6ca69..02b9cc6 100644
--- a/sync/vps.scm
+++ b/sync/vps.scm
@@ -280,7 +280,7 @@ pki " mail-domain " key  \"" (tls-priv-for mail-domain) "\"")))
              tls-prefixes)))

 (define generate-ci-index-html-job
-  #~(job "* * * * *" ;; "*/5 * * * *"
+  #~(job "*/5 * * * *"
          #$(program-file
             "generate-ci-index-html.scm"
             (with-imported-modules (modules:source-module-closure
```

Other tools such as [cgit][cgit] will also show notes on the web interface:
[https://euandreh.xyz/vps.git/commit?id=930ba1888f49f11e52a4a715438cd9f5f413dd9c](https://euandreh.xyz/vps.git/commit?id=930ba1888f49f11e52a4a715438cd9f5f413dd9c)

You can go even further: since cgit can serve raw blob directly, you can even
serve such artifacts (log files, release artifacts, binaries) from cgit itself:

```shell
$ SHA="$(git notes --ref=refs/notes/ci-logs list 930ba1888f49f11e52a4a715438cd9f5f413dd9c)"
$ echo "https://euandreh.xyz/vps.git/blob?id=$SHA"
https://euandreh.xyz/vps.git/blob?id=b3a6438a0c7a47864c54c61359b6ef50e864dbff
```

And like that you'll have cgit serving the artifacts for you:
[https://euandreh.xyz/vps.git/blob?id=b3a6438a0c7a47864c54c61359b6ef50e864dbff](https://euandreh.xyz/vps.git/blob?id=b3a6438a0c7a47864c54c61359b6ef50e864dbff)

[previous-article]: {% link _tils/2020-11-12-diy-bare-bones-ci-server-with-bash-and-nix.md %}
[git-notes]: https://git-scm.com/docs/git-notes
[cgit]: https://git.zx2c4.com/cgit/