* Tasks
** TODO External volume
** TODO Backups
If possible, put every data subfolder under the same folder, and just backup the
top-level folder. This also allows me to put it on an external volum and grow it
more easily.

No real need to backup cgit, Jekyll, documetation and Cuirass, but useful to
have if available.

The certificates should be backed up, so that restoring doesn't involve
re-creating everything from scratch.
*** TODO Email
*** TODO Matrix
*** TODO Certificates
: /etc/letsencrypt
* Services
** TODO =git.$tld=: cgit
** TODO =$project.$tld=: static documentation for projects
** TODO =ci.$tld=: cuirass
** TODO =mail.$tld=: email
** TODO =chat.$tld=: Matrix
** TODO =meet.$tld=: Jitsi/Nextcloud Talk
** TODO =$tld=: Jekyll blog
** TODO =audio.$tld=: FunkWhale
* Decisions
** Matrix over XMPP
I'm picking Matrix. Not because of the protocol or anything else, but because it
has the two relevant double-puppeting bridges: mautrix-telegram and
mautrix-whatsapp.

TBH I like XMPP much more, but without working puppeting bridges, I would stay
isolated with it, which would defeat the purpose of having a chat server on the
first place.

Maybe an XMPP double-puppeting bridge could allow me to use an XMPP client to
talk with Telegram and WhatsApp chats.
* COMMENT Old
** Tasks - v4
*** TODO Symlink stateful directories to inside =/data/=
- State "TODO"       from              [2020-09-23 mer. 14:45]

Repositories to be kept across instances:
- /var/lib/acme/
- /var/lib/matrix-synapse/
- /var/lib/nextcloud/
*** DONE "Migration": use =euandreh_nextcloud_= prefix to database table
CLOSED: [2020-09-22 mar. 18:55]
*** DONE cgit: show README in about section
CLOSED: [2020-09-06 dim. 08:59]
- State "DONE"       from              [2020-09-06 dim. 08:59]
*** CANCELLED Update matterbridge version
CLOSED: [2020-09-06 dim. 08:59]
- State "CANCELLED"  from              [2020-09-06 dim. 08:59] \\
I found matterbridge to be too simplistic for what it was proposing to do. The
puppeting bridges from Matrix are a much more robust solution, and I'll try that
instead later.

https://github.com/42wim/matterbridge/issues/1061
*** DONE Add Prosody DNS record to allow me to use eu@euandreh.xyz as an XMPP address
CLOSED: [2020-08-23 dim. 18:40]
*** CANCELLED Configure MUC on Prosody
CLOSED: [2020-09-22 mar. 18:56]
No XMPP, for now, I'm stick with Matrix and their puppeting bridges.
*** CANCELLED Fix conversejs login
CLOSED: [2020-09-22 mar. 18:56]
*** DOING Terraform restore from backup when provisioning
*** DONE cron: borg, PostgreSQL backup and NixOS update and collect gargabe
CLOSED: [2020-09-23 mer. 15:34]
- State "DONE"       from "DOING"      [2020-09-23 mer. 15:35]
*** DOING snapshot with swap: nixos-rebuild switch takes too much memory
Adding swap memory maybe be a lot slower, but allows me to stay inside the 512MB memory server plan
*** DONE Configure PostgreSQL permissions correctly
CLOSED: [2020-09-23 mer. 14:44]
- State "DONE"       from "TODO"       [2020-09-23 mer. 14:44]
Or even better: switch to SQLite.
*** TODO Harden the server
**** TODO [#C] [[https://www.reddit.com/r/selfhosted/comments/bw8hqq/top_3_measures_to_secure_your_virtual_private/][Top 3 measures to secure your Virtual Private Server? (VPS)]]
**** TODO [#A] [[https://docs.nextcloud.com/server/stable/admin_manual/installation/harden_server.html][Nextcloud: Hardening and security guidance]]
**** TODO [#A] [[https://ownyourbits.com/2017/03/25/nextcloud-a-security-analysis/][NextCloud, a security analysis]]
**** TODO [#B] [[https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.md][Check for HSTS header configuration]]
**** DONE Move secrets outside the Nix store
CLOSED: [2020-09-06 dim. 11:35]
- State "DONE"       from              [2020-09-06 dim. 11:35]
**** DONE Properly configure PostgreSQL
CLOSED: [2020-09-23 mer. 14:45]
- State "DONE"       from "TODO"       [2020-09-23 mer. 14:45]
**** TODO Separate lighttpd user ownership from =$GIT_ROOT=?
*** DOING Add volume
And put in to fstab
Can I use Terraform to automate the mounting?
#+BEGIN_SRC hcl
variable "storage_name" {
  type        = string
  description = "Name of the block storage volume, which will also be the name of it's mount point."
}

resource "vultr_block_storage" "vps_storage" {
  size_gb     = 10
  region_id   = 9
  attached_id = vultr_server.vps_server.id
  label       = var.storage_name
  live        = "yes"
}
#+END_SRC
*** CANCELLED Prosody HTTP uploads
CLOSED: [2020-09-22 mar. 18:57]
*** DONE Use nixos-unstable
CLOSED: [2020-09-22 mar. 19:00]
*** TODO Telegram in Matrix
*** TODO WhatsApp in Matrix
*** TODO Matrix federation
*** TODO Use euandreh.xyz instead of matrix.euandreh.xyz for Matrix identifiers
*** TODO Run =borg.sh= when in maintenance mode
- State "TODO"       from              [2020-09-23 mer. 15:51]
** Tasks - v5
*** DONE Stop doing chmod to share certificates
CLOSED: [2020-09-22 mar. 19:04]
*** TODO Run cgit from nginx instead of using lighttpd
*** TODO EteSync
*** DONE Use =$SOMETHING_DOMAIN= instead of =$SOMETHING_TLD=
CLOSED: [2020-09-22 mar. 19:04]
*** TODO Create less certificates
*** TODO Bundle favicons instead of separate rsync
** Services - v2
*** DONE =cloud.$tld=: Nextcloud: storage, calendar, contacts, notes and talk
CLOSED: [2020-08-14 ven. 09:29]
*** DONE =chat.$tld=: [[https://conversejs.org/][Converse]] interface to Prosody server
CLOSED: [2020-08-16 dim. 21:58]
*** DONE =git.$tld=: cgit server with repositories from ~/dev/libre/
CLOSED: [2020-08-14 ven. 09:29]
*** TODO =mail.$tld=: postfix, dovecot, spamassasin, opendkim, etc
No need for roundcube, Nextcloud has a web interface client.
*** TODO =$tld=: current Jekyll blog
*** TODO =audio.$tld=: FunkWhale
** Resources
*** [[https://github.com/mail-in-a-box/mailinabox][Mail-in-a-Box]]
*** [[https://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/][NSA-proof your e-mail in 2 hours]]
*** [[https://www.iredmail.org/][iRedMail]]
*** [[https://blog.harveydelaney.com/hosting-websites-using-docker-nginx/][Hosting Multiple Websites with SSL using Docker, Nginx and a VPS]]
*** [[https://github.com/sovereign/sovereign/][Sovereign]]
*** [[https://github.com/nixcloud/nixcloud-webservices][nixcloud-webservices]]
*** [[https://github.com/Kickball/awesome-selfhosted#email][Awesome-Selfhosted: Email]]
*** [[https://arstechnica.com/information-technology/2014/04/taking-e-mail-back-part-4-the-finale-with-webmail-everything-after/][Taking e-mail back]]
*** [[https://jacobneplokh.com/how-to-setup-nextcloud-on-nixos][How to Setup Nextcloud on NixOS]]
*** [[https://www.vultr.com/docs/install-nixos-on-vultr][Install NixOS on Vultr]]
*** [[https://www.linode.com/docs/tools-reference/custom-kernels-distros/install-nixos-on-linode/][Install and Configure NixOS on a Linode]]
** Decisions
*** Use external git repository as an encrypted database
Terraform does have the support for "backends" where it can store =.tfstate= files.

From the list of supported backends, the [[https://www.terraform.io/docs/backends/types/s3.html][S3]] option initially stands out as the simplest to configure. It doesn't however support state locking, only if also configuring DynamoDB.

This extra configuration and complexity isn't attractive, and I can achieve similar outcomes by using the =local= backend and storing it properly. Even better than sending to S3 and setting up the proper revision headers is to just use a separate repository to keep it.

Using the same repository would create an unwanted cyclic process where the repository pipeline commits in itself.

All data stored on git is encrypted with [[https://www.agwa.name/projects/git-crypt/][git-crypt]], which means git isn't being actually used as a source code repository, but as a versioned filesystem database.

By taking advantage of the sourcehut ecosystem, it was easier to setup the access of the pipeline to the ad-hoc Terraform backend.

I created a repository called [[https://git.sr.ht/~euandreh/vps-state/][=vps-state=]] to store the encrypted =.tfstate= and =.tfplan= files. During the CI run, the pipeline creates new a =.tfplan= file and commits it into =vps-state=, and after applying the plan it updates the =.tfstate= file and adds this change to =vps-state=.
*** Move external =vps-state= into =vps=
I want to move the Terraform state into this repository, I don't like the inconsistency that having 2 repos create.

If I move vps-state into vps, I'll have to remove the terraform steps from the pipeline: it can't commit to itself (as described above). This has an upside too: instead of automatically approving whatever plan Terraform creates, I review the plan before applying.

It makes the deploying less automatic, but this removes the IP reputation email issue.

This means that the Terraform provisioning should stay out of the CI and be run only locally.
*** Run locally instead of on CI
It makes it less automagic, but greatly simplifies the configuration, like removing custom =ssh.env.conf=, =mail.sh=, =vps-box-client.pub=, etc.
*** Configuration of =StrictHostKeyChecking=
We have 3 cases where I'm pushing things to the server and I'm dealing with it differently:
**** 1. Pushing updates to the =vps-state= repository
I could whitelist the SSH keys from the =git.sr.ht= servers, but this could break on every key rotation of the server.

In can of the server address being spoofed, the content would be readable by the attacker, since we're doing all the encryption on the client. We would, however, lose a Terraform state file update. As of right now, I'm OK with this trade-off.
**** 2. Running =scp= to the deployed VPS
On this situation I want to be sure I know where I'm pushing to.

In order to avoid adding =StrictHostKeyChecking= when running =ssh= and =scp=, every time the SSH key is rotated I generate a new =./secrets/ssh/known-hosts.txt= file with the proper SSH public key.

This way we can avoid prompting for SSH server fingerprint trust on the CI and avoid adding =StrictHostKeyChecking= on those calls.
**** 3. Backup server
Even though the backup is encrypted before sending the data, I don't want to risk loosing a backup to an spoofed server. I'd rather break the build instead.
*** Don't use Ansible as a =local-exec= provisioner from Terraform
Instead, explicitly call =ansible-playbook= after =terraform apply= finished running.

This way we test the DNS A record -> Floating IP -> Droplet IP path. We can't do that inside Terraform declaration because the =local-exec= provisioning command runs before the =digitalocean_floating_ip_assignment= is created, and we can't create a cyclic dependency between the two resources.

We could use the raw Droplet IP instead of the DNS A record, but I prefer calling it later in order to always test the full DNS resolution.
*** On public SSH key leakage
As described in "[[https://rushter.com/blog/public-ssh-keys/][Public SSH keys can leak your private infrastructure]]", public
SSH keys can expose undesired infrastructure, specially for targeted attacks.

I'm not considering this a threat, since the link between the server and me is
already public. It may be much more effective to just change the SSH port away
from the default: it doesn't accomplish the same thing, but it prevents simple
detections. It is still possible to find this out via a script, but is orders of
magnitute harder for the attacker.
** Questions
*** How to best handle IP changes when the server changes? How does this affect the email sending IP reputation?
*** Enable 2FA for Nextcloud?
Do XMPP support that too?
** Scratch


# FIXME: get SSH environment from /data/secrets/ ?


# borg init FIXME

FIXME: add ci-build.sh test in VPS itself?
* Scrath
