* Tasks - v4
** TODO Run =sudo= as =nixos= user in server
** TODO nginx magic =sslCiphers= value
Why not the default? What do those mean?
** TODO How to handle IP changes in mail server?
** TODO Add borg backup to crontab
** TODO Add 2FA to Vultr
** TODO Clean-up garbage backups from rsync.net
** TODO Harden the server
*** TODO [#C] [[https://www.reddit.com/r/selfhosted/comments/bw8hqq/top_3_measures_to_secure_your_virtual_private/][Top 3 measures to secure your Virtual Private Server? (VPS)]]
*** TODO [#A] [[https://docs.nextcloud.com/server/stable/admin_manual/installation/harden_server.html][Nextcloud: Hardening and security guidance]]
*** TODO [#A] [[https://ownyourbits.com/2017/03/25/nextcloud-a-security-analysis/][NextCloud, a security analysis]]
*** TODO [#B] [[https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.md][Check for HSTS header configuration]]
** TODO README with setup instructions
** TODO Fix =file: command not found= in CI
The =file= package is imported in =shell.nix= but =~/.buildenv= is sourced before.
** TODO Use =--pure= for =nix-shell= scripts
** TODO Add volume to fstab
Can I use Terraform form this?
* Services - v2
** TODO =cloud.$tld=: Nextcloud: storage, calendar, contacts, notes and talk
** TODO =chat.$tld=: Matrix Synapse server, or a XMPP server
** TODO =git.$tld=: git-instaweb (or cgit) server with repositories from ~/dev/libre/
** TODO =audio.$tld=: FunkWhale
** TODO =mail.$tld=: postfix, dovecot, spamassasin, opendkim, etc
No need for roundcube, Nextcloud has a web interface client.
** TODO =$tld=: current Jekyll blog
* Resources
** [[https://github.com/mail-in-a-box/mailinabox][Mail-in-a-Box]]
** [[https://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/][NSA-proof your e-mail in 2 hours]]
** [[https://www.iredmail.org/][iRedMail]]
** [[https://blog.harveydelaney.com/hosting-websites-using-docker-nginx/][Hosting Multiple Websites with SSL using Docker, Nginx and a VPS]]
** [[https://github.com/sovereign/sovereign/][Sovereign]]
** [[https://github.com/nixcloud/nixcloud-webservices][nixcloud-webservices]]
** [[https://github.com/Kickball/awesome-selfhosted#email][Awesome-Selfhosted: Email]]
** [[https://arstechnica.com/information-technology/2014/04/taking-e-mail-back-part-4-the-finale-with-webmail-everything-after/][Taking e-mail back]]
** [[https://jacobneplokh.com/how-to-setup-nextcloud-on-nixos][How to Setup Nextcloud on NixOS]]
* Decisions
** Use external git repository as an encrypted database
Terraform does have the support for "backends" where it can store =.tfstate= files.

From the list of supported backends, the [[https://www.terraform.io/docs/backends/types/s3.html][S3]] option initially stands out as the simplest to configure. It doesn't however support state locking, only if also configuring DynamoDB.

This extra configuration and complexity isn't attractive, and I can achieve similar outcomes by using the =local= backend and storing it properly. Even better than sending to S3 and setting up the proper revision headers is to just use a separate repository to keep it.

Using the same repository would create an unwanted cyclic process where the repository pipeline commits in itself.

All data stored on git is encrypted with [[https://www.agwa.name/projects/git-crypt/][git-crypt]], which means git isn't being actually used as a source code repository, but as a versioned filesystem database.

By taking advantage of the sourcehut ecosystem, it was easier to setup the access of the pipeline to the ad-hoc Terraform backend.

I created a repository called [[https://git.sr.ht/~euandreh/vps-state/][=vps-state=]] to store the encrypted =.tfstate= and =.tfplan= files. During the CI run, the pipeline creates new a =.tfplan= file and commits it into =vps-state=, and after applying the plan it updates the =.tfstate= file and adds this change to =vps-state=.
** Move external =vps-state= into =vps=
I want to move the Terraform state into this repository, I don't like the inconsistency that having 2 repos create.

If I move vps-state into vps, I'll have to remove the terraform steps from the pipeline: it can't commit to itself (as described above). This has an upside too: instead of automatically approving whatever plan Terraform creates, I review the plan before applying.

It makes the deploying less automatic, but this removes the IP reputation email issue.

This means that the Terraform provisioning should stay out of the CI and be run only locally.
** Run locally instead of on CI
It makes it less automagic, but greatly simplifies the configuration, like removing custom =ssh.env.conf=, =mail.sh=, =vps-box-client.pub=, etc.
** Configuration of =StrictHostKeyChecking=
We have 3 cases where I'm pushing things to the server and I'm dealing with it differently:
*** 1. Pushing updates to the =vps-state= repository
I could whitelist the SSH keys from the =git.sr.ht= servers, but this could break on every key rotation of the server.

In can of the server address being spoofed, the content would be readable by the attacker, since we're doing all the encryption on the client. We would, however, lose a Terraform state file update. As of right now, I'm OK with this trade-off.
*** 2. Running =scp= to the deployed VPS
On this situation I want to be sure I know where I'm pushing to.

In order to avoid adding =StrictHostKeyChecking= when running =ssh= and =scp=, every time the SSH key is rotated I generate a new =./secrets/ssh/known-hosts.txt= file with the proper SSH public key.

This way we can avoid prompting for SSH server fingerprint trust on the CI and avoid adding =StrictHostKeyChecking= on those calls.
*** 3. Backup server
Even though the backup is encrypted before sending the data, I don't want to risk loosing a backup to an spoofed server. I'd rather break the build instead.
** Don't use Ansible as a =local-exec= provisioner from Terraform
Instead, explicitly call =ansible-playbook= after =terraform apply= finished running.

This way we test the DNS A record -> Floating IP -> Droplet IP path. We can't do that inside Terraform declaration because the =local-exec= provisioning command runs before the =digitalocean_floating_ip_assignment= is created, and we can't create a cyclic dependency between the two resources.

We could use the raw Droplet IP instead of the DNS A record, but I prefer calling it later in order to always test the full DNS resolution.
* Questions
** How to best handle IP changes when the server changes? How does this affect the email sending IP reputation?
* Scrath
https://federationtester.matrix.org/
EteSync?