* Tasks
** DONE Provision DigitalOcean's droplet from Terraform
CLOSED: [2019-05-25 Sat 13:29]
** DONE Properly provision Ubuntu droplet
CLOSED: [2019-05-25 Sat 17:50]
** DONE Automate deployment of updates
CLOSED: [2019-05-29 Wed 17:42]
*** DONE Subtasks
CLOSED: [2019-05-29 Wed 17:42]
**** DONE Fix Debian import of GPG keys
CLOSED: [2019-05-26 Sun 14:34]
While NixOS image isn't fixed, use Debian instead.

The GPG data was all in a single line. I copied and pasted it properly and it was identified correctly.
**** CANCELLED Properly install Nix on Debian image
NixOS patch was applied.
**** DONE Fix NixOS GPG key importing in builds.sr.ht
CLOSED: [2019-05-26 Sun 17:37]
See patch and discussion in [[https://lists.sr.ht/~sircmpwn/sr.ht-dev/%3C20190526162135.1646-1-eu%40euandre.org%3E][sr.ht-dev mailing list]].
**** DONE Use ssh configuration from environment instead of creating and alias for =ssh=
CLOSED: [2019-05-26 Sun 19:44]
***** DONE Relative =IdentityFile= path
CLOSED: [2019-05-26 Sun 19:42]
Used =envsubst= to properly interpolate variables in =ssh.conf=
***** DONE Omit =-F ssh.conf= from command
CLOSED: [2019-05-26 Sun 19:42]
Put it in an environment variable?

Done by appending to content of =~/.ssh/config=.
**** DONE Use DigitalOcean's Floating IP in front of the droplet
CLOSED: [2019-05-28 Tue 23:22]
**** DONE Automate deployment with Terraform and deployment scripts
CLOSED: [2019-05-29 Wed 15:54]
**** DONE Backup data during deployments
CLOSED: [2019-05-28 Tue 00:48]
Is this approach feasible? Will it make the deployment take too much longer? What are the alternatives?

Initial sketch of the backup commands:
#+BEGIN_SOURCE shell
rsync --verbose --progress --stats --update --recursive "$HOME/backups/" "$RSYNC_REMOTE"
borg create -svp -C lzma,6 "~/borgbackup::{hostname}-{now}-${VPS_COMMIT_SHA} ${VOLUME_HOME}"
rsync --verbose --progress --stats --update --recursive "$RSYNC_REMOTE" "$HOME/borgbackups/"
#+END_SOURCE

Implemented with help from https://jstaf.github.io/2018/03/12/backups-with-borg-rsync.html
**** DONE Namecheap whitelist IP limitation
CLOSED: [2019-05-26 Sun 17:14]
Namecheap requires you to specifically whitelist an IP that can perform changes to their API.

[[https://lists.sr.ht/~sircmpwn/sr.ht-discuss/%20%3CCAJk2QMbq8uE1pcG3Uy6w37HUY7W15cQ+sHoj-UBWN-W11AtcrA%40mail.gmail.com%3E][builds.sr.ht]] don't guarantee any specific IP, so whitelisting it isn't an option.

The best candidate so far is using DigitalOcean's Floating IP feature to link a hardcoded IP to a droplet, while the droplet's IP may change. This way any new deployment wouldn't change the public IP of the box, and wouldn't require me to change the DNS A and AAAA records on Namecheap.

This has also an advantage of allowing the email server to keep it's IP address.

The downside is that the deployment of DNS registries isn't fully automated: whenever I have to change a DNS entry, either for adding a new CNAME record or changing an AAAA record, I'll have to:
1- get my own IP;
2- whitelist it on Namecheap's web interface;
3- run a separate Terraform recipe.

The upside is that this should happen less often than a deployment, but still not ideal. The ideal would be to run the Terraform recipe every time, and Terraform would realize that there was no DNS related change and do nothing.
*** Limitations
During build, decrypt content of files and update the deployment.

How does Terraform tfstate file can be handled in this case?

UPDATE:
Terraform does support the so called "backends" to coordinate lock and usage of the =.tfstate= files. On this regard there are no restrictions on continously deploying with Terraform from the CI pipelines.

However the current applications do *not* properly support blue/green deployment, like email, Nextcloud, etc.

We could try to share a shared volume, but that would be a consistency nightmare.

The other option is to always recreate everything, with downtime. The advantage is that we get actual immutable deployments with stateful storage, but there would be downtime for every deployment. This is due to the nature of most of the packaged applications being single node *only*.

There's also the IP reputation issue: recreating everything from scratch every time would lead to new droplets with new IP addresses, which is not a good thing to be changing in a server box.

A reasonable alternative would be to redeploy everything on a different node, with a different TLD, and manually check that. But that would be just like an staging environment, with all of it's downsides too.

In this situation, I if go on with automating the deployment I'd rather pick the downtime option.

I'll start with other services other than email and consider alternatives later.
** DONE Correctly load the SSH keypair using =user_data=
CLOSED: [2019-06-05 Wed 18:16]
*** DONE Disable the =user_data=
CLOSED: [2019-06-05 Wed 17:39]
*** DONE Generate and manually copy the =user-data.env= file
CLOSED: [2019-06-05 Wed 17:39]
*** CANCELLED Run it on the system
*** DONE Run each step individually and check them
CLOSED: [2019-06-05 Wed 18:15]
Check the content of the generated key files.
*** DONE Try to login
CLOSED: [2019-06-05 Wed 18:15]
Problem was on file typo and private key permissions.

Bonus: change SSH port
** DONE Test key rotation
CLOSED: [2019-06-05 Wed 19:28]
See if it is actually working as expected.
** DONE Use Digital Ocean's Volumes for persistent extended storage
CLOSED: [2019-06-05 Wed 20:38]
** TODO Make VPS provisioning more robust
*** DONE Use Ansible (or an equivalent tool) instead of custom Bash scripts
CLOSED: [2019-06-05 Wed 16:41]
They are now more fragile, ad-hoc and imperative than I would like.

Today Terraform won't run the =deploy.sh= if no infrastructure changes are required. Split infrastructure provisioning from server configuration with somethong like Ansible or =nix copy closure= and add extra command in the pipeline run.
*** TODO Always perform a blue/green infrastructure deployment with Terraform
Recreate a new Droplet from scratch, even if no changes happened.

This way every deployment tests the code path of creating everything from scratch again, from the DNS public IP all the way to restoring backups.
*** TODO Destroy and recreate the volume on deployment
Restore from the latest backup with:
#+BEGIN_SOURCE shell
borg list --short --sort-by timestamp | tail -n 1
#+END_SOURCE
** WAITING Configure DNS from Terraform
*** TODO Test provisioning DNS entries with other DNS registrars
*** TODO Have dynamic Floating IP (a.k.a. =$PINNED_IP=)
** TODO Create snapshots before destroying resources
This way the previous good state can be reverted if the deployment fails or the backup can't be restored.

Can a TTL be added to the Droplet and the Volume's snapshots?
** TODO Harden the server
https://www.reddit.com/r/selfhosted/comments/bw8hqq/top_3_measures_to_secure_your_virtual_private/
https://docs.nextcloud.com/server/stable/admin_manual/installation/harden_server.html
https://ownyourbits.com/2017/03/25/nextcloud-a-security-analysis/
Check for HSTS header configuration
** TODO Use git-remote-gcrypt instead of git-crypt for vps-state
Also put all of the content of =secrets/*= into vps-state? Maybe rename it to vps-secret?

Right now, secrets are scattered between the two repositories. By moving I can completely remove =git-crypt= from this repository.
** TODO Run backup on Terraform destroy action instead of manually in =provision.sh=
** DONE Explicitly destroy Droplets before running Terraform apply
CLOSED: [2019-06-05 Wed 19:48]
** TODO Store updated =.tfstate= even in case of deployment failure
Right now the script fails on Terraform commands before reaching git commands. I should trap the error, store on git and only then fail.
** TODO Fix alias in =bash-profile.sh=
** TODO Email verbose (Ansible) log files in case of error
builds.sr.ht only emails the link. Should it be extended to support encrypted log attachments?
** TODO Use environment variables for SSH key paths and volume mounts
** TODO Don't allow backups to fail
* Must
** Fully deployable from code
Use NixOps and Terraform to fully automate all of the configuration.
* Services
** DONE =$tld=: Static webhosting
CLOSED: [2019-05-26 Sun 10:17]
** DONE =wallabag.$tld=: Wallabag
CLOSED: [2019-05-25 Sat 18:02]
A bookmark application must:
- allow me to save and read articles on Android and Firefox;
- allow me to have tags for links;
** NEXT =nextcloud.$tld=: Nextcloud: storage, calendar, contacts, notes
https://github.com/nextcloud/docker

Do I need to configure =NEXTCLOUD_TRUSTED_DOMAINS= or it should work without it?

Start with =cloud.$tld= before =mail.$tld= so I can retire =arrobaponto.org= and reuse it for other projects.

Activate client-side [[https://docs.nextcloud.com/server/11/user_manual/files/encrypting_files.html][encryption]] of files. Activate two-factor authentication for admin and user accounts.

Nextcloud bookmarks instead of Wallabag? Does it have browser extension and Android app? How about the password manager? Is is client-side encrypted?

Should I consider using an external storage provider, like S3, instead of solely local (DigitalOcean's attached volume)?
** TODO =mail.$tld=: Email + webmail
https://github.com/tomav/docker-mailserver
https://mailu.io/master/demo.html
https://mailcow.email/
https://poste.io/
https://github.com/hardware/mailserver

=mail.$tld= could be the Nextcloud mail application!
** TODO =www.$tld= and =blog.$tld=: Redirect to =$tld=
** TODO =hydra.$tld=: Hydra
Does Hydra support release management? The source tarball can live in git.sr.ht, but what about compiled outputs?

I'd like to release both pre-compiled binaries and Docker images.
** TODO =cuirass.$tld=: [[https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git][Cuirass]]
** TODO =annex.$tld=: Public content from Git Annex repositories
Only an static file server, with folders for individual assets.
** TODO =pi-hole.$tld=: Pi-hole
** TODO =pwk.$tld=: Piwik
** TODO =git.$tld=: CGit or GitWeb
https://github.com/iconoeugen/docker-gitweb
** TODO =songbooks.$tld=: Songbooks demo application
** TODO =pires.$tld=: Pires deployment
** TODO =paste.$tld=: Static pastebin
Use Hakyll like in =euandre.org/pastebin/*=, but with root at =paste.$tld/paste-title.html=.
** TODO =link.$tld=: YOURLS
No need for short URLs, but maybe useful for tracking link usage?

What are the privacy implications? Related relevant article: http://stage.vambenepe.com/archives/1596
** CANCELLED =perkeep.$tld=: Perkeep
I'm already covered by using Git Annex for almost everything.
** TODO =matrix.$tld=: Matrix Synapse server
I'm not using IRC a lot right now. Wait for me to interact more with mailing lists and gauge the need of IRC.

It's better than IRC, OTR (XMPP) and everything else, and interoperates with everything. As well said by @matthew, I'm using Matrix mostly as [[https://discourse.mozilla.org/t/matrix-and-irc-mozillians-custom-client/2911/7][a glorified IRC bouncer]]. I do use some Matrix rooms, but mostly for IRC itself.

Also from https://matrix.org/blog/2015/06/22/the-matrix-org-irc-bridge-now-bridges-all-of-freenode:

: Doing this is trivial - you just /join #freenode#channelname:matrix.org from a typical Matrix client - e.g. /join #freenode#Node.js:matrix.org will give you access to all of #Node.js via Matrix, effectively using Matrix as a great big distributed IRC bouncer in the sky ;)

Should continue to consider doing as I continue to use IRC.

Test the Emacs Matrix client along with the server installation.
** WAITING =search.$tld=: Searx instance
Would it be actually more private?
* Questions
** TODO How to dynamically handle Floating IPs?
Right now the current Floating IP defined in =.envrc= was created manually in DigitalOcean's web UI and copied from it to the environment variable.

If everything was teared down, I couldn't recreate everything from source, because the Floating IP would be different.

The ultimate goal would be to upsert a Floating IP address? If no Floating IP address exists, create one. If one already exists (I don't how to get a reference to it), use it.

In other words, I don't want any hardcoded IPs in the recipe. The IP address has to be fixed, and the same on the DNS registrar and DigitalOcean's Floating IP.
** TODO Critiques of Docker?
What does NixOps, DisNix and Dysnomia are trying to accomplish that overlap with Docker? Use sqldiff for NixOps?

Do they do a better job? Why? Why not?

Get a book on advanced Docker, or container fundamentals and dig deeper.
** TODO Should I have an extra backup location?
Maybe rsync the contents of the Borg repository into S3. Should I restore backups from these too?
** TODO Should I be using something like [[https://www.vaultproject.io/][Vault]] instead of git-crypt?
Can it do key rotation?
** TODO Can the =setup.sh= and =provision.sh= scripts be run inside a chroot or a NixOS contained environment?
Right now they can't simply be a derivation because =setup.sh= needs access to the disk and =provision.sh= needs to access the internet.
** DONE Do I want or need Docker? Should I use it?
CLOSED: [2019-05-25 Sat 18:1980]
It was a better path than sticking with NixOps and nixcloud-webservices. It's more widespread and has more things done for it.
** CANCELLED How to share the Nix store across services?
** DONE How to leverage DigitalOcean's block storage?
CLOSED: [2019-05-25 Sat 18:19]
Provision it using Terraform, and use it's path as the =$VOLUME_HOME= variable for containers.

This was I can compartimentalize the data storage to easily backup and duplicate, but also destroy a running droplet and create a new one.
* Nice to have
** =euandreh.org= as =$tld=
** Nix Terraform provisioning
** Upgrade =terraform-godaddy= to 0.12 to support looping over CNAME records
** Upgrade =docker-compose.yaml= file from version 2 to version 3
** Full blue/green deployments without downtime
Only when doing a voluntary restore from backup in a newly created volume.

Is there email software capable of doing this? A distributed email server that doesn't rely so much on the server file system, but on a database instead?
** Do all this in a Raspberry Pi
Even the email server can be in it. Is RAM the beiggest limitation for it?

Raspberry Pi vs VPS

Imagine 2 Raspberry Pis, doing immutable blue/green deployments on it, with a large local of a few TBs!
** README with setup instructions
* Resources
** [[https://github.com/mail-in-a-box/mailinabox][Mail-in-a-Box]]
** [[https://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/][NSA-proof your e-mail in 2 hours]]
** [[https://www.iredmail.org/][iRedMail]]
** [[https://blog.harveydelaney.com/hosting-websites-using-docker-nginx/][Hosting Multiple Websites with SSL using Docker, Nginx and a VPS]]
** [[https://github.com/sovereign/sovereign/][Sovereign]]
** [[https://github.com/nixcloud/nixcloud-webservices][nixcloud-webservices]]
** [[https://github.com/Kickball/awesome-selfhosted#email][Awesome-Selfhosted: Email]]
** [[https://arstechnica.com/information-technology/2014/04/taking-e-mail-back-part-4-the-finale-with-webmail-everything-after/][Taking e-mail back]]
* Decisions
** Use external git repository as an encrypted database
Terraform does have the support for "backends" where it can store =.tfstate= files.

From the list of supported backends, the [[https://www.terraform.io/docs/backends/types/s3.html][S3]] option initially stands out as the simplest to configure. It doesn't however support state locking, only if also configuring DynamoDB.

This extra configuration and complexity isn't attractive, and I can achieve similar outcomes by using the =local= backend and storing it properly. Even better than sending to S3 and setting up the proper revision headers is to just use a separate repository to keep it.

Using the same repository would create an unwanted cyclic process where the repository pipeline commits in itself.

All data stored on git is encrypted with [[https://www.agwa.name/projects/git-crypt/][git-crypt]], which means git isn't being actually used as a source code repository, but as a versioned filesystem database.

By taking advantage of the sourcehut ecosystem, it was easier to setup the access of the pipeline to the ad-hoc Terraform backend.

I created a repository called [[https://git.sr.ht/~euandreh/vps-state/][=vps-state=]] to store the encrypted =.tfstate= and =.tfplan= files. During the CI run, the pipeline creates new a =.tfplan= file and commits it into =vps-state=, and after applying the plan it updates the =.tfstate= file and adds this change to =vps-state=.
** Configuration of =StrictHostKeyChecking=
We have 3 cases where I'm pushing things to the server and I'm dealing with it differently:
*** 1. Pushing updates to the =vps-state= repository
I could whitelist the SSH keys from the =git.sr.ht= servers, but this could break on every key rotation of the server.

In can of the server address being spoofed, the content would be readable by the attacker, since we're doing all the encryption on the client. We would, however, lose a Terraform state file update. As of right now, I'm OK with this trade-off.
*** 2. Running =scp= to the deployed VPS
On this situation I want to be sure I know where I'm pushing to.

In order to avoid adding =StrictHostKeyChecking= when running =ssh= and =scp=, every time the SSH key is rotated I generate a new =./secrets/ssh/known-hosts.txt= file with the proper SSH public key.

This way we can avoid prompting for SSH server fingerprint trust on the CI and avoid adding =StrictHostKeyChecking= on those calls.
*** 3. Backup server
Even though the backup is encrypted before sending the data, I don't want to risk loosing a backup to an spoofed server. I'd rather break the build instead.
** Don't use Ansible as a =local-exec= provisioner from Terraform
Instead, explicitly call =ansible-playbook= after =terraform apply= finished running.

This way we test the DNS A record -> Floating IP -> Droplet IP path. We can't do that inside Terraform declaration because the =local-exec= provisioning command runs before the =digitalocean_floating_ip_assignment= is created, and we can't create a cyclic dependency between the two resources.

We could use the raw Droplet IP instead of the DNS A record, but I prefer calling it later in order to always test the full DNS resolution.
* Scrath