1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
* Tasks
** DONE Provision DigitalOcean's droplet from Terraform
CLOSED: [2019-05-25 Sat 13:29]
** DONE Properly provision Ubuntu droplet
CLOSED: [2019-05-25 Sat 17:50]
** DOING Automate deployment of updates
*** DOING Subtasks
**** DONE Fix Debian import of GPG keys
CLOSED: [2019-05-26 Sun 14:34]
While NixOS image isn't fixed, use Debian instead.
The GPG data was all in a single line. I copied and pasted it properly and it was identified correctly.
**** CANCELLED Properly install Nix on Debian image
NixOS patch was applied.
**** DONE Fix NixOS GPG key importing in builds.sr.ht
CLOSED: [2019-05-26 Sun 17:37]
See patch and discussion in [[https://lists.sr.ht/~sircmpwn/sr.ht-dev/%3C20190526162135.1646-1-eu%40euandre.org%3E][sr.ht-dev mailing list]].
**** DONE Use ssh configuration from environment instead of creating and alias for =ssh=
CLOSED: [2019-05-26 Sun 19:44]
***** DONE Relative =IdentityFile= path
CLOSED: [2019-05-26 Sun 19:42]
Used =envsubst= to properly interpolate variables in =ssh.conf=
***** DONE Omit =-F ssh.conf= from command
CLOSED: [2019-05-26 Sun 19:42]
Put it in an environment variable?
Done by appending to content of =~/.ssh/config=.
**** DOING Automate deployment with Terraform and deployment scripts
**** NEXT Use DigitalOcean's Floating IP in front of the droplet
**** DONE Backup data during deployments
CLOSED: [2019-05-28 Tue 00:48]
Is this approach feasible? Will it make the deployment take too much longer? What are the alternatives?
Initial sketch of the backup commands:
#+BEGIN_SOURCE shell
rsync --verbose --progress --stats --update --recursive "$HOME/backups/" "$RSYNC_REMOTE"
borg create -svp -C lzma,6 "~/borgbackup::{hostname}-{now}-${VPS_COMMIT_SHA} ${VOLUME_HOME}"
rsync --verbose --progress --stats --update --recursive "$RSYNC_REMOTE" "$HOME/borgbackups/"
#+END_SOURCE
Implemented with help from https://jstaf.github.io/2018/03/12/backups-with-borg-rsync.html
**** TODO Test provisioning DNS entries with other DNS registrars
**** DONE Namecheap whitelist IP limitation
CLOSED: [2019-05-26 Sun 17:14]
Namecheap requires you to specifically whitelist an IP that can perform changes to their API.
[[https://lists.sr.ht/~sircmpwn/sr.ht-discuss/%20%3CCAJk2QMbq8uE1pcG3Uy6w37HUY7W15cQ+sHoj-UBWN-W11AtcrA%40mail.gmail.com%3E][builds.sr.ht]] don't guarantee any specific IP, so whitelisting it isn't an option.
The best candidate so far is using DigitalOcean's Floating IP feature to link a hardcoded IP to a droplet, while the droplet's IP may change. This way any new deployment wouldn't change the public IP of the box, and wouldn't require me to change the DNS A and AAAA records on Namecheap.
This has also an advantage of allowing the email server to keep it's IP address.
The downside is that the deployment of DNS registries isn't fully automated: whenever I have to change a DNS entry, either for adding a new CNAME record or changing an AAAA record, I'll have to:
1- get my own IP;
2- whitelist it on Namecheap's web interface;
3- run a separate Terraform recipe.
The upside is that this should happen less often than a deployment, but still not ideal. The ideal would be to run the Terraform recipe every time, and Terraform would realize that there was no DNS related change and do nothing.
*** Limitations
During build, decrypt content of files and update the deployment.
How does Terraform tfstate file can be handled in this case?
UPDATE:
Terraform does support the so called "backends" to coordinate lock and usage of the =.tfstate= files. On this regard there are no restrictions on continously deploying with Terraform from the CI pipelines.
However the current applications do *not* properly support blue/green deployment, like email, Nextcloud, etc.
We could try to share a shared volume, but that would be a consistency nightmare.
The other option is to always recreate everything, with downtime. The advantage is that we get actual immutable deployments with stateful storage, but there would be downtime for every deployment. This is due to the nature of most of the packaged applications being single node *only*.
There's also the IP reputation issue: recreating everything from scratch every time would lead to new droplets with new IP addresses, which is not a good thing to be changing in a server box.
A reasonable alternative would be to redeploy everything on a different node, with a different TLD, and manually check that. But that would be just like an staging environment, with all of it's downsides too.
In this situation, I if go on with automating the deployment I'd rather pick the downtime option.
I'll start with other services other than email and consider alternatives later.
** TODO Use Digital Ocean's Volumes for persistent extended storage
** WAITING Configure DNS from Terraform
* Must
** Fully deployable from code
Use NixOps and Terraform to fully automate all of the configuration.
* Services
** DONE =euandreh.org=: Static webhosting
CLOSED: [2019-05-26 Sun 10:17]
** DONE =wallabag.euandreh.org=: Wallabag
CLOSED: [2019-05-25 Sat 18:02]
A bookmark application must:
- allow me to save and read articles on Android and Firefox;
- allow me to have tags for links;
** NEXT =cloud.euandreh.org=: Nextcloud: storage, calendar, contacts, notes
https://github.com/nextcloud/docker
Do I need to configure =NEXTCLOUD_TRUSTED_DOMAINS= or it should work without it?
** TODO =mail.euandreh.org=: Email + webmail
https://github.com/tomav/docker-mailserver
https://mailu.io/master/demo.html
https://mailcow.email/
https://poste.io/
https://github.com/hardware/mailserver
=mail.euandreh.org= could be the Nextcloud mail application!
** TODO =www.euandreh.org= and =blog.euandreh.org=: Redirect to =euandreh.org=
** TODO =hydra.euandreh.org=: Hydra
Does Hydra support release management? The source tarball can live in git.sr.ht, but what about compiled outputs?
I'd like to release both pre-compiled binaries and Docker images.
** TODO =annex.euandreh.org=: Public content from Git Annex repositories
Only an static file server, with folders for individual assets.
** TODO =pi-hole.euandreh.org=: Pi-hole
** TODO =pwk.euandreh.org=: Piwik
** TODO =git.euandreh.org=: CGit or GitWeb
https://github.com/iconoeugen/docker-gitweb
** TODO =songbooks.euandreh.org=: Songbooks demo application
** TODO =pires.euandreh.org=: Pires deployment
** CANCELLED =perkeep.euandreh.org=: Perkeep
I'm already covered by using Git Annex for almost everything.
** WAITING =matrix.euandreh.org=: Matrix Synapse server
I'm not using IRC a lot right now. Wait for me to interact more with mailing lists and gauge the need of IRC.
* Questions
** TODO Critiques of Docker?
What does NixOps, DisNix and Dysnomia are trying to accomplish that overlap with Docker?
Do they do a better job? Why? Why not?
Get a book on advanced Docker, or container fundamentals and dig deeper.
** DONE Do I want or need Docker? Should I use it?
CLOSED: [2019-05-25 Sat 18:1980]
It was a better path than sticking with NixOps and nixcloud-webservices. It's more widespread and has more things done for it.
** CANCELLED How to share the Nix store across services?
** DONE How to leverage DigitalOcean's block storage?
CLOSED: [2019-05-25 Sat 18:19]
Provision it using Terraform, and use it's path as the =$VOLUME_HOME= variable for containers.
This was I can compartimentalize the data storage to easily backup and duplicate, but also destroy a running droplet and create a new one.
* Nice to have
** =*.euandreh.org=
** Nix Terraform provisioning
** Upgrade =docker-compose.yaml= file from version 2 to version 3
* Resources
** [[https://github.com/mail-in-a-box/mailinabox][Mail-in-a-Box]]
** [[https://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/][NSA-proof your e-mail in 2 hours]]
** [[https://www.iredmail.org/][iRedMail]]
** [[https://blog.harveydelaney.com/hosting-websites-using-docker-nginx/][Hosting Multiple Websites with SSL using Docker, Nginx and a VPS]]
** [[https://github.com/sovereign/sovereign/][Sovereign]]
** [[https://github.com/nixcloud/nixcloud-webservices][nixcloud-webservices]]
** [[https://github.com/Kickball/awesome-selfhosted#email][Awesome-Selfhosted: Email]]
* Decisions
** Use external git repository as an encrypted database
Terraform does have the support for "backends" where it can store =.tfstate= files.
From the list of supported backends, the [[https://www.terraform.io/docs/backends/types/s3.html][S3]] option initially stands out as the simplest to configure. It doesn't however support state locking, only if also configuring DynamoDB.
This extra configuration and complexity isn't attractive, and we can achieve similar outcomes by using the =local= backend and storing it properly. Even better than sending to S3 and setting up the proper revision headers is to just use a separate repository to keep it.
Using the same repository would create an unwanted cyclic process where the repository pipeline commits in itself.
All data stored on git is encrypted with [[https://www.agwa.name/projects/git-crypt/][git-crypt]], which means git isn't being actually used as a source code repository, but as a versioned filesystem database.
|
