1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
|
* Tasks - v4
** TODO Prosody in localhost
** TODO Prosody in NixOS with NGINX
https://www.reddit.com/r/linuxadmin/comments/7z0ahv/prosody_behind_reverse_proxy/dunbdmv/:
#+BEGIN_SRC nginx
server {
listen 80;
listen [::]:80;
server_name xmpp.domain.com
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
ssl on;
ssl_certificate /path/to/cert;
ssl_certificate_key /path/to/key;
location / {
proxy_pass http://192.168.2.101:5222;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
#+END_SRC
** TODO matterbridge
#+BEGIN_SRC nix
matterbridge = {
enable = false;
configFile = ''
'';
};
#+END_SRC
** TODO cgit
** TODO snapshot with swap: nixos-rebuild switch takes too much memory
Adding swap memory maybe be a lot slower, but allows me to stay inside the 512MB memory server plan
** TODO cron: borg, PostgreSQL backup and NixOS update and collect gargabe
** TODO Configure PostgreSQL permissions correctly
** TODO Harden the server
*** TODO [#C] [[https://www.reddit.com/r/selfhosted/comments/bw8hqq/top_3_measures_to_secure_your_virtual_private/][Top 3 measures to secure your Virtual Private Server? (VPS)]]
*** TODO [#A] [[https://docs.nextcloud.com/server/stable/admin_manual/installation/harden_server.html][Nextcloud: Hardening and security guidance]]
*** TODO [#A] [[https://ownyourbits.com/2017/03/25/nextcloud-a-security-analysis/][NextCloud, a security analysis]]
*** TODO [#B] [[https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.md][Check for HSTS header configuration]]
*** TODO Move secrets outside the Nix store
*** TODO Properly configure PostgreSQL
*** TODO Separate lighttpd user ownership from =$GIT_ROOT=?
** TODO Add volume
And put in to fstab
Can I use Terraform to automate the mounting?
#+BEGIN_SRC hcl
variable "storage_name" {
type = string
description = "Name of the block storage volume, which will also be the name of it's mount point."
}
resource "vultr_block_storage" "vps_storage" {
size_gb = 10
region_id = 9
attached_id = vultr_server.vps_server.id
label = var.storage_name
live = "yes"
}
#+END_SRC
** TODO EteSync?
** TODO Run cgit from nginx instead of using lighttpd
* Services - v2
** TODO =cloud.$tld=: Nextcloud: storage, calendar, contacts, notes and talk
** TODO =chat.$tld=: [[https://conversejs.org/][Converse]] interface to Prosody server
Setup under Prosody server, and bridge it with Telegram and WhatsApp using matterbridge.
** TODO =git.$tld=: cgit server with repositories from ~/dev/libre/
** TODO =mail.$tld=: postfix, dovecot, spamassasin, opendkim, etc
No need for roundcube, Nextcloud has a web interface client.
** TODO =$tld=: current Jekyll blog
** Nice to have
*** TODO =audio.$tld=: FunkWhale
* Resources
** [[https://github.com/mail-in-a-box/mailinabox][Mail-in-a-Box]]
** [[https://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/][NSA-proof your e-mail in 2 hours]]
** [[https://www.iredmail.org/][iRedMail]]
** [[https://blog.harveydelaney.com/hosting-websites-using-docker-nginx/][Hosting Multiple Websites with SSL using Docker, Nginx and a VPS]]
** [[https://github.com/sovereign/sovereign/][Sovereign]]
** [[https://github.com/nixcloud/nixcloud-webservices][nixcloud-webservices]]
** [[https://github.com/Kickball/awesome-selfhosted#email][Awesome-Selfhosted: Email]]
** [[https://arstechnica.com/information-technology/2014/04/taking-e-mail-back-part-4-the-finale-with-webmail-everything-after/][Taking e-mail back]]
** [[https://jacobneplokh.com/how-to-setup-nextcloud-on-nixos][How to Setup Nextcloud on NixOS]]
** [[https://www.vultr.com/docs/install-nixos-on-vultr][Install NixOS on Vultr]]
** [[https://www.linode.com/docs/tools-reference/custom-kernels-distros/install-nixos-on-linode/][Install and Configure NixOS on a Linode]]
* Decisions
** Use external git repository as an encrypted database
Terraform does have the support for "backends" where it can store =.tfstate= files.
From the list of supported backends, the [[https://www.terraform.io/docs/backends/types/s3.html][S3]] option initially stands out as the simplest to configure. It doesn't however support state locking, only if also configuring DynamoDB.
This extra configuration and complexity isn't attractive, and I can achieve similar outcomes by using the =local= backend and storing it properly. Even better than sending to S3 and setting up the proper revision headers is to just use a separate repository to keep it.
Using the same repository would create an unwanted cyclic process where the repository pipeline commits in itself.
All data stored on git is encrypted with [[https://www.agwa.name/projects/git-crypt/][git-crypt]], which means git isn't being actually used as a source code repository, but as a versioned filesystem database.
By taking advantage of the sourcehut ecosystem, it was easier to setup the access of the pipeline to the ad-hoc Terraform backend.
I created a repository called [[https://git.sr.ht/~euandreh/vps-state/][=vps-state=]] to store the encrypted =.tfstate= and =.tfplan= files. During the CI run, the pipeline creates new a =.tfplan= file and commits it into =vps-state=, and after applying the plan it updates the =.tfstate= file and adds this change to =vps-state=.
** Move external =vps-state= into =vps=
I want to move the Terraform state into this repository, I don't like the inconsistency that having 2 repos create.
If I move vps-state into vps, I'll have to remove the terraform steps from the pipeline: it can't commit to itself (as described above). This has an upside too: instead of automatically approving whatever plan Terraform creates, I review the plan before applying.
It makes the deploying less automatic, but this removes the IP reputation email issue.
This means that the Terraform provisioning should stay out of the CI and be run only locally.
** Run locally instead of on CI
It makes it less automagic, but greatly simplifies the configuration, like removing custom =ssh.env.conf=, =mail.sh=, =vps-box-client.pub=, etc.
** Configuration of =StrictHostKeyChecking=
We have 3 cases where I'm pushing things to the server and I'm dealing with it differently:
*** 1. Pushing updates to the =vps-state= repository
I could whitelist the SSH keys from the =git.sr.ht= servers, but this could break on every key rotation of the server.
In can of the server address being spoofed, the content would be readable by the attacker, since we're doing all the encryption on the client. We would, however, lose a Terraform state file update. As of right now, I'm OK with this trade-off.
*** 2. Running =scp= to the deployed VPS
On this situation I want to be sure I know where I'm pushing to.
In order to avoid adding =StrictHostKeyChecking= when running =ssh= and =scp=, every time the SSH key is rotated I generate a new =./secrets/ssh/known-hosts.txt= file with the proper SSH public key.
This way we can avoid prompting for SSH server fingerprint trust on the CI and avoid adding =StrictHostKeyChecking= on those calls.
*** 3. Backup server
Even though the backup is encrypted before sending the data, I don't want to risk loosing a backup to an spoofed server. I'd rather break the build instead.
** Don't use Ansible as a =local-exec= provisioner from Terraform
Instead, explicitly call =ansible-playbook= after =terraform apply= finished running.
This way we test the DNS A record -> Floating IP -> Droplet IP path. We can't do that inside Terraform declaration because the =local-exec= provisioning command runs before the =digitalocean_floating_ip_assignment= is created, and we can't create a cyclic dependency between the two resources.
We could use the raw Droplet IP instead of the DNS A record, but I prefer calling it later in order to always test the full DNS resolution.
* Questions
** How to best handle IP changes when the server changes? How does this affect the email sending IP reputation?
* Scrath
# ssl = {
# cert = "/var/lib/acme/${envsubstConfiguration.TLD}/fullchain.pem";
# key = "/var/lib/acme/${envsubstConfiguration.TLD}/key.pem";
# };
# virtualHosts = {
# "${envsubstConfiguration.prosodyTLD}" = {
# enabled = true;
# domain = "${envsubstConfiguration.prosodyTLD}";
# ssl = {
# cert = "/var/lib/acme/${envsubstConfiguration.TLD}/fullchain.pem";
# key = "/var/lib/acme/${envsubstConfiguration.TLD}/key.pem";
# };
# };
# };
# muc = [{
# domain = envsubstConfiguration.prosodyMUCTLD;
# }];
# uploadHttp = {
# domain = envsubstConfiguration.prosodyHTTPUploadTLD;
# };
|
