(TL;DR at the end)
I have been rebuilding my lab over the past couple of months, to get finally rid of the last 2 windows server VMs (active directory) and rebuild everything with mostly open source tools.
I was just setting up my main OpenShift cluster when the whole gitops rabbithole finally showed its true depths - so after setting up argo and trying to implement the app-of-apps pattern I realised that my strategy for getting secrets into my cluster needed some more thought. I found “external secrets” to be the best bet for this, maybe adding some SOPS into the mix, where needed.
So now I found myself setting up a vault instance, which brings me here today. Vault and consul should be configured with SSL (as everything should tbh) and I’m not too sure where to get my certs.
I have a domain, that I only use for internal stuff, so I would be able to get valid SSL certs through certbot, but my DNS server is internal only, which brought me to the http-01 challenge. This needs a file on a webserver (under http://example.com/.well-known/acme-challenge/ ) which is easily doable for VMs.
It gets a bit more tricky for cert-manager and Kubernetes. Here, the above URL is provided through the ingress-controller, which is internal only. I could theoretically expose those endpoints to the internet, but I don’t want the extra overhead of configuring all of that. At that point, it’d be easier to just manually generate certs and add them to the endpoints.
On the other hand, I have a freeIPA server running - for DNS and LDAP - that could theoretically also act as a CA, so I could generate my own certs and import the root cert on my devices. That’s nice for desktops and laptops, but with mobile devices, this seems like a lot of tinkering, which - again - I’d like to avoid.
So ultimately the question is: How do you handle SSL certs and ACME in your homelab and all the devices that are connecting to it?
TL;DR: Reduilding lab and rethinking SSL. I have a valid domain, but DNS is internal only. Letsencrypt is not too easy because of internal DNS and cert-manager on kubernetes ingrsses that aren’t connected to the internet - DNS challenge wouldn’t work and HTTP challenge would probably be a lot of work. Running my own CA seems like a lot of hassle with mobile devices. Need advice.
Use vault as a CA and support ACME protocol for Cert manager and is quite easy to use. The documentation is straight forward(https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-acme-caddy). Good luck
Use vault as a CA and support ACME protocol for Cert manager and is quite easy to use. The documentation is straight forward
I have considered this, but this brings me back to same issue as with FreeIPA:
On the other hand, I have a freeIPA server running - for DNS and LDAP - that could theoretically also act as a CA, so I could generate my own certs and import the root cert on my devices. That’s nice for desktops and laptops, but with mobile devices, this seems like a lot of tinkering, which - again - I’d like to avoid.
I’d have to rolly my certs out to all my mobile devices, which doesn’t sound fun at all.
You can import the other CA from ipa in Vault and still use acme protocol. As makes your job easier.
You have FreeIPA already, its CA can act as ACME provider so you can enable and use it: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_certificates_in_idm/deploying-and-managing-the-acme-service-in-idm_managing-certificates-in-idm
I have considered this as the most viable of all my options. But theres still the issue with distributing my root certs to my mobile devices.
On the other hand, I have a freeIPA server running - for DNS and LDAP - that could theoretically also act as a CA, so I could generate my own certs and import the root cert on my devices. That’s nice for desktops and laptops, but with mobile devices, this seems like a lot of tinkering, which - again - I’d like to avoid.
After tinkering for a few days I think I finally found the best comnpromise for my setup.
I’m using my public DNS (at cloudflare) to deploy the DNS challenge txt entries, to then issue certs using cert-manager and certbot.
This is security wise not the best solution because all these DNS entries are public, so mapping my internal infrastructure would be somewhat possible, but since it’s a homelab I’m not too concerned about this attack vector.