Hey y’all, looking to land my first DevOps Engineering role soon, and figured I should use enterprise software as much as possible for some resume building and personal practice. For reference, I’ve set up a NAS server once before but haven’t got too much experience outside of that. Basing this on some DevOps Engineers I’ve talked to IRL and some friends who hire engineers, but wanted extra community feedback.
Use case: parents are data hoarders, probably have at least 4tb saved composed of every type of media you can think of, so hopefully the whole family can use this when I’m done with it all. Otherwise, aiming to be able to claim experience with enterprise grade DevOps software.
Some of this is personal research, a lot of Reddit research, and some LLM comparisons used to choose between two software systems. Please let me know what you’d keep or change! I’m still kinda new to this :p
Hardware: (old gaming pc)
- Intel i5-9600K
- 32GB DDR4 RAM
- GTX 1070
- Gigabyte Z370XP SLI
- Seagate IronWolf 12TB 3.5" SATA
Hypervisor & OS:
- Proxmox VE (type-1 hypervisor)
- Ubuntu Server 24.04 LTS (VM operating system)
- cloud-init (VM provisioning automation)
Infrastructure as Code & Automation:
- Terraform (infrastructure provisioning)
- Proxmox Terraform Provider (VM automation)
- Ansible (configuration management)
- GitHub Actions (CI/CD pipelines)
Containerization & Orchestration:
- Docker (container runtime/builds)
- Kubernetes/k3s (container orchestration)
- Helm (Kubernetes package manager)
- ArgoCD (GitOps continuous deployment)
Networking & Ingress:
- Traefik (ingress controller/reverse proxy)
- MetalLB (bare-metal load balancer)
- cert-manager (TLS certificate automation)
- WireGuard (VPN software)
- Surfshark (VPN service)
Secrets & Security:
- HashiCorp Vault (secrets management)
- External Secrets Operator (Kubernetes secret syncing)
- SSH hardening (secure remote access)
Observability & Monitoring:
- Prometheus (metrics collection)
- Grafana (monitoring dashboards/visualization)
- Loki (centralized log aggregation)
- Promtail (log shipping agent)
- Alertmanager (alert routing/notifications)
Storage & Backups:
- ZFS (filesystem/storage management)
- NFS (network storage)
- Persistent Volumes/PVCs (Kubernetes storage)
- Restic (encrypted backups)
- Velero (Kubernetes backup/disaster recovery)
Container Registry & CI Infrastructure:
- GitHub Container Registry or Harbor (container registry)
- GitHub Runner (self-hosted CI runner)
AWS Emulation:
- LocalStack (AWS cloud emulation)
- Terraform AWS Provider (AWS IaC practice)
- MinIO (S3-compatible object storage)
Self-Hosted Applications:
- Prowlarr (indexer manager)
- Sonarr (TV show management automation)
- Radarr (movie management automation)
- LazyLibrarian (book management automation)
- Lidarr (music management automation)
- Homarr (application dashboard)
- Seerr/Overseerr (media request management)
- Jellyfin (media server)
- qBittorrent (torrent client)
- NZBGet (Usenet downloader)
- Immich (photo gallery & backup)
- Mealie (meal planner)
- Moonlight (low-latency remote gaming)
- Kavita (ebook/manga/audiobook reader)
- Funkwhale (music streaming)
- Grafana (monitoring dashboards)
Allright, here is my thoughts :)
All tech you list is relevant, but if you’ve never used any of it before, maybe pick out a small subset and learn that to begin with instead. It is exciting and I want to be an expert on all of these aswell, but it is really a lot. Better to be an expert on 3 things than to kind of understand what 30 different things do. When you get comfortable with some basics like for example Ubuntu server and Docker, you can then move on to something else depending on what your priorities are.
On the topic of hosting stuff for family and friends; Really make sure you follow best practises for backups. 3 copies on 2 different devices and 1 copy must be off-site. If you only play around, backups are optional, but as soon as someones memories, important documents or similar relies on your stuff, you better have proper backups.
What are your current status? What do you know today? :)
I agree but running headfirst into this wall was a learning experience for me, even if I did fail miserably.
Good points! I’ve set up an Ubuntu server before, didn’t have RAID but passed through my drives and set up Jellyfin and Nextcloud storage in Docker containers using Portainer. Ended up nuking it to go for Proxmox running Ubuntu server + TrueNAS, but wanted to figure out if I could just as easily run full enterprise grade stuff.
I think I’d be capable enough to struggle my way through a working setup, and there seems to be fewer YT tutorials and stuff for this specific use case, but I still believe there’s enough documentation for each piece that I’d be able to get them connected in time.
Also, not to be too dependent on others, but it seems like between this and r/homelab, there’s a pretty good number of people who’d be able to help me find my way should there be one or two things that I just can’t get figured out.
Okay long post so I’m probably gonna make multiple comments:
GPU usage: you have moonlight, and also jellyfin. Both of those are gonna want your gpu.
You are gonna have to figure out hoa to do this. I suspect if you are using k3s, the easiest way is for one (virtual machine) node of the k3s cluster to have the gpu passed through to it and activated, and k3s will assign containers that need the gpu to that host. Although I’m not sure jf moonlight itself runs in a container.
It technically possible to share a gpu between vms (https://github.com/DualCoder/vgpu_unlock) but it is a hack and it is a lot of trouble. It is very easy to share a gpu between containers. You can also use proxmox’s LXC containers, and share a gpu between them, but there are caveats about running (k3s/docker) containers in containers.
Although I’m not sure jf moonlight itself runs in a container.
Pretty sure I can just pass it through when active, so I should be set and not have to split (I believe)
Minio is now closed source. Check out garage, seaweadfs or rustfs as alternatives.
RE: Traefik
My recommendation is to avoid using any of the Traefik custom CRD’s to avoid vendor lock in. I use Traefik strictly as an Ingress Controller (and I don’t bother to use Gateway API, which is more complex. Kubernetes Ingresses still work and will be maintained).
Maybe look into self-hostable but enterprise grade identity provider services such as KeyCloak or Authentik, and learn how to integrate that into your home lab applications for SSO functionality.
I’ll probably go with KeyCloak since it seems to be the standard enterprise solution, and it’s a Red Hat developed tool so +1 for FOSS. Having SAML experience should also be a good resume boost for me
https://integrations.goauthentik.io/media/jellyfin/
The jellyfin sso plugin linked there is now archived and dead.
Authentik supports ldap keycloak does not.
Authentik also has proxy based authentication, to authwall any app that doesn’t support external sso. People use that with jellyfin a lot.
Also SAML sucks, oidc is way better and more popular nowadays.
“Log in with google/github/etc” is actually OIDC (I’ve had people think it was saml before so I am clarifying.
Most of the apps you mentioned will only support oidc and not saml.
No don’t use keycloak it sucks. Trust me authentik is way better (also FOSS + under the Cloud native computing foundation).
You can deploy keycloak as a test, but authentik has way more features, like it supports being ldap server.
Ahhh I get you. Seems like KeyCloak is great if you’re all in on Red Hat systems, but that Authentik would be a better choice for a wider suite of integrations.
For secrets and security, hashicorp vault and ESO are a pain. It’s probably easier to encrypt them directly in the git repo and get argocd to decrypt them (this is what I do with fluxcd).
I really want to use HashiCorp Vault for the resume experience, several companies I know use lots of HashiCorp products so while I’m sure it’s clunkier I think it’ll be worth the learning curve
ESO is worth knowing if you’re going into an enterprise environment. It’s not difficult. It will also teach you something about dependency management (or lack thereof) in k8s :)
Sweet! I’ll make sure I throw that in the mix, having a sync later for k8s would be so nice. I think I’m going to run k3s for now, since I don’t have a database style rack, though.
Re: storage/backups:
You need to figure out a persistentvolume class/provider that you can use.
I recommend Longhorn.
Are you going to use both docker and k3s?
You could only use k3s, but if you are using both docker and k3s see my gpu notes below.
If you are using docker, check out https://komo.do/ to gitops it.
I may be misunderstanding but aren’t k3s a container orchestrator? as in docker containers managed via k3s? So I’d have to use both, and probably eventually set up a shared GPU. For the time being, I’ll likely just run without Moonlight until I feel comfortable getting another card or splitting it
No. K3s uses a different container rundime, containerd as a backend.
If you set up k3s you don’t need docker. Though I suspect you’ll want it fir apps that have better support for docker (offer docker compose but no official helm, more docs, more forum posts, etc).
Gotcha! I misunderstood, but that makes sense. I believe everything I listed is k8s/k3s compatible with containerd, so hopefully it’s pretty straightforward. I’ll stick that instead of docker for enterprise experience
I’m sure you’ve thought of it, but I would make sure to add sensible backups in order to secure the important data.
I’ll run a 3-2-1 in time! for now, just getting server #1 set up is my starting point. I have a location for my offsite backup, too, so I think I’m set
You probably don’t need to host your own container registry or even CI, it’s more trouble than it’s worth for a homelab.
You can deploy them as learning experiences but I would not recommend having any part of you lab be dependent on them, it’s simpler to just use public container images/registries.
That’s more for being able to say I have professional CI/CD experience, I was told that’s crucial for landing a DevOps Engineering role