+1, if you really don’t need much processing power there’s always the HP T630 thin client. On a good day you can get one of those for <$60 shipped with its power brick. Just add storage and you’re good to go. I’ve run Debian and Proxmox on these before to act as cheap docker or LXC hosts.
You’re probably better off defining storage mounts for your containers’ state data and then backing that up with a more standard tool.