I can’t think of a way to do it with an unmanaged switch because, if I’m understanding correctly, it’s going to be a routing issue.

I think you need a router in the middle so that your entire home subnet has the same gateway. Then, when one of the uplinks goes down you don’t have to change gateways on anything but the primary router.