If you run frigate in an lxc container on proxmox you should be able to use any device visible to the host without worrying about pass through

I myself have frigate running on docker running on unraid virtualised under proxmox with a gpu passed through and it works like a dream a