sounds like an unsanctioned chat bot going through internal private data
Probably better than handing that internal private data to a cloud provider. At least with this setup, it will all stay in the network under their control. There should be no reason to give the inference server access to the internet.
5070 Ti
for the entire 15-20 person firm
Local AI is a great option to look into, but I can’t imagine that’s going to go well… 16GB of VRAM is going to limit you to very small models and small context size. I imagine for a law firm, you’re going to want the AI to be reading lots of documents, so lots of context. Maybe it will be fine depending on how the 15-20 people access it, but I’m doubtful.
Is this machine just a proof of concept to start putting together a process and testing the waters? I wouldn’t call total bullshit immediately, but I’d expect you’ll eventually find that you need much more VRAM and probably a heavier development lift to integrate with n8n.
Then yeah, that’s for sure BS. For getting started and testing a PoC, it is “reasonable” depending on a lot of factors, but there’s almost no way that a 5070 Ti will be a permanent production solution.