Hi everyone! I’ve got a question about some guys at work who are pitching that we need a server. This is mostly a sanity check, but I’m pretty sure they’re full of shit
The law firm I work for uses a CRM that doesn’t have an open API. This is important. You can access a daily updated copy of your data from an S3 bucket with an API, but it’s essentially read-only for that reason.
They said they want to set up a server for the firm for " automation" using n8n. We can’t use n8n because it doesn’t have a native integration with our CRM. They also set up Dropbox for the firm (prior to my time starting) and the firm pays for OneDrive with Microsoft suite already…
Then they said they want local ai for automation so they’ll
“Just put a 5070ti in there” For an ai agent for the entire 15-20 person firm. They also never specified what said ai would do. I also think it’s completely not viable
Then they said all 3 locations can just use tailscale to access the server simultaneously. All of these people minus me and one other are completely non-technical. I help them restart Excel once a week non-technical.
I cannot possibly think of a viable use case for what they’re describing. Am I cinical or are they just looking to make some cash off whatever project they don’t know anything about?
So to play the devils advocate, this really comes down to:
- Are these internal people or contractors
- Are they looking to test out a Workflow or implement small automations
If they’re contractors, always be suspicious. If they’re internal, it sounds like they may want a server so it’s a one off cost to setup some work flows. You can burn through hundreds to thousands of dollars per month using LLM APIs, but if you self host you’ve basically got a fixed cost.
If they’re just looking to create some work flows that crawl the data dumps and give some insights, this approach could work. Although, as others mentioned, the hardware they’re talking about would be somewhat limited. Small LLMs can be quite functional, but this server might not be a long term fix if the automations they set up gain traction.
Everything about this story sounds wrong. If they are a Microsoft shop they would probably be better off waiting for M365 E7.
Thank you! Will look into this
Looks like someone just wants to put AI on their resume then bounce out. Being a legal firm I’d also be concerned what vulnerabilities these CV builders would be opening just to have what sounds like an unsanctioned chat bot going through internal private data.
I think it could be even more nefarious. If the firm has 3 locations, they must have some huge clients.
They don’t have huge clients, just a steady flow. 3 locations due to small 5-7 person groups per office instead of one big office. Tbh idk why they do it like that lol
sounds like an unsanctioned chat bot going through internal private data
Probably better than handing that internal private data to a cloud provider. At least with this setup, it will all stay in the network under their control. There should be no reason to give the inference server access to the internet.
From what OP is describing, it doesn’t sound like anyone asked or was looking for this functionality.
Kinda, kinda not? The lawyer who owns the firm really wants new Ai features and asks about stuff all the time, but he doesn’t necessarily have a specific goal, rather, just gives times to create automations with reports or whatever
5070 Ti
for the entire 15-20 person firm
Local AI is a great option to look into, but I can’t imagine that’s going to go well… 16GB of VRAM is going to limit you to very small models and small context size. I imagine for a law firm, you’re going to want the AI to be reading lots of documents, so lots of context. Maybe it will be fine depending on how the 15-20 people access it, but I’m doubtful.
Is this machine just a proof of concept to start putting together a process and testing the waters? I wouldn’t call total bullshit immediately, but I’d expect you’ll eventually find that you need much more VRAM and probably a heavier development lift to integrate with n8n.
That’s what I was thinking, they were saying that a 5070 Ti would be good enough for “whatever Ai automations the firm wants to build” which is where I was calling BS
Then yeah, that’s for sure BS. For getting started and testing a PoC, it is “reasonable” depending on a lot of factors, but there’s almost no way that a 5070 Ti will be a permanent production solution.
If it’s for a single low frequency workflow that GPU is enough, but you will be limited to small models which are mostly useless unless fine-tuned for your use case. If it’s serving users for the entire company with a big enough model to be useful you would need 192-384GB of VRAM. So a server between $20k and $40k.
The server will require maintenance, and somebody will have to develop the workflow and integration with your data.
It’s also important to know what they want to do, a basic embedding model for semantic search would work, agents not so much.
You’re clearly an expert.
This is a scam and you know it.
The first joke was running an LLM for a company on a 5070ti… That’s nowhere near enough VRAM. Even two 3090s linked together would have been way more plausible.
I’m not an expert at all, I’ve never built out an enterprise system before so that’s why I wanted to verify.
I know you can run really small models on some 16gb VRAM cards, so I didn’t know if people use tiny models for very basic automation systems or something. I haven’t heard of that being viable but wanted to sanity check myself
In what way are you getting ripped off? What’s your role in this? Are you paying these people money out of your pocket? Then yes. Are you supposed to approve their expenses for the business? Then also yes. So, yes.
I’d demand a full business plan and technical design for review and approval with stakeholders. Are you going to buy this hardware and they will thumb with it for 5 months with zero product for the firm? What’s the expected outcome and value to the business? I’d get more eyes on this and ask tough questions, and get it documented. Maybe let them use a public AI first to make the design doc, and then stakeholders can use a public AI to review said doc, and then you all can confuse each other and shelve this project to Neverland.
The lawyer who owns the firm has me approve all tech vendors. I have not built out my personal homelab, know l some jargon and theory, but don’t have practical experience here.
Saw red flags and wanted to verify cause it’s part of my role
Then I would absolutely share your concerns. If you can’t share the value proposition clearly with the owner in your own words, then it will reflect poorly on you when this goes sideways with negative results. IMO you don’t even need to say no. Just say, “there’s no way I can explain this clearly to <owner> with what you’ve told me so far. I can’t sell a plan I don’t understand and I don’t have enough details to understand the steps here. Let’s write this down in a document that we can go back and forth on to get the details right that I can share with others.” Then make some suggestions for them to do the leg work.
Before any hardware or software gets built or acquired it can be helpful to go through some exercises. Probably the most helpful here:
-
Business Requirements Document (BRD) - stakeholders, business constraints, goals. Should sell the idea “we need to do something here because it’s important to the business for XYZ reasons”.
-
Product Requirements Document (PRD) - Critical User Journies (CUJ), user stories, use case analysis. Should iron out how a solution should look like to achieve the stated goals of the business doc. Specify success criteria, what metrics are important for this to be a success “we saved staff X hours, we cut costs Y dollars, we brought in Z new clients”.
-
Technical Specification or Technical Design Doc (TDD). Should explain how to achieve CUJs through a technical design. Focus should be both “why” to do it this way AND “why not” do it some other way. Tradeoff analysis.
Then there are a dozen related drill-downs exercises: legal review, security/privacy/compliance review, internationalization, launch review, UX review, on and on. They all serve a common goal: get everyone on the same page. You don’t need to contort what your are doing into any of the above, but it’s just kind of evolved into these aspects because it’s so difficult to get everyone on the same page.
Again, I’m not saying adopt all of the above, but if you don’t have the technical knowledge then your role is closer to a project manager and that is its own set of skills. It’s very helpful when PMs have some technical background (so absolutely continue enjoying Homelab hobbies), but (from a random internet stranger’s POV) here you need to wear a project manager hat.
Good luck!
-
This does seem tenuous, yes.
If they want to experiment with AI a bit and try to find a use-case for it, a bit of R&D every now and then can be good, right? But the way to do that would be to temporarily rent a VPS with a GPU. From AWS or whatever. That way when/if it looks good they’ll know roughly what size hardware to buy (way more than a 5070 ti most likely). Also if it doesn’t pan out all you’ve lost is a couple of hundred $ on one month’s rental and you can close it down and move on.
Or just use the OpenAI API, that way you don’t need to figure out how to run a model at all and can just concentrate on the data integration to see if that’s viable.
That’s what I was thinking too, I’m pretty sure just using the API for those models or a VM with a GPU like you said would be the only viable options
