Local AI and Tunneling Experimentation

Standing up a local LLM behind a private network on an old workstation GPU, and what it had to say about the word ‘seven.’

The pitch to myself was simple: an old laptop was already sitting around doing light VM-hosting duty, it has a workstation-class GPU, and a GPU is a GPU. Point Ollama at it, run a model end to end on hardware I own — no API key, nothing leaving the house. The GPU had other plans.

It’s a Quadro M2000M, 4GB of VRAM — a real workstation part when the laptop was new, and old enough now that the current CUDA toolchain won’t touch it. Ollama starts, looks for a compatible device, and quietly falls back to CPU. So the inference box is, at present, a Xeon doing the GPU’s job by brute force. It works. “Fast” is not the word for it.

That pushed the model choice down in size rather than up: small, quick-to-respond models for interactive chat, with the larger coder variants held in reserve for anything that can run unattended. A chat frontend sits in front of Ollama so the models are reachable from a browser rather than a terminal, and the whole thing is tunneled out behind a private network to a subdomain, so I can open it from my phone without carrying a VPN client around for the privilege.

The honest test of a setup like this isn’t a benchmark, it’s asking it something trivial and seeing if trivial actually holds.

qwen2.5:3b, asked how many s’s are in “seven.”

There is no s in seven. I know this. The model, running on hardware that cost more in electricity to assemble than it would to replace, does not. It offers three follow-up questions about words with multiple s’s, apparently confident in the premise that got it there.

This is the part of local inference nobody puts in the pitch. The appeal of running your own model is real — no data leaving the house, no provider deciding to deprecate the thing you built a habit around — but appeal and capability are different axes, and a 3-billion-parameter model on a CPU-bound decade-old GPU sits well down the second one. It’s good for exactly what it’s sized for: boilerplate, a quick rewrite, something to bounce a line of code off of at 1am without opening a browser tab to a service that’s logging the query. It is not a reasoning engine, and asking it to count letters is a fair way to find the edge of that boundary fast.

The setup stays. The expectations got recalibrated on contact.