June 6, 2026

Disposable Inference: Treating LLM Compute Like a Light Switch

Self-hosted inference does not have to mean paying for an idle GPU all month.

I have been playing with the idea of self-hosted LLM inference because the two things I care about most are privacy and flexibility. I do not love the idea of pushing private client context through somebody else's API forever, and I do not want my architecture welded to one model, one provider, or one billing scheme. Cost matters too, obviously, but mostly because bad cost shape limits your options. If the only version of self-hosting is keeping an expensive GPU warm all month, then privacy becomes a luxury feature. I want something more practical than that.

The other issue I've tried to address is burst compute for orchestrated agentic coding. The models that fit comfortably on a laptop are not capable of the kind of multi-step, context-heavy agentic work I am doing. The models that are capable need more compute than a laptop can reasonably provide.

This is getting more relevant as coding assistants move toward metered compute. GitHub Copilot has moved toward usage-based billing, and Anthropic has started separating some Claude CLI and Agent SDK usage from normal Pro and Max plan limits into a usage-credit model. Claude Code itself still shares limits with Pro and Max plans, but the direction is clear enough: serious usage is being carved up, measured, and priced more explicitly. The plans heavy users depend on are not guaranteed to stay stable.

Hosted models are convenient, and that is why people use them. But the cost is not just the token bill. Your prompts go somewhere else. Your context goes somewhere else. Your retention policy becomes partly someone else's retention policy. That may be fine for a lot of workloads, but it is not fine for all of them.

The annoying part is that self-hosting usually swings too far in the other direction. Now you are renting some obscene GPU box, keeping it warm all month, and pretending that a machine sitting idle for 26 days is somehow more principled because at least it is your idle machine. The economics do not actually work out.

So the real question is not "can I run Ollama on a cloud instance?" Of course you can. The better question is: can I make the expensive machine matter as little as possible?

The Shape of It

The architecture in my proof of concept is pretty simple. There is an always-on proxy. That proxy can run on a tiny VPS, a cheap app host, or even my laptop while I am proving the idea out. In my proof of concept I built a status dashboard on a slim frontend, but you could hypothetically run the proxy on something as stripped down as a Lambda function.

The proxy receives the request. If there is already a warm inference node, the request is piped through. If there is not, the proxy provisions one through a provider adapter, waits for it to boot, starts Ollama, makes sure the model is actually available, then forwards the request. After the node sits idle for some configurable amount of time, the proxy destroys it. This is where the model starts to make sense. Sometimes you do not need a private LLM running all month. You need the best self-hosted model you can reasonably get, on a machine big enough to run it, for one intense analysis session that lasts an hour or two.

The GPU machine is not the system. It is an appliance the system turns on and off. The end users should not be aware of any of that, either. To them it should feel like using any other hosted model. In my proof of concept, the proxy speaks Ollama's native API, which means any tool that supports Ollama can use it as a backend. I tested this with Cline, an agentic coding assistant that runs inside VS Code, and it worked as a drop-in replacement. Cline has no idea it is talking to a disposable node that did not exist five minutes ago.

The Proxy Owns the State

The inference node should be stateless. It should not own the user, the conversation, or the memory. Nothing you would care about losing. Again, the compute resource should be treated as an appliance, not a system.

Every request rebuilds the conversation context from the proxy's database. The node gets the prompt, produces tokens, and goes away when it is no longer needed. If it disappears halfway through a conversation, that is a slight annoyance but not catastrophic. The conversation is still in the database. Spin up another node, rebuild the context, keep going.

That boundary changes how you think about the compute node. It stops being a precious server and starts being more like a light bulb. Useful when it is on, replaceable when it burns out, neatly turned off when nobody is in the room.

A Couple Things I Got Wrong First

The first was baking model weights into a container image. Multi-gigabyte image pushes to a registry are brutal, iterating on the image becomes miserable. Docker registries were not designed to be model warehouses, and it works against the whole point of spinning infrastructure up quickly. This way of doing things might be acceptable with a different provider, but for my proof of concept with DigitalOcean, it was unusable.

Volume-backed model storage is less cute, but much better. First boot pulls the model. Later boots find it already there. If you're running a system like this at an organizational level, your business downloads the model once, and you're done.

I also learned that provider choice matters more than I expected. In my proof of concept, a cold start with the model already in block storage still takes three or four minutes to provision and boot. For internal tools or dev workflows that is probably fine. For customer-facing applications, it might not be.

One thing I missed on the first pass is having a daemon for automatic decommissioning on the compute node as a fallback. If the proxy shuts off for some reason or fails to signal the compute node to delete itself, there needs to be redundancy.

Making Privacy Practical

There are three basic cost models here. Hosted APIs are easy. Pay per token. No servers. No boot time. No drama, at least not the infrastructure kind. Always-on compute is the blunt self-hosted version. Rent the machine. Keep it running forever, and pay the bill whether anyone uses it or not.

Disposable compute is the middle path. Pay while the session is active. Destroy the node when it is idle. Keep the state somewhere durable. It is not free. The proxy still costs money. The database costs money. Storage costs money. But the expensive part, the GPU compute, does not sit there all month waiting for someone to ask it a question. The difference is not zero cost. It is zero idle GPU cost. The other really awesome thing is that you can spin up as much model and infra as you need, at will. If you need a monster model and the compute node to match, you can do it ad hoc and only pay for the window where it is actually useful.

The Tradeoff

The primary tradeoff here is latency. Cold starts are not subtle. You are not replacing a hosted API where the first token comes back immediately. Depending on your provider, you may be waiting minutes for the node to exist, boot, install what it needs, attach storage, load the model, and prove it is healthy.

For a production chat product, that may be unacceptable. For internal tools, batch analysis, private client data, dev workflows, or low-frequency expensive inference, it might be completely fine. That being said, you could split the difference again. Keep a warm pool of one worker to handle requests immediately, and still go wide with ad hoc nodes to handle traffic.

This is not about pretending disposable compute is better at everything. It is about understanding the tradeoffs and deciding whether the control boundary is worth the latency cost for your particular use case.

Flexibility Is the Escape Hatch

I do not want to fall in love with a GPU vendor. I want the provider to be boring enough that replacing it is slightly annoying, not existential.

The provider interface should be boring: create a node, destroy a node, get status, attach storage, maybe run a bootstrap script. Nothing interesting should live there.

If the proxy owns the state and the provider is just an adapter, switching vendors becomes a lot less dramatic. DigitalOcean today. AWS tomorrow. Something cheaper or weirder later. The inference logic should not care.

This matters because cloud GPU providers are not interchangeable in practice. Pricing changes, availability changes, and boot times vary wildly. What works today with one provider might not be the best option six months from now.

My next step is to write provider interfaces for AWS and RunPod. AWS lets you sleep instances and not get billed. Cold starts are much faster on starting an instance rather than provisioning one. RunPod is purpose built for this kind of ephemeral compute. No provider makes this free. There are always fixed costs: storage, networking, state, monitoring, and the proxy itself. Idle GPU spend is eliminated, not infrastructure spend altogether. I'd love to do some analysis and see how these different providers stack up with cold starts and traffic stress tests.

From Proxy to Control Plane

Single-user disposable inference is useful enough. Multi-user is where it gets really interesting.

Now the proxy is not just turning one node on and off. It is managing a pool of workers. It needs to know which conversations are pinned to which warm nodes, which users can share capacity, which requests should queue, and when to spill over to another provider.

This starts to look like the landlord pattern in multi-tenant systems. The proxy becomes the landlord. It knows what exists, who belongs where, which workers are warm, and which ones need to be shut down before they run up the bill.

This is where the architecture stops being a wrapper around an inference server and starts becoming a control plane. That is also where it gets interesting, and where I am planning to spend most of my time next.