Self-Hosted LLMs: Choose Hardware, Keep Cloud Where It Helps

Decide which job deserves its own model

The common belief is that downloading model weights turns an ordinary server into a replacement for a cloud coding assistant. It does not hold because generating text is only part of the job: finding relevant files and applying safe edits also matter. Start by moving a bounded task to local inference, then judge whether the complete workflow deserves a permanent server.

Good initial candidates have clear inputs and outputs you can check cheaply. Extracting fields from internal documents or assigning support tickets to known categories gives you something concrete to evaluate. Asking a model to maintain an unfamiliar application without supervision leaves much more room for plausible mistakes.

We recommend Claude Code for day-to-day coding, with production access bounded separately from its ability to edit a repository. Treat a local model as a separate service until it earns a defined role. A familiar chat interface does not establish that it can complete the same work with the same amount of human correction.

Write down the acceptance condition before selecting a model. For document extraction, that might mean valid output with every extracted value traceable to the source. For code changes, require relevant checks to pass and review the resulting behavior, rather than accepting a confident explanation.

Choose between local inference and a cloud service

Self hosted describes who operates the inference service, while local describes where computation happens relative to you or your application. A rented VPS can run self hosted inference, but the hosting provider still operates the underlying infrastructure. If your requirement is that documents never leave your premises, renting a remote server does not meet it.

Owning hardware creates an ongoing commitment even when the model is idle. A hosted service moves hardware operation elsewhere, but leaves you responsible for access control and deciding what data may be sent. Compare the effort required to obtain an accepted result, including review and retries, rather than comparing inference alone.

For the rest of this guide, the default is Ubuntu 24.04 LTS on a single VPS dedicated to inference. That keeps service management familiar and avoids sharing memory pressure with your production database. Use the VPS, cloud and dedicated server comparison when deciding whether that deployment shape fits your workload.

A hybrid arrangement needs an explicit failure policy. Sensitive requests must fail closed when the local service is unavailable, rather than silently retrying through a hosted provider. Allow cloud fallback only for request classes whose data handling rules permit it.

Size hardware around memory and waiting time

Choose a candidate model and its intended workload before choosing a machine. Parameter count alone is an incomplete sizing guide because representation and runtime behavior also affect memory use. A model file fitting on disk tells you little about whether the service will remain usable under your actual requests.

Quantization stores model weights with reduced precision, lowering their memory requirement while potentially changing output quality. The useful choice is the representation that fits your hardware and still passes your task checks. Do not assume the smallest available download is suitable for code changes or exact extraction.

Budget memory for model weights plus working space and the context cache used during inference. Longer inputs and simultaneous requests can increase demand substantially, so leave headroom instead of filling available memory with weights. Under memory pressure, a service may reject requests or become slow enough that callers retry and make the queue worse.

A CPU VPS is a reasonable place to establish correctness for small, delay-tolerant jobs. Do not assume it will provide comfortable interactive coding simply because inference starts successfully. Renting a GPU machine becomes worth considering when waiting time blocks a workload that has already passed its quality checks.

Before renting, check the exact device and driver combination against the runtime's GPU support documentation. Confirm that the provider exposes a supported accelerator to your guest; a larger CPU plan does not imply GPU access. The hosting shortlist for AI agents can help frame provider selection, but the chosen plan must explicitly meet your inference requirements.

Pick a runtime you can comfortably operate

The runtime loads the model and serves requests; it does not supply the model's abilities. Start with the runtime whose configuration you can explain during an outage. Adding several serving stacks to an early experiment makes it harder to identify whether a failure comes from the model or its deployment.

We would start a modest internal deployment with Ollama, then change runtimes only to solve a specific limitation. Choose llama.cpp when direct control over model artifacts and execution matters. Consider vLLM when serving concurrent requests becomes a central requirement and you can support the resulting configuration.

Choose the model artifact with equal care: verify its origin and read its license before making it part of an application. Permission to download weights does not establish permission for every intended use. Record the artifact identity and your runtime configuration so a later model replacement is an intentional change.

A compatible API shape also needs application-level verification. Check structured output and cancellation behavior if your client depends on them. Do not redirect Claude Code to an arbitrary endpoint and assume that matching a request format reproduces its supported coding workflow.

Keep inference private before connecting clients

Prepare the inference VPS with normal access controls before loading private documents onto it. The first-hour Ubuntu server procedure is the right prerequisite if the machine is new. Run inference under a dedicated service account that cannot read unrelated application secrets.

For Ollama, retain the loopback listener and use its local-only mode when cloud features are outside your data policy. Its configuration FAQ documents listener settings, cloud disabling and memory inspection. Check the listening address after installation and after configuration changes rather than assuming the intended setting took effect.

Connect your laptop through an SSH tunnel for an initial private deployment. A tunnel carries requests through SSH without requiring you to expose the inference listener publicly. If applications need network access, put authentication and request limits at a controlled gateway, and prevent clients from reaching the backend directly.

Encryption protects traffic in transit, while authentication decides who may consume the service. A public HTTPS endpoint without access control still lets strangers submit work and occupy your hardware. Browser origin settings are also insufficient as the sole boundary because clients outside a browser can call an exposed endpoint.

Map the full request path before calling the setup private. A local inference backend does not prevent an attached document processor or search integration from sending data elsewhere. Disable unnecessary integrations and decide whether prompts or generated answers may appear in application logs.

Evaluate complete tasks before granting tool access

Build an evaluation set from work you actually need done, using sanitized material where necessary. Include awkward inputs and requests where the correct response is to decline or report missing information. A polished answer to a friendly prompt proves very little about unattended operation.

• For extraction, compare every returned field against the source and reject invented values.
• For coding, inspect the patch and run checks that exercise the changed behavior.
• For document answers, require source references and verify that they support the response.
• For classification, include ambiguous cases and define when human review is required.
• For tool use, validate arguments against an allowlist before any action executes.

Record waiting time separately from correctness. A result can be accurate yet unsuitable for an interactive workflow, while a slow background task may still be useful. Repeat the exercise with the longest inputs you intend to allow and with overlapping requests.

Use Claude Code to inspect service configuration and prepare reviewed changes, with credentials scoped to the work. Keep inference output away from unrestricted shell execution, regardless of where the model runs. The AI operations playbook explains how to divide investigation from actions that need stronger control.

Documents and repository files can contain instructions that conflict with your intended task. Treat that content as untrusted input, and enforce permissions outside the model. A convincing generated explanation must never be enough to authorize a deployment or disclose a secret.

Operate the service within a fixed capacity

Put request limits in place before inviting other users. Bound input length and simultaneous work so one oversized request cannot consume the service indefinitely. Use a finite queue and return a clear busy response when capacity is exhausted.

Give clients deadlines and a bounded retry policy. Immediate retries against a saturated service add work precisely when it cannot keep up. For background processing, preserve jobs outside the inference process so a restart does not lose the work awaiting completion.

Monitor successful generation as well as process uptime. A service can answer a health request while its model fails to load or useful requests remain stuck. Watch memory pressure alongside queue waiting time, and keep diagnostic logs free of private prompt content unless retention is explicitly required.

Treat model replacement like an application release. Preserve the prior artifact and configuration until the replacement passes your acceptance set, and change the runtime separately where practical. If behavior regresses, restore the known configuration instead of trying to repair prompts during an outage.

Back up the material you cannot cheaply reconstruct, including service configuration and application state. Public model weights may be downloadable again, but record their identity and verify that retrieval remains possible. Rebuilding the service should not depend on remembering which download happened to work.

Run a bounded pilot before committing hardware

Choose a task whose failure will not interrupt production, such as classifying sanitized historical tickets. Write its acceptance conditions and permitted data path in a short runbook. Keep your existing workflow available while you establish whether local inference helps.

1. Prepare a dedicated Ubuntu 24.04 LTS VPS and document how clients will reach its private listener.
2. Select a model artifact with suitable licensing and record the runtime configuration.
3. Submit representative tasks, including long inputs and ambiguous examples, then inspect correctness and resource use.
4. Repeat with overlapping requests and verify that excess work is rejected or queued within your chosen limits.
5. Restart the service and confirm that clients recover without switching sensitive requests to a hosted provider.
6. Approve only the workload that passed, or stop the pilot before committing to larger hardware.

Use the pilot result to make the next purchase decision. If correctness passes but waiting time fails, evaluate suitable accelerated hardware with the same tasks. If correctness fails, change the model or keep that workload in Claude Code rather than buying capacity for an unsuitable setup.