> Blog Post

Build a Personal AI Dev Environment: Hybrid Models, Local Inference, and a Workflow That Costs Almost Nothing

aillmlocal-modelsollamaclaude-codedeveloper-tools

We have written a lot about running AI at the team and infrastructure scale: the hybrid playbook for routing requests between cloud and local models, self-hosting LLMs on Kubernetes, observability for the inference layer, and the agent control plane that lets a fleet decompose and execute open-ended goals. Those posts are about systems that serve a whole organization.

This one is smaller and more personal. It is about the environment on your machine — the one you code in every day. The same principles that make a hybrid fleet economical at scale make a personal setup fast, private, and nearly free to run. You do not need a GPU cluster or a task ledger. You need a good local model, a frontier API key, a thin router, and the discipline to send each piece of work to the right place.

This is the setup we run ourselves, and the one we recommend to individual engineers at the teams we work with.

The Core Principle, Scaled Down

The hybrid playbook makes one argument: use frontier models for thinking, local models for doing. At the org scale that means a routing layer in front of a vLLM cluster. At the personal scale it means exactly the same split, just running on the hardware already under your desk.

A typical developer's AI usage breaks down into three buckets:

  1. Deep work — architecting a feature, debugging something subtle, reviewing a tricky diff, planning a migration. This genuinely needs a frontier model. Reach for Claude.
  2. Volume work — generating boilerplate, writing a commit message, summarizing a file, drafting a docstring, explaining an error, transforming JSON. High frequency, low judgment. A local 8B–32B model handles this within a few percent of frontier quality.
  3. Ambient work — autocomplete, inline suggestions, "what does this regex do." Constant, latency-sensitive, and ideally never leaves your machine.

Buckets 2 and 3 are where most of your tokens go, and they are exactly the work a local model does well. Send them to the cloud and you are, in the words of the hybrid post, hiring a senior architect to paint walls — except now you are paying the bill personally and waiting on a network round-trip for every keystroke.

The Hardware You Actually Have

You do not need the rack of GPUs from the self-hosting guide. Local inference on a single workstation is comfortable on hardware many developers already own:

Machine Practical model ceiling What it's good for
Apple Silicon (M-series, 16GB) 8B quantized Autocomplete, commit messages, summaries
Apple Silicon (32GB+) 14B–32B quantized Code generation, refactors, most volume work
NVIDIA 12GB VRAM (3060/4070) 8B–14B quantized Code generation, fast inference
NVIDIA 24GB VRAM (3090/4090) 32B quantized Near-frontier on well-scoped tasks

The unified memory on Apple Silicon is the quiet hero here — a 32GB Mac runs a quantized Qwen 32B that would otherwise need a $1,500 GPU. Quantization (4-bit / Q4_K_M) is the lever that makes this work: it trades a sliver of quality for fitting a much larger model in memory, and for bucket-2 work that trade is almost always worth it.

Layer One: The Local Inference Server

Ollama is the path of least resistance for a personal setup. It is a single binary, manages model downloads, and exposes both its own API and an OpenAI-compatible endpoint — which matters because nearly every tool you already use can point at it.

# Install (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a coding-focused model sized to your hardware
ollama pull qwen2.5-coder:32b      # 24GB VRAM or 32GB+ unified memory
ollama pull qwen2.5-coder:7b       # 16GB machines, fast
ollama pull llama3.1:8b            # general-purpose fallback

# Verify it serves on the OpenAI-compatible endpoint
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "messages": [{"role": "user", "content": "Write a bash one-liner to find the 10 largest files under cwd"}]
  }'

The Qwen 2.5 Coder family is the current sweet spot for local development work — the 7B is genuinely useful for autocomplete and small generations, and the 32B is good enough that you will reach for the cloud less than you expect. DeepSeek-Coder and Codestral are reasonable alternatives if you want to compare.

A practical note that bites everyone: keep the model warm. Ollama unloads models from memory after an idle timeout, and the first request after an unload pays a multi-second load penalty. For a dev environment where you want snappy responses, set OLLAMA_KEEP_ALIVE=-1 so your primary model stays resident:

# In your shell profile
export OLLAMA_KEEP_ALIVE=-1        # never unload
export OLLAMA_MAX_LOADED_MODELS=2  # keep coder + general model both warm

Layer Two: Claude Code as the Frontier Brain

For bucket-1 work — the deep reasoning, the multi-file changes, the "I genuinely do not know how to approach this" problems — you want a frontier model with agentic capability. This is exactly what we used to build FlagSignals end-to-end, and the workflow holds up just as well for personal projects.

The key to keeping this affordable is the same discipline the orchestration post preaches at the fleet level: keep the expensive model off the hot path. Claude Code is where you bring the hard, ambiguous goals — not where you autocomplete a loop. A few habits that keep frontier spend low without sacrificing capability:

  • Let Claude plan, then let local models execute the plan. Ask Claude to design the approach and write the tricky core; hand the repetitive scaffolding it identifies to your local model. This is single-developer task decomposition — the same pattern as the control plane, minus the queue.
  • Use a CLAUDE.md in each repo so the frontier model spends tokens on your problem, not on rediscovering your conventions every session.
  • Reserve frontier context for what needs it. Don't paste a 2,000-line generated file in for a formatting fix your local model can do offline.

Layer Three: The Router

The piece that ties it together is a thin router — the personal-scale version of the routing layer from the hybrid playbook. Its only job is to send each request to the cheapest model that can do it well. Unlike the orchestration system, there is no durable ledger and no queue: it is a synchronous function that picks a backend.

The single most important design decision, straight from the hybrid post, is explicit routing over clever classification. You almost always know at call time which bucket a request is in. Don't build an ML classifier to decide; tag the call.

import OpenAI from "openai";

// Two clients, one interface — Ollama speaks the OpenAI protocol.
const local = new OpenAI({ baseURL: "http://localhost:11434/v1", apiKey: "ollama" });
const cloud = new OpenAI({
  baseURL: "https://api.anthropic.com/v1", // Anthropic's OpenAI-compatible endpoint
  apiKey: process.env.ANTHROPIC_API_KEY,
});

type Lane = "execute" | "ambient" | "reason";

const LANES: Record<Lane, { client: OpenAI; model: string }> = {
  execute: { client: local, model: "qwen2.5-coder:32b" }, // bucket 2: volume work
  ambient: { client: local, model: "qwen2.5-coder:7b" },  // bucket 3: fast, small
  reason:  { client: cloud, model: "claude-opus-4-7" },    // bucket 1: deep work
};

async function ask(lane: Lane, prompt: string, extra: Record<string, unknown> = {}): Promise<string> {
  const { client, model } = LANES[lane];
  const resp = await client.chat.completions.create({
    model,
    messages: [{ role: "user", content: prompt }],
    ...extra,
  });
  return resp.choices[0].message.content ?? "";
}

// You know the lane at call time. Tag it.
const commitMsg = await ask("execute", `Write a conventional commit message for this diff:\n${diff}`);
const plan = await ask("reason", "Design a migration off the deprecated auth API in this service.");

That is the whole router. Three lanes, a dictionary, and a function. The sophistication lives in where you call which lane, not in the router itself.

There is one borrowed pattern worth adding from the orchestration post: escalate when uncertain. Instruct your local-lane prompts to emit a sentinel — say NEED_FRONTIER — when a task is beyond them, rather than inventing an answer. Then your wrapper retries on the reason lane. This gives you most of the cost savings of local-first routing with a safety net against a small model confidently producing garbage.

async function askWithEscalation(prompt: string, extra: Record<string, unknown> = {}): Promise<string> {
  const guarded = prompt + "\n\nIf this requires reasoning beyond your ability, reply with exactly NEED_FRONTIER.";
  const out = await ask("execute", guarded, extra);
  if (out.includes("NEED_FRONTIER")) {
    return ask("reason", prompt, extra);
  }
  return out;
}

Wiring It Into the Tools You Already Use

A router is only useful if your editor and shell talk to it. Because Ollama exposes an OpenAI-compatible endpoint, most tools need nothing more than a base-URL change:

  • Editor autocomplete — Continue and similar extensions let you set separate models for autocomplete (point at your 7B local model) and chat (point at Claude). This is the router pattern expressed in editor config: ambient work stays local, deep questions go to the cloud.
  • Shell — a tiny wrapper around the execute lane gives you instant, offline explain, commit-msg, and summarize commands with zero token cost.
  • Claude Code — runs alongside as your bucket-1 agent for anything that touches multiple files or needs real reasoning.

The result is a setup where the frequency of your AI usage is decoupled from your bill. You can hammer autocomplete and commit-message generation thousands of times a day at the cost of electricity, and spend frontier tokens only on the handful of problems each day that actually warrant them.

Know What It's Costing You

Even at the personal scale, the observability lesson applies: a system you cannot see into is a system you cannot tune. You do not need OpenTelemetry and Grafana for a one-person setup, but you do want to know the answer to two questions: what fraction of my requests went to the cloud, and what did that cost.

A few lines in the router's logging gets you there:

import { appendFileSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";

type Usage = { prompt_tokens: number; completion_tokens: number };

// `started` is a performance.now() timestamp captured before the call.
function logCall(lane: Lane, model: string, usage: Usage, started: number): void {
  const entry = JSON.stringify({
    ts: Date.now() / 1000,
    lane,
    model,
    prompt_tokens: usage.prompt_tokens,
    completion_tokens: usage.completion_tokens,
    latency_ms: Math.round(performance.now() - started),
  });
  appendFileSync(join(homedir(), ".ai-dev", "usage.jsonl"), entry + "\n");
}

Tail that file at the end of a week and the picture is usually stark: 90%+ of calls served locally, a frontier bill in the low single-digit dollars, and the local model handling the volume invisibly. If your cloud percentage is creeping up, the same diagnosis from the orchestration post applies — you are over-routing to frontier, and the fix is tightening which calls you tag reason.

Privacy Is a Feature, Not an Afterthought

There is a benefit here that does not show up on the cost ledger: the volume of your day-to-day code — the proprietary logic, the half-finished functions, the internal API shapes — never leaves your machine. Bucket-2 and bucket-3 work runs entirely local. Only the deliberate, deep-reasoning requests you consciously route to the cloud go over the wire, and you decide each one. For developers working under NDA, on regulated codebases, or simply allergic to shipping their keystrokes to a third party, local-first is the design that lets you use AI aggressively without the data-exfiltration anxiety.

The Bottom Line

A personal AI dev environment is the same architecture as the fleet, minus the orchestration plumbing. The local inference server replaces the vLLM cluster. The router replaces the dispatcher. Claude Code replaces the frontier orchestrator. The discipline is identical: send volume work to local models, reserve frontier intelligence for genuine reasoning, and measure the split so you can keep tuning it.

Start with one model. Install Ollama, pull qwen2.5-coder:7b, and point your editor's autocomplete at it. Notice how much of your daily AI usage never needed the cloud in the first place. Then add the execute lane for commit messages and summaries, keep Claude Code in your pocket for the hard problems, and let the local model absorb everything in between.

The endpoint is a workflow where doing more AI-assisted work costs you essentially nothing — which, as with the fleet, is exactly where you want to be.