Local Reasoning Model Token Cost — thinking ON/OFF Measured
I ran 13 questions on gemma4:12b with thinking ON and OFF. Reasoning got one more right while spending 68x the output tokens and 19x the wall-clock.
Tags
LocalLLM
7 posts
Ollama num_ctx Silent Truncation — Why My Agent Forgot Its Prompt
My local agent kept ignoring its system prompt on long inputs. Past num_ctx, Ollama silently trims the front of the prompt — no error. I measured where it breaks.
Local LLM Cold Starts — Why First Replies Take 10 Seconds
After idling, my agent's first reply dragged. I pulled Ollama's load_duration across model sizes: 1.5s for 2GB up to 9.7s for 9.6GB, and split it by keep_alive.
Why Local LLMs Slow Down in Long Chats — Prefill vs Generation
A 9,700-token prompt took 55s to its first token, then 65ms on the identical second call. I split Ollama's timings into prefill vs generation to see why.
Local LLM Reproducibility: Testing Temperature and Seed
I sent the same prompt to local Gemma 4 dozens of times. temperature=0 was deterministic, and even at higher temperature a fixed seed collapsed output to one line.
Ollama Structured Outputs with Pydantic — Type-Safe JSON
A hands-on guide to Ollama's JSON schema enforcement with Pydantic for type-safe local LLM responses. Measured: 6x faster with near-100% parse success.
Private MCP Server with Gemma 4 + FastMCP — Fully Offline
Run an offline AI tool pipeline with Ollama, Gemma 4, and FastMCP, no internet needed. Built for medical, legal, and finance where data stays on the premises.