Local LLM Cold Starts — Why First Replies Take 10 Seconds
After idling, my agent's first reply dragged. I pulled Ollama's load_duration across model sizes: 1.5s for 2GB up to 9.7s for 9.6GB, and split it by keep_alive.
Tags
InferenceOptimization
2 posts
Why Local LLMs Slow Down in Long Chats — Prefill vs Generation
A 9,700-token prompt took 55s to its first token, then 65ms on the identical second call. I split Ollama's timings into prefill vs generation to see why.