Google TurboQuant: 3-Bit KV Cache With Zero Accuracy Loss
A deep dive into Google TurboQuant's PolarQuant and QJL techniques — 6x KV cache memory reduction and 8x attention speedup, and what that actually means in practice.
Tags
quantization
5 posts
IQ*_K/IQ*_KS Quantization Merged into llama.cpp
IQ-series quantization methods developed in ik_llama.cpp are being merged into llama.cpp mainline. Learn about IQ2_K through IQ4_KS precision improvements and local LLM inference optimization.
Heretic 1.2: 70% VRAM Reduction via Quantization and MPOA Explained
Heretic 1.2 is here with 4-bit quantization cutting VRAM usage by up to 70% and MPOA delivering higher-quality abliteration. A deep dive into the latest cost-saving techniques for local LLM operations.
How to Run Qwen3-Coder-Next 80B on 8GB VRAM
Analyzing quantization and lazy loading techniques to run an 80B parameter coding AI model on consumer 8GB VRAM GPUs. Exploring the practicality and limitations of local LLM coding.
NVIDIA's NVFP4 Cuts LLM Inference Costs by 8x
NVIDIA NVFP4 cuts LLM inference costs 8x while preserving accuracy. RTX 4090 AdaLLM benchmarks plus monthly GPU cost simulations prove the FP32-to-FP4 savings.