llama.cpp

tl;dr: llama.cpp is the C/C++ inference engine most local-LLM tools are built on. It runs GGUF models on CPU, Apple Silicon, NVIDIA, AMD, Vulkan, SYCL, Hexagon, Ascend, Moore Threads, and WebGPU — and it ships its own llama-server with an OpenAI-compatible API plus a built-in web UI.

What it does

llama.cpp runs LLM and VLM (vision-language) inference with no external dependencies. Quantized GGUF models (1.5–8 bit) let the same code path serve 8B models on a phone and 400B models across multi-GPU setups. The library sits underneath Ollama, LM Studio’s extension packs, llama-cpp-python, and dozens of other tools.

Repo sits at 127,617 stars, 22,945 forks, MIT. Latest stable is v0.4.0 (Sep 4, 2026). Build on master, run from pre-built binaries on the releases page, or grab the desktop app from llama.app.

Install

Pre-built binaries are the path of least friction. Source build is the path of most control.

# Pre-built (Linux x64 / Apple Silicon / Windows)
# Download from https://github.com/ggml-org/llama.cpp/releases

# Build from source
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

# Docker
# see https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md

The pre-built binary ships llama-cli, llama-server, llama-completion, and quantization tools. After install:

# Download and run a model directly from Hugging Face
llama-cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF

llama-server defaults to http://localhost:8080 and serves /v1/chat/completions, /v1/completions, /v1/models, plus a built-in chat web UI at /.

Backends

CPU+GPU hybrid inference is supported — partial offload lets models larger than total VRAM run with the spillover on CPU.

What v0.4.0 added

• Initial Qwen3.8-Flash-Next (qwen4exp) and NVIDIA Nemotron-3-Puzzle-75B-A9B support
• ggml bumped to v0.23.0 with sparse flash attention, async RPC backends, Apple RDMA as an RPC transport
• Lazy tensor reading (--lazy-mode) and --n-cpu-ffn for partial CPU FFN offload
• Per-slot server context limits and data: URL media inputs on the server
• Video input flags (--video-*) and DeepSeek-V4-Flash-Vision-Exp multimodal support
• Gemma-4 vision handling fixes, Idefics3 preprocessing fix, Qwen3-TTS fix
• Built-in UI: tool policy per conversation replaces per-conversation MCP overrides; Settings and MCP routes moved into dialogs

Gotchas

• The repo moved from ggerganov/llama.cpp to ggml-org/llama.cpp — old redirects work but new issues/PRs land in the org.
• Default parameters come from the GGUF metadata, not from your CLI. Upgrade can change model behavior silently.
• Quantization format and -ngl (number of layers to GPU) interact — a fully-quantized model on CPU is fine, but a partly offloaded model needs VRAM for the kept layers.
• The server binds localhost with no auth. Don’t expose 8080 to the internet without a reverse proxy.

Related: Ollama wraps this engine with a CLI + library. LM Studio ships llama.cpp packs under a GUI agent. best local LLM.

Untested here — facts from the llama.cpp GitHub repo and v0.4.0 release notes fetched 2026-09-09.