Get More Tokens per Second from llama.cpp

I run large language models locally on my Mac. Not through a cloud API, not through someone else’s server, but on the machine sitting on my desk, an Apple Silicon Mac (an M3 Pro). The main reason is control: my prompts stay on my hardware, there is no per-token bill, and I can experiment as much as I want without watching a meter.

Recently I set up llama.cpp, one of the most popular open-source tools for running these models locally, to serve a coding model I could query all day. The easiest way to install almost anything on a Mac is Homebrew, the standard package manager, so I ran the obvious command:

brew install llama.cpp

It installed cleanly. No errors. The server started. The model answered my questions. Everything looked fine. And then I checked the speed, and it was not fine at all.

First, what “tokens per second” even means

When a language model writes text, it does not produce whole words in one shot. It produces tokens, which are small chunks of text, roughly three-quarters of a word on average. “Cheeseburger” might be two or three tokens; a common word like “the” is one. The model generates them one at a time, each token informed by all the ones before it.

So tokens per second (tok/s) is just the speed at which the model produces text. The higher the number, the faster words stream onto your screen. This step, generating new tokens one after another, is called decode. Decode speed is the number you actually feel when you are waiting for an answer. At 40 tok/s, text pours out faster than you can read it. At 6 tok/s, you sit there watching it crawl.

What I expected versus what I saw

I had a reference point. I also run Ollama, another popular local-inference tool, on the same Mac. With the same model, Ollama decodes at roughly 45 tok/s. Snappy. Comfortable. So I expected my fresh llama.cpp install to land somewhere in the same ballpark. Same machine, same model, same class of tool.

Instead I got 6.3 tok/s. Not 15 percent slower. Not half as fast. Roughly seven times slower than the tool running the identical model a few inches away on the same laptop.

A 7x gap between two tools doing the same job on the same hardware is not a tuning difference. Something structural was wrong. The question was what, and I did not want to guess.

CPU inference versus GPU inference, and why the gap is so large

To understand what I went looking for, you need one more piece of background: where the math actually runs.

Running a language model is, underneath everything, a huge pile of matrix multiplications, the same arithmetic operation repeated across billions of numbers. There are two very different places that work can happen:

• The CPU (the general-purpose processor) is a small number of very capable cores. Great at doing complicated, varied tasks one after another. Not great at doing the same simple multiplication billions of times in parallel.
• The GPU (the graphics processor) is thousands of simple cores built to do exactly one thing: the same math on huge batches of numbers at the same time. This is precisely the shape of the work a language model needs.

On Apple Silicon, the GPU is built into the same chip as the CPU, and Apple exposes it through a framework called Metal. When llama.cpp uses Metal, the heavy math runs on the GPU cores and it is fast. When it does not, that same math falls back to the CPU, and it is slow. A 7x gap is exactly the kind of penalty you pay for running GPU-shaped work on a CPU. That gave me a strong hypothesis: my llama.cpp was somehow not using the GPU at all.

How I traced it: measure, do not guess

The temptation with a slow program is to start changing settings and hope. That is guessing, and it wastes hours. Instead I went looking for direct evidence of which backend was actually loaded.

Quick definition, because this is the crux of the whole story. A backend here means the specific engine that does the computation, CPU or Metal (GPU). llama.cpp ships these engines as separate dynamic libraries, or dylibs. A dylib is just a chunk of compiled code kept in its own file that a program loads when it starts, instead of baking everything into one big executable. Think of it as a swappable part the program picks up at launch. If the part is not there, the program cannot use it.

The relevant files for llama.cpp are named around libggml (GGML is the underlying math engine that powers llama.cpp). I went to look at which of those dylibs Homebrew had actually installed. Here is what I found in the installed package:

libggml-base.dylib     # core GGML code
libggml.dylib          # the top-level library
# ... and that was it for the compute backends

What was missing jumped out immediately: there was no libggml-metal.dylib. That is the Metal backend, the dylib that lets llama.cpp run on the Apple GPU. The Homebrew “bottle” (a bottle is Homebrew’s term for a prebuilt, ready-to-pour binary package, so you do not have to compile anything yourself) shipped the base libraries and the CPU path, but not the Metal GPU library.

To confirm it was not just hiding somewhere, I also watched llama.cpp’s own startup logs. When llama.cpp loads a model onto the GPU, it announces it: you see it register a Metal device and, crucially, allocate Metal buffers (chunks of GPU memory) and load the model’s layers into them. In my brew build, none of that appeared. No Metal device, no Metal buffer allocation, no layers offloaded to GPU. Everything sat on the CPU.

The root cause

So the diagnosis was concrete, not a hunch: the Homebrew bottle for llama.cpp shipped without the Metal backend dylib. With no GPU backend available to load, llama.cpp did the only thing it could and ran the entire model on the CPU. The 6.3 tok/s was exactly what CPU-only inference of that model looks like on this machine. The model was never on the GPU at all.

The fix: build from source with Metal turned on

The convenience package could not use my GPU, so I gave up the convenience and built llama.cpp from source myself. The advantage of building from source is that you control the build options, and llama.cpp has a build option specifically for this.

llama.cpp uses CMake, a common tool for configuring how a C++ program gets compiled. CMake options are passed as flags that start with -D. The one that matters here is GGML_METAL, which tells the build to compile and include the Metal GPU backend. Turning it on looks like this:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Configure the build with the Metal GPU backend enabled
cmake -B build -DGGML_METAL=ON

# Compile
cmake --build build --config Release

That single flag, -DGGML_METAL=ON, is the whole difference. It ensures the Metal backend actually gets built and shipped alongside the binaries, so llama.cpp has a GPU engine to load at runtime. After rebuilding and rerunning the exact same model, decode speed jumped to 33 to 39 tok/s.

How to check GPU residency yourself

The real lesson is not “this one package was broken.” It is that you should never trust a local-inference speed number until you have confirmed the model is actually on the GPU. Here is the cheap check that would have saved me an evening, and it works whether you installed from a package or built from source.

When you start llama.cpp and load a model, read the startup logs. You are looking for two things:

1. A Metal device gets registered. Early in startup, a GPU-accelerated build announces that it found and initialized the Metal backend. If you never see Metal mentioned, that is your warning sign.
2. Layers get offloaded to the GPU and Metal buffers get allocated. A model is made of many layers. In a healthy GPU run, the logs show those layers being assigned to the GPU and memory (Metal buffers) being allocated on the GPU to hold them. If instead you see everything loaded to CPU, or no buffer allocation on the GPU at all, the model is running on the CPU and your speed will show it.

A second, even simpler cross-check: while the model is generating, open Activity Monitor and watch GPU usage. If the GPU is basically idle while the model churns out text, the work is on the CPU. If GPU usage spikes during generation, good, the model is where it belongs. Two independent signals, the startup logs and the live GPU meter, are enough to catch this class of bug in seconds instead of hours.

Caveats

A few honest limits on all of this:

• This is one machine and one model. An M3 Pro with a specific coding model. Your exact tokens-per-second numbers will differ with different hardware, different models, and different model sizes. The 7x gap is my measurement, not a universal constant. The shape of the lesson generalizes; the exact figures do not.
• Homebrew may fix the bottle. Packages change. It is entirely possible that by the time you read this, the llama.cpp bottle ships the Metal backend and installs GPU-accelerated out of the box. That would be great, and it would not change the point: verify residency rather than assuming.
• A missing backend is only one way to lose the GPU. You can also fail to offload layers to the GPU through a runtime setting, run a model too large to fit in GPU memory, or hit other configuration issues. The startup-log and GPU-meter checks catch all of them, which is why they are worth building into your habits.
• Building from source is more work. You trade the one-line install for managing a build and its toolchain. For me the speed was worth it, but “just use the package” is a reasonable default until the package proves it is leaving performance on the table.

The takeaway I keep coming back to: a convenience package can hand you a backend-less build that silently drops to the CPU, with no error and no warning, just a slow number you might never question. Before you trust any local-inference benchmark, prove the model is on the GPU. Do not trust the happy path.