Hardware guide

The most important number when running AI models on your own hardware is memory. A model has to fit in memory to run, and it runs fastest when it fits in the memory of a GPU.

Memory decides which models fit

A model’s size is usually given in parameters, for example “7B” for seven billion. Models are often quantised, which stores each parameter in fewer bits so the model needs less memory, with a small loss in quality.

As a rough guide, a 4-bit quantised model needs a little over half a gigabyte of memory per billion parameters, plus extra for the conversation it is working on (the context). Longer conversations and documents need more.

Memory available	What usually fits
8 GB	Small models, short contexts
16 GB	Small to mid-sized models
32 GB or more	Mid-sized models, longer contexts
A GPU with 24 GB or more	Mid-sized models running quickly

These are rules of thumb, not guarantees. The model finder checks your actual machine.

CPU or GPU

• CPU only works, and is fine for small models and light use. It is slower, especially with longer answers.
• A GPU speeds things up a lot when the model fits in the GPU’s own memory (VRAM). At launch, the server supports NVIDIA GPUs with a current driver.
• Apple Silicon Macs share memory between the processor and the GPU, which makes them good machines for the desktop app.

Disk

Model files are large, from a few gigabytes to tens of gigabytes each. Keep plenty of free disk space, and prefer a fast SSD so models load quickly.

Many users at once

One machine can only serve so many requests at the same time. The server measures this on your hardware and queues extra requests instead of failing. If many people will use it at once, plan for a GPU server.

Our promise on numbers

We do not publish speed figures we have not measured. Measured results by hardware and model, with the method and date, will appear on the limitations page before public release.