Methodology

Benchmarks are only useful if you can trust them. This page defines every metric we report, explains how we measure each one honestly, and documents the result format our benchmark scripts emit — the same files our report pages are rendered from.

What we measure

Artifact size

Everything that must be pulled onto a fresh machine before it can serve a single token. For a conventional deployment that is the container image — base OS, Python, CUDA userspace, framework, and weights. For a Muna-compiled model it is the compiled binary plus the handful of shared libraries it links against, and the weights. We report both total bytes and file count, taken directly from registry manifests.

Cold-start Time to First Token (CTTFT)

The time from a cold start to a user receiving their first token. This is the full end-to-end path a real user hits on a fresh machine: process startup, artifact fetch, runtime initialization, model load, prefill, and the first decode step. Most inference benchmarks skip this entirely because containerized stacks make it painful to measure; we consider it the defining metric for elastic inference.

Warm Time to First Token (TTFT)

The time from request to first token on an already-warm server, measured across a range of prompt lengths. This is dominated by prefill compute, so it primarily reflects kernel quality.

Decode throughput (TPS)

Sustained tokens per second during decode, measured across distinct prompt categories (reasoning, math, code, knowledge, summarization), since speculative-decoding accept rates — and therefore throughput — vary by domain. The per-token flip side of this is inter-token latency (ITL, also called time-per-output-token or TPOT): at 500 tok/s, tokens arrive every 2ms.

Measuring honestly

Cold means cold. Modern operating systems keep a page cache that holds model weights resident in RAM indefinitely after first load. Naive re-runs are served from memory at tens of GB/s — measuring memory bandwidth, not the cold-start path. On bare metal we flush the page cache before every run using posix_fadvise(POSIX_FADV_DONTNEED), evicting every prediction resource from RAM before benchmarking.

Managed platforms measured as they are. Cold-start numbers on Modal reflect the realities of a flexible, fully-managed platform: weights are served from network-backed volumes rather than local NVMe, and each container runs inside gVisor, which adds a small per-syscall cost visible during I/O-heavy weight loading. These are reasonable platform trade-offs that a metric as demanding as CTTFT simply makes visible — so we report platform and bare-metal numbers separately rather than blending them.

Why compiled models cold-start differently. A container cold start pays for pulling and unpacking tens of gigabytes, starting an interpreter, importing a framework, and JIT-compiling kernels before any weights load. Compilation moves all of that work to build time: the artifact is a self-contained native binary with kernels already selected for the target hardware, so the cold path collapses to fetching a small binary, loading weights, and running prefill.

Rules we hold ourselves to

Every multiplier ("18× lower") is derived at build time from the raw result files — never hand-written — and the exact run configuration is published alongside it.
Raw per-run samples are published in every result file, not just summary statistics, so you can check the variance yourself.
We compare against the strongest sensible baseline (e.g. the stock SGLang image, not an unoptimized reference server).
Cold and warm numbers are always reported separately, with the cache and platform conditions stated.
Every report links a script that reproduces the run end to end.

The result format

Our benchmark scripts emit JSON result files with a fixed schema, and the report pages on this site are constructed from those files at build time. A metric result carries the benchmark configuration, the environment it ran in, and a statistical summary alongside the raw samples:

{
  "metric": "cttft",                    // "cttft" | "ttft" | "tps" | ...
  "target": "muna-on-ssh-runpod",       // what was benchmarked
  "runtime": "muna",
  "provider": "ssh",
  "tag": "@google/gemma-4-26b-a4b-it",
  "config": {                           // exact benchmark parameters
    "boots": 5,
    "prompt": "What is the capital of France?",
    "max_tokens": 1,
    "batch_size": 1
  },
  "environment": {                      // where it ran
    "host": "RunPod B200",
    "gpu": "NVIDIA B200 183GB",
    "storage": "local NVMe overlay (/)",
    "cache_eviction": "posix_fadvise(POSIX_FADV_DONTNEED) per boot",
    "cold_read": true
  },
  "summary": { "n": 5, "min": 5.96, "p50": 6.02, "p90": 6.28, "max": 6.31, "mean": 6.108 },
  "samples": [6.31, 5.96, 6.02, 6.28, 5.97]
}

Metrics measured across a sweep — TTFT by prompt length, throughput by prompt category — nest one summary-and-samples series per bucket:

{
  "metric": "ttft",
  "tag": "@google/gemma-4-26b-a4b-it",
  "config": { "max_tokens": 1, "rounds": 50, "warmup": 3, "batch_size": 1 },
  "lengths": {                          // one series per prompt length
    "512":   { "summary": { ... }, "samples": [ ... ] },
    "16384": { "summary": { ... }, "samples": [ ... ] }
  }
}

Artifact-size results list everything a registry pull fetches:

{
  "label": "@google/gemma-4-26b-a4b-it",
  "total_size": 797042640,              // bytes pulled before first token
  "file_count": 5,
  "files": [
    {
      "name": "libgemma4.so",
      "size": 100778776,
      "description": "Gemma 4 library compiled by Muna."
    }
  ]
}

Want this report for your own model? We run the same scripts against your workload — let's talk.