We Compiled Gemma 4 26B Into a 96MB Binary That Cold-Starts in Five Seconds.

We think containers are extremely wasteful for AI inference. To make the case, we compiled Gemma 4 26B A4B into a self-contained 96MB binary—engine, kernels, and all—using our general-purpose Python compiler. Measuring the payoff required a new metric we call Cold-start Time to First Token (CTTFT): the time from a cold start to a user's first token. On bare metal, our compiled Gemma 4 reaches a 6.3-second CTTFT at p90, an 18x reduction versus SGLang on Modal.

To see why, compare what actually has to be pulled onto a machine before it can serve a single token. First, the stock SGLang container image lmsysorg/sglang:latest; then, the entire Muna-compiled Gemma 4 binary. Hover any slice to see what it is.

A container is a full Linux userland: a Python interpreter, CUDA toolkits, hundreds of transitive Python dependencies; and much more. If we could compile Gemma 4, we would gather only the exact files we need: an inference engine with embedded kernels, plus the handful of CUDA libraries that are actually used at runtime. This lets us go from pulling 200,000 files totaling ~28GB to pulling just five.

At the core of our platform is a general-purpose compiler for Python. Our fundamental insight is that containers—the standard format for shipping software—are the wrong unit of distribution for AI inference. So instead of packaging a model inside an image, we compile it directly into a self-contained binary.

But what does the developer actually compile? We designed our compiler with two goals in mind: First, developers should write regular Python code that just works—no bespoke DSL, no rewriting their model. Second, they should be able to declare every optimization (speculative decoding, disaggregated prefill, etc) right alongside their code. In practice, compiling Gemma 4 looked something like this:

from muna import compile
from muna.beta import TorchToSGLangInferenceMetadata
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load Gemma 4 from Huggingface
REPO_ID = "google/gemma-4-26B-A4B-it"
model = AutoModelForCausalLM.from_pretrained(REPO_ID)
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
manager = model.init_continuous_batching(...)

# Compile our LLM inference function
@compile(
    metadata=[
        # Lower the PyTorch model to C++ for inference with SGLang
        TorchToSGLangInferenceMetadata(
            model=model,
            compute_architecture="sm_100"
        )
    ]
)
def gemma_4_26b_a4b_it(messages):
    """
    Stream chat completions with Gemma 4 26B A4B IT.
    """
    # Submit the request to the continuous batching manager
    request_id = f"chatcmpl-1234"
    input_ids = tokenizer.apply_chat_template(messages, tokenize=True)
    manager.add_request(
        request_id=request_id,
        input_ids=input_ids,
        streaming=True
    )
    # Consume chunks for this request from the engine
    for chunk in manager.request_id_iter(request_id=completion_id):
        new_token_ids = chunk.generated_tokens[-1:]
        token_text = tokenizer.decode(new_token_ids)
        yield ...

The full script is ordinary Python code that you can run yourself. It builds upon the continuous batching API introduced in Transformers v5, which allows our compiler to lower calls like manager.add_request into C++. The only Muna-specific detail is the @compile decorator, and specifically its metadata argument.

Developers can provide additional context to the compiler, in the form of small Plain Old Data objects we call metadata types. In this case, TorchToSGLangInferenceMetadata tells the compiler to lower the PyTorch model to SGLang for inference at runtime. For the astute reader, this raises a question: how can the compiler use SGLang—a Python library—inside the compiled binary? This is where the pain fun begins.

Because Muna compiles models ahead-of-time to run on bare metal, we couldn't ship upstream SGLang with its Python dependencies. We needed a pure C++ inference engine, so we built nanosgl.

Rather than follow SGLang's framework design, we built nanosgl as a library of parts. It owns no model or runtime of its own; instead, each model—here, Gemma 4—is its own small library that composes the pieces it needs (scheduler, KV cache, kernels, and so on) into a fully specialized engine. The upshot is that every model pulls in exactly what it requires and nothing more, and adding a new model never touches the core library.

Kernels are perhaps the most crucial part of any LLM serving engine, and here we took direct inspiration from TokenSpeed. We built a kernel registry that holds multiple interchangeable implementations for a given family (GQA, MoE grouped GEMM, RoPE), selected by hardware capability (Nvidia sm_100, Apple metal_4) and dtype, and swappable with a single policy setting. And because there's an open, thriving ecosystem of high-performance kernels, we built the infrastructure to integrate ones written in CuTe DSL, Triton, TileLang, and more.

There is one final architectural bet we made: nanosgl runs entirely within a single process. Everything, including tensor parallelism across multiple GPUs, lives in one process. We did this for two reasons. First, as a down-payment on running nanosgl in environments that can't spawn new processes, like WebAssembly or iOS. And second, because we are actively exploring GPU multi-tenancy, where we load multiple distinct models into the same process and onto the same GPU to push hardware utilization higher. More on this below.

Because compiled binaries are so small, it finally becomes feasible to measure something most inference stacks can't: the time from a cold start to a user receiving their first token. We call this Cold-start Time to First Token (CTTFT). We benchmarked CTTFT for both the SGLang and Muna engines across Modal and a bare-metal B200.

These Modal numbers reflect the realities of a flexible, fully-managed platform. Weights are served from network-backed volumes, so a cold boot streams 50GB+ over the network rather than off local NVMe, and where a container lands relative to storage adds some run-to-run variance. Modal also runs each container inside gVisor (runsc) for strong isolation, which carries a small per-syscall cost that shows up during the I/O-heavy weight-loading phase. Both are reasonable trade-offs that a metric as demanding as CTTFT simply makes visible.

The bare-metal numbers required care of a different kind. Modern operating systems keep a page cache, holding model weights and other files resident in memory indefinitely after first load. This means that naive re-runs are served from RAM at tens of GB/s, measuring memory bandwidth rather than the cold-start path a real user hits on a fresh machine. To measure honestly, we flushed the page cache before every run, using posix_fadvise(..., POSIX_FADV_DONTNEED) to evict every prediction resource from RAM before benchmarking. We were able to consistently hit a CTTFT of ~6 seconds, and have since been able to bring this down further to ~3 seconds. At a CTTFT this low, the cold-start time becomes indistinguishable from a median time to a first answer token.

The compiled engine also delivers competitive time-to-first-token (TTFT): at 10k input tokens, it comfortably produces a token in under 200ms at batch size 1. Much of this comes down to the compiler's choice of kernels, which it sources from FlashInfer, TensorRT-LLM, and CUTLASS.

Finally, throughput. Across reasoning, math, code, knowledge, and summarization prompts, the engine sustains 200-590 tokens per second at batch size 1, with remarkably tight run-to-run variance. Much of this is thanks to DFlash, a relatively new speculative decoding technique that offers higher accept lengths than other methods. We are also exploring JetSpec and DSpark as alternate speculative decoding methods.

We are working on compiling GLM 5.2. Beyond rivaling frontier models, this model gives us an opportunity to run two experiments with real implications for hardware utilization and cost.

The first is disaggregated prefill, which splits the prefill and decode stages of inference across separate GPUs or nodes. Baseten has used it to achieve over 280 tokens/sec on GLM 5.2, but for most teams it is effectively out of reach unless you pay a provider to build it for you as a black-box. We are designing something a developer can opt into with three extra lines of Python code:

@compile(
    metadata=[
        # Use disaggregated inference
        TorchToSGLangInferenceMetadata(
            ...,
            disaggregation=SGLangDisaggregationConfig(
                topology="inter_node"
            )
        )
    ]
)
def glm_5_2(messages):
    ...

Our compiler will generate a sidecar binary that handles peer-to-peer disaggregated inference, building upon Muna's ability to dispatch inference to remote workers.

The second is GPU multi-tenancy, which falls out of the same insight: prefill is compute-bound, but decode often isn't. When a decode worker can't saturate the GPU's SMs, that idle capacity is room to co-locate a second model, like an embedding model, alongside it. And because Muna compiles models into binaries that share a single process, a team could serve two or more models on one GPU node with minimal throughput loss. We have proven this out with embedding models, and now we intend to do the same with fronteir LLMs.

You can run the compiled Gemma 4 model on our inference endpoint. You can also deploy it on your own infrastructure; today we support Baseten and Modal, with more on the way:

# Deploy compiled Gemma 4 on Modal
$ muna deploy @google/gemma-4-26b-a4b-it --provider modal --gpu b200

If you are running production LLM workloads, let's talk; we'll put together a benchmark report showing how compiling your models improves utilization, cold starts, and throughput.