Correctness

mojo-dllm is checked at four levels, each against an implementation it does not share code with.

1. Formats and kernels (unit tests, every commit)

  • GGUF parsing, Q4_K / Q5_K / Q6_K / Q5_0 / Q8_0 / F16 / BF16 decoding: compared with the gguf Python package's dequantization on fixtures it generated (max abs difference 1e-6).
  • Both GEMMs: compared with an f64 reference over the same dequantized weights; relative error under 1% with int8 activations.
  • RMSNorm, RoPE, softmax, bidirectional attention, SwiGLU: closed-form cases.
  • The forward pass of a 2-layer model with random K-quant weights: compared with a numpy forward pass; relative logit error under 2% on the LLaDA model and under 5% on the Dream model (its large Q/K biases sharpen the softmax, which amplifies int8 error), and the same argmax on every position.

The GPU has its own suite, tests/gpu/test_gpu.mojo, which runs on a machine with an NVIDIA GPU (scripts/run-gpu-tests.sh; CI only builds it). It checks each GPU kernel against the CPU function it replaces, the int8 mma fragment layout against a CPU matmul, and the GPU forward pass of both tiny models against the numpy reference, with a 5% bound on LLaDA (attention runs in f16) and 2% on Dream, and the same argmax as the CPU path on every position.

2. Tokenizer (unit tests, every commit)

36 strings covering English, contractions, numbers, accents, combining marks, CJK, Cyrillic, Arabic, Devanagari, emoji, CRLF, non-breaking and em spaces, code, and LLaDA's chat-template special tokens. Every id matches llama-tokenize on the real LLaDA GGUF and on the real Dream GGUF, and decoding restores the input byte for byte.

DiffusionGemma's gemma4 tokenizer has its own 54 cases (runs of spaces and newlines, CJK, emoji, byte-fallback characters, a literal U+2581, and the chat template's markers), with ids from llama-tokenize on the real model file (tests/fixtures/gemma4-tokenizer-golden.tsv, and tests/fixtures/gemma4-tokenizer.gguf holds the vocabulary they are encoded with). Each case is also checked with the BOS the model adds, and decoded back; 53 of the 54 restore the input, and the one that holds U+2581 comes back with spaces, as llama.cpp decodes it. tests/test_gemma4_tiny.mojo pins each rule on a vocabulary small enough to work out by hand.

3. Logits on the real model

tools/ref/llada_ref.cpp links libllama and dumps llama.cpp's logits for a fixed token sequence; mojo-dllm logits dumps ours; bench/parity.py compares them. The sections marked GPU run mojo-dllm logits --device gpu.

The GPU sections also compare against llama.cpp built with CUDA. That build quantizes activations with one scale per 32 values, as the GPU path does, while llama.cpp on the CPU uses one per 256, as the CPU path does. On Dream's prompt rows the two llama.cpp builds disagree with each other, so a single reference would mix that spread into mojo-dllm's numbers. The table under each GPU section shows all three pairs.

LLaDA-1.5.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
64 tokens640.999870.999890.9972 (row 13)0.01861 / 644.75 / 5
126 tokens1260.999900.999910.9993 (row 19)0.016112 / 1264.67 / 5

Evidence: bench/parity/2026-10-30-llada-LLaDA-1.5.Q4_K_M.json (mojo-dllm 5c82e8c).

LLaDA-8B-Instruct.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
64 tokens640.999850.999880.9951 (row 13)0.01862 / 644.83 / 5
126 tokens1260.999880.999890.9992 (row 19)0.017120 / 1264.67 / 5

Evidence: bench/parity/2026-10-03-llada.json (mojo-dllm 0b3e17a).

LLaDA-8B-Instruct.Q4_K_M.gguf, GPU

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
64 tokens640.999810.999830.9960 (row 13)0.02162 / 644.77 / 5
126 tokens1260.999880.999880.9984 (row 24)0.017120 / 1264.74 / 5

The same canvases against a second reference, llama.cpp CUDA, and the two references against each other:

Canvascomparedmedian cosineworst cosinemedian rel. RMSsame argmax
64 tokensmojo-dllm vs llama.cpp CPU0.999810.99600.02162 / 64
64 tokensmojo-dllm vs llama.cpp CUDA0.999880.99520.01760 / 64
64 tokensllama.cpp CPU vs llama.cpp CUDA0.999720.99600.02660 / 64
126 tokensmojo-dllm vs llama.cpp CPU0.999880.99840.017120 / 126
126 tokensmojo-dllm vs llama.cpp CUDA0.999920.99870.015120 / 126
126 tokensllama.cpp CPU vs llama.cpp CUDA0.999830.99730.021117 / 126

Evidence: bench/parity/2026-10-09-llada-gpu.json (mojo-dllm cedc0b5).

DiffuCoder-7B-cpGRPO.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999940.999960.9964 (row 10)0.01966 / 714.86 / 5
133 tokens1330.999930.999940.9612 (row 2)0.016128 / 1334.72 / 5

Evidence: bench/parity/2026-10-30-dream-DiffuCoder-7B-cpGRPO.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999920.999930.9795 (row 6)0.03664 / 714.76 / 5
133 tokens1330.999890.999900.9872 (row 22)0.031115 / 1334.41 / 5

Evidence: bench/parity/2026-10-30-dream-Dream-Coder-v0-Instruct-7B.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-v0-Instruct-7B-Q4_K_M.gguf

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999930.999940.9668 (row 26)0.03264 / 714.61 / 5
133 tokens1330.999860.999880.9782 (row 11)0.03392 / 1334.22 / 5

Evidence: bench/parity/2026-10-03-dream.json (mojo-dllm 0b3e17a).

Dream-v0-Instruct-7B-Q4_K_M.gguf, GPU

Canvasrowsmedian cosinemasked rows medianworst cosinemedian rel. RMSsame argmaxtop-5 overlap
71 tokens710.999910.999930.9455 (row 26)0.04262 / 714.63 / 5
133 tokens1330.999840.999860.9016 (row 11)0.03785 / 1334.22 / 5

The same canvases against a second reference, llama.cpp CUDA, and the two references against each other:

Canvascomparedmedian cosineworst cosinemedian rel. RMSsame argmax
71 tokensmojo-dllm vs llama.cpp CPU0.999910.94550.04262 / 71
71 tokensmojo-dllm vs llama.cpp CUDA0.999940.81790.02064 / 71
71 tokensllama.cpp CPU vs llama.cpp CUDA0.999870.64920.04462 / 71
133 tokensmojo-dllm vs llama.cpp CPU0.999840.90160.03785 / 133
133 tokensmojo-dllm vs llama.cpp CUDA0.999900.89750.02297 / 133
133 tokensllama.cpp CPU vs llama.cpp CUDA0.999790.87840.03980 / 133

Evidence: bench/parity/2026-10-09-dream-gpu.json (mojo-dllm cedc0b5).

Exact equality is not the goal: both runtimes quantize activations to int8, but they block and accumulate in different orders. The llama.cpp maintainers recommend the same kind of statistical comparison for diffusion models.

The table scores every position of each canvas. On the original Dream-v0 checkpoint the median agrees as closely as on LLaDA, but a few prompt positions do not: they sit on special tokens and newlines, where Qwen-family models carry very large outlier activations that magnify int8 rounding. Masked positions, the ones the sampler reads, agree closely. Which of the two runtimes is nearer an f32 reference at those outlier rows is not measured yet.

Agreement with llama.cpp shows that two int8 implementations land near each other. It does not show that either is right. The only f32 forward-pass reference in this repository is the numpy pass on the tiny models (section 1), and the GPU tests compare against it too.

4. Generated tokens on the real model

llama.cpp's own diffusion example cannot serve as a token-level reference. On the pinned commit it never applies --diffusion-cfg-scale or --diffusion-alg-temp, adds Gumbel noise to one row only, ranks entropy in the wrong direction, and samples from the distribution even at temperature 0. So llada_ref generate re-implements LLaDA's generate.py (temperature 0, low-confidence remasking) on top of llama.cpp's logits, and the result is compared token by token with mojo-dllm run. For Dream, llada_ref generate-dream does the same with Dream's diffusion_generate.

A diffusion loop can amplify small differences: when two candidates are nearly tied, the runtimes can commit different tokens at one step, and every later step then sees a different canvas. The table's first difference is a position in the final output; it does not identify the denoising step or the cause of the mismatch. Exact token agreement is specific to the measured prompt and configuration.

What this section covers: two prompts, 32 and 64 generated tokens (the benchmarks use 128), temperature 0, 32 steps, with low-confidence remasking for LLaDA and entropy ranking for Dream. The tables below give the count of equal tokens and the first difference for each model; on the speculative-decoding prompt several are well short of a full match, and Dream-v0's code prompt diverges within its first few tokens. Not covered: Gumbel noise (temperature above 0), random remasking, Dream's other remasking algorithms, classifier-free guidance (cfg_scale), alg_temp, and generation at 128 tokens. --threshold and --dual-cache have a paired GSM8K comparison instead (Benchmarks).

LLaDA-1.5.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323224 / 32position 21
Write a short Python function that checks whether a number is prime.643264 / 64none

Evidence: bench/parity/2026-10-30-llada-LLaDA-1.5.Q4_K_M.json (mojo-dllm 5c82e8c).

LLaDA-8B-Instruct.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.32328 / 32position 5
Write a short Python function that checks whether a number is prime.643264 / 64none

Evidence: bench/parity/2026-10-03-llada.json (mojo-dllm 0b3e17a).

LLaDA-8B-Instruct.Q4_K_M.gguf, GPU

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.32328 / 32position 5
Write a short Python function that checks whether a number is prime.643264 / 64none

Evidence: bench/parity/2026-10-09-llada-gpu.json (mojo-dllm cedc0b5).

DiffuCoder-7B-cpGRPO.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323222 / 32position 17
Write a short Python function that checks whether a number is prime.643263 / 64position 14

Evidence: bench/parity/2026-10-30-dream-DiffuCoder-7B-cpGRPO.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323219 / 32position 18
Write a short Python function that checks whether a number is prime.643213 / 64position 13

Evidence: bench/parity/2026-10-30-dream-Dream-Coder-v0-Instruct-7B.Q4_K_M.json (mojo-dllm 5c82e8c).

Dream-v0-Instruct-7B-Q4_K_M.gguf

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323224 / 32position 22
Write a short Python function that checks whether a number is prime.643243 / 64position 3

Evidence: bench/parity/2026-10-03-dream.json (mojo-dllm 0b3e17a).

Dream-v0-Instruct-7B-Q4_K_M.gguf, GPU

Promptgeneratedstepssame tokensfirst difference
Explain speculative decoding in two sentences.323226 / 32position 23
Write a short Python function that checks whether a number is prime.643228 / 64position 1

Evidence: bench/parity/2026-10-09-dream-gpu.json (mojo-dllm cedc0b5).

Reproduce

The added LLaDA-1.5, Dream-Coder-v0-Instruct-7B and DiffuCoder-7B-cpGRPO records also contain CPU smoke output, peak RSS, prompt token IDs, GGUF SHA256 and pinned Hub revisions. Their metadata audits compare vocabulary, layer count, context length and RoPE theta with the source HF configs. These runs use four CPU threads and llama.cpp 836d57176dc699a726c55418e4f96b8ca628e1bf; they add no GPU evidence.

LLaDA-1.5 and Dream-Coder carry the same chat templates as their HF tokenizer configs. DiffuCoder-cpGRPO has no HF chat template, while its GGUF uses the Dream-Coder template. Its HF pad token is <|dlm_pad|> (151667), but its GGUF padding ID is 151643. The smoke records explicitly note whether that pad token or a mask token appeared in the text. DiffuCoder's HF license is apple-amlr, despite the GGUF license field saying unknown; obtain it separately and pass a local file.

Build llama.cpp at the revision above with GGML_CUDA=OFF, then set LLAMA to that checkout before compiling the reference:

c++ -O2 -std=c++17 tools/ref/llada_ref.cpp -I$LLAMA/include -I$LLAMA/ggml/include \
    -L$LLAMA/build/bin -lllama -lggml -lggml-base -Wl,-rpath,$LLAMA/build/bin -o build/llada_ref
uv run --only-group fixtures python bench/parity.py --arch llada --model models/LLaDA-8B-Instruct.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --model models/Dream-v0-Instruct-7B-Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch llada --threads 4 --model models/LLaDA-1.5.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --threads 4 --model models/Dream-Coder-v0-Instruct-7B.Q4_K_M.gguf
uv run --only-group fixtures python bench/parity.py --arch dream --threads 4 --model models/DiffuCoder-7B-cpGRPO.Q4_K_M.gguf
uv run --no-project python scripts/bench_table.py --write

llada_ref refuses a token, a row or a count that is not an integer, a row outside the tokens, a token outside the vocabulary, an output it cannot write and any non-finite logit in a row it reads. Each refusal prints a line that begins llada_ref: and exits non-zero (1 for the arguments, 2 for the model or a file, 3 for llama.cpp or a NaN). It writes OUT.part and moves it to OUT only after every row passed, so a failed run leaves no file under the name bench/parity.py reads. tests/python/test_llada_ref.py builds it with the host compiler against a stand-in for llama.cpp and runs each refusal.

The GPU tables need --device gpu and a second reference. Compile llada_ref once more against the CUDA build of llama.cpp (build-cuda/, see Development), name the result build/llada_ref_cuda, and add --device gpu --ref2 build/llada_ref_cuda to the bench/parity.py commands.

DiffusionGemma

Mainline llama.cpp cannot load a DiffusionGemma GGUF. The reference is the branch of ggml-org/llama.cpp#24423, built at one commit by scripts/build_dg_ref.sh (CPU only, no network after the fetch) into its own tree, $DG_LLAMA_SRC, beside the mainline checkout. The commit is the PINNED_COMMIT in that script, and tests/fixtures/gemma4-tokenizer-golden.provenance.json records the same one. tools/ref/dg_ref.cpp is linked against that build and writes the canvas logits of one denoising step: the unified [prompt | canvas] pass or the prompt K/V path (a prefill of the prompt, then one decode step), each without self-conditioning and with it, from a previous-step buffer given on the command line.

The model file is 16.81 GB and mojo-dllm keeps its own repacked copy beside the mapping the reference holds, so the two never run together. bench/parity.py runs them in turns and compares saved files:

scripts/build_dg_ref.sh
uv run --only-group fixtures python bench/parity.py --arch diffusion-gemma --phase ref \
    --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --llama-src "$DG_LLAMA_SRC"
# mojo-dllm writes build/dg-parity/<model>/caseN/<variant>.mojo.f32 (the forward pass is #64)
uv run --only-group fixtures python bench/parity.py --arch diffusion-gemma --phase compare \
    --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf

Each <variant>.mojo.f32 has a <variant>.mojo.f32.json beside it with the case_sha256 of its case (bench/parity.py dg_case_hash: prompt, prompt tokens and canvas) and the mojo_dllm_commit of the build that wrote it. The comparison needs the cases case0 to case{N-1} of the current prompt set and stops with a message while any mojo-dllm file or sidecar is missing, was written for another case, or comes from a different build than the others, and refuses a file that is byte-identical to the reference's. It writes evidence to bench/parity/ only when every variant is no worse than the worst case of the LLaDA comparison in bench/parity/2026-10-03-llada.json, on each of minimum cosine, argmax agreement, top-5 overlap and maximum relative RMS. The bound is read from that file, not typed. A fifth metric holds the largest logit of every row to the same relative error as LLaDA's rows reach: the final soft-cap pins that logit near 30, so the first four cannot see the cap removed. bench/check_dg_bound.py shows what the bound rejects.

Each case records how far the reference lies from itself: the prompt K/V path against the unified pass, and the unified pass again with the previous-step logits changed by one part in a million. On the two LLaDA prompts with a random canvas those distances exceed the LLaDA bound (the one-in-a-million change moved the output by a median relative RMS of 0.18 to 0.26). No LLaDA spread was measured for comparison. The cause is untested: expert routing flips (8 of 128 experts per token) carried to every row by bidirectional attention are a hypothesis. A failed comparison prints these distances beside the verdict. No DiffusionGemma result is recorded yet, so the tables above hold none.

The tokenizer goldens in tests/fixtures/gemma4-tokenizer-golden.tsv fix the encoding, and tests/test_tokenizer_gemma4.mojo checks the Mojo tokenizer against them. Decoding the ids through the GGUF vocabulary restores 53 of the 54 strings; the string holding a literal U+2581 does not come back, because decoding turns every U+2581 into a space.