Development

Setup

uv sync --frozen                      # the pinned Mojo toolchain, in .venv
scripts/ci-local.sh --install-hooks

scripts/ci-local.sh also needs shellcheck, markdownlint-cli2 (npm install --global markdownlint-cli2@0.23.2), mdBook 0.4.40 and an authenticated gh. A missing tool fails its step; it never skips it.

The hooks run scripts/ci-local.sh --fast before each commit (hygiene scan, mojo format, -Werror build) and the full set before each push. The pre-commit build runs before its commit exists, so its binary records the parent HEAD, with a -dirty suffix when the commit changes src/. The launcher rebuilds when a source file is newer than the binary, or when build/mojo-dllm.stamp (written by the launcher's own build, not by scripts/build.sh) is missing or names another Mojo version or GPU architecture. After a hook build with a matching stamp the launcher sees a current binary and keeps reporting the parent commit: run scripts/build.sh after committing to stamp the new HEAD.

The gates

scripts/ci-local.sh runs the gates of .github/workflows/ci.yml and, on a machine with a GPU, the GPU tests that CI only builds:

StepCommandFails when
hygienescripts/check_hygiene.sh --trackeda credential pattern (a provider token, a private key block, a database URL with a password) or the author's employer name appears in tracked text, or a tracked Markdown file outside bench/results/ holds an em or en dash; file names are read NUL separated, so a name with non-ASCII or special characters is scanned too)
formatscripts/check_format.sh (mojo format -q src tests bench, comparing every file before and after)mojo format changes any file, an untracked one included (Mojo has no check mode)
buildscripts/build.shany compiler warning (-Werror)
embedded build commitscripts/check_build_id.shthe built binary does not report the expected commit, or a build without Git metadata does not report unknown
testsscripts/run-tests.sha test fails, a file runs zero tests, the total differs from tests/test-count.txt, or a file's count differs from tests/test-counts.txt
launcher testsuv run --locked --only-group test pytest tests/python -qa test of the ./mojo-dllm launcher fails
script lockuv lock --script mojo-dllm --checkmojo-dllm.lock is missing, or the # dependencies header of ./mojo-dllm no longer matches it. After you change the pin, run uv lock --script mojo-dllm and commit the lock
shellcheck, yamllint, markdownlintshellcheck --severity=warning scripts/*.sh .githooks/*; uv run --frozen --only-group lint yamllint --strict .github; markdownlint-cli2 "*.md" "docs/**/*.md" "assets/*.md" ".github/*.md" "#node_modules"any of the three reports a finding
claimsuv run --no-project python scripts/bench_table.py --checka table in the README, docs/benchmarks.md or docs/correctness.md differs from what bench/results, bench/parity, bench/quality and bench/eval render (a missing table marker, target file or quality or eval evidence passes, #141; an unknown mode exits 0, #142)
docsmdbook build + scripts/check_links.pya relative link is broken
action pinsscripts/verify-action-pins.sh (needs an authenticated gh)a pinned SHA is not the tag its comment names
eval self-testuv run --no-project python bench/eval_gsm8k.py --self-testanswer extraction or the paired statistics break, or gsm8k-200.jsonl is not 200 rows with numeric answers
checker self-testuv run --frozen --only-group fixtures python scripts/check_weights.py --self-testa safetensors or index guard in check_weights.py stops rejecting its case, or stops naming it
no GPUscripts/check_no_gpu.sh--device gpu without a GPU does anything but exit 1 with a message
visual optionsscripts/check_visual.shan invalid --visual-interval does not exit 1, or --json with --visual does not exit 2, with a message naming the flag before model loading
fixturesuv run --frozen --only-group fixtures python scripts/check_fixtures.pyscripts/gen_fixtures.py or scripts/gen_moe_fixture.py writes a file that differs from the tracked one (they run into a scratch directory), a file that is not tracked, or a tracked fixture that no generator writes and that is not a tokenizer fixture, a tokenizer golden or golden-generation.tsv
GPU test countscripts/check_gpu_test_count.shthe def test_ functions of tests/gpu/test_*.mojo number other than tests/gpu/test-count.txt pins or differ per file from tests/gpu/test-counts.txt, or a file defines none; needs no GPU, runs in CI and ci-local.sh
GPU testsscripts/run-gpu-tests.sh (local only)a GPU kernel or the GPU forward pass disagrees with the CPU or the f32 reference, or the count differs from tests/gpu/test-count.txt or tests/gpu/test-counts.txt

GitHub's runners have no GPU, so CI builds the GPU suite without running it. ci-local.sh runs it when nvidia-smi lists a GPU and otherwise builds it and prints a SKIP line for the run, never a PASS; run-gpu-tests.sh on its own exits 1 without a GPU.

The launcher tests run pytest -rs, so every skip is listed in the summary, and tests/python/skip_policy.py fails the run on a skip that tests/python/allowed-skips.txt does not name (a test id and the text of the reason). Add a line there in the commit that adds a skip. --locked makes uv run fail when uv.lock no longer matches pyproject.toml; --frozen does not compare them.

Branch protection on main requires four checks: hygiene, mojo and docs of the CI workflow and trufflehog of the secret scan. The job build of the "Deploy docs to GitHub Pages" workflow skips while the repository is private, and a skipped job satisfies a required check, so it must not be required. CI already runs the same mdbook build in docs.

The test count is pinned because Mojo's TestSuite exits 0 after running zero tests. If you add tests, run UPDATE_TEST_COUNT=1 scripts/run-tests.sh (for tests/gpu/, UPDATE_TEST_COUNT=1 scripts/run-gpu-tests.sh) and commit the new count with them.

Tests

scripts/run-tests.sh compiles each tests/test_*.mojo and runs it. Fixtures live in tests/fixtures/ and come from scripts, never from hand:

FixtureMade byChecks
kquants.gguf, bad-magic.gguf, tiny-llada.gguf, tiny-llada.expected.f32, tiny-dream.gguf, tiny-dream.expected.f32, malformed/, metadata/, gen_fixtures.manifestscripts/gen_fixtures.pyGGUF parsing and rejection, decoders, ModelConfig validation, the forward pass against a numpy reference
tiny-diffusion-gemma.gguf, diffusion-gemma/scripts/gen_fixtures.pythe diffusion-gemma config (seed 610061): a 2-layer model with one sliding and one global layer, 4 experts, the tensor names and types of the real file, and one file per refusal (metadata only, or zero-filled tensors)
tiny-diffusion-gemma.p0sc.case, .p9.case, .p9sc.case, .p150.case, .p150sc.case (tiny-diffusion-gemma.*.case)scripts/gen_fixtures.py (the forward pass is scripts/dg_numpy.py)the tokens and numpy logits that logits on the tiny DiffusionGemma file must reproduce, for a prompt of 0, 9 and 150 tokens, with and without self-conditioning (sc)
entropy-rows.tsvscripts/gen_fixtures.py (seed 65065, so no other fixture changes)logits rows with the numpy f64 argmax, entropy and inverse-CDF draw that tests/test_entropy_bound.mojo pins
utf8-cases.tsvscripts/gen_fixtures.pybyte strings with the code points and the first invalid offset that Python's own UTF-8 decoder gives, for utf8_decode
dequant-types.gguf, unsupported-embedding.ggufscripts/gen_fixtures.pyQ8_0/Q5_K/BF16/Q5_0 references (seed 340034) and embedding rejection before CPU/GPU allocation (no RNG)
llada-tokenizer.gguf, llada-tokenizer-golden.tsv, llada-tokenizer-golden.provenance.json, dream-tokenizer.gguf, dream-tokenizer-golden.tsv, dream-tokenizer-golden.provenance.jsonscripts/gen_tokenizer_fixture.py (--golden-only rewrites the golden and its provenance)tokenizer ids against llama-tokenize. Each provenance file records the llama.cpp commit, the model file and its sha256, the command and the golden's sha256. The two goldens were committed with the first import, and regenerating them with llama.cpp 2e715c0639 on 2026-11-09 gave the same bytes
gemma4-tokenizer.ggufscripts/gen_tokenizer_fixture.py ... gemma4the 262,144 tokens of the DiffusionGemma file and the 472 of its 514,906 merges that apply to the 54 golden strings, tokenizer keys only and without the scores (5.5 MB)
gemma4-tiny/scripts/gen_fixtures.pya 279-token gemma4 vocabulary whose ids can be worked out by hand, and variants with one thing wrong or unusual each (a missing byte token, a merge with no space, mismatched counts, nested special tokens, an unknown-type token, an end-of-turn token the vocabulary lacks)
gpt2-tiny/scripts/gen_fixtures.pya six-token byte-level BPE vocabulary with padding, unknown and EOS ids and no end-of-turn token: the ids that LLaDA's and Dream's files leave out
gemma4-tokenizer-golden.tsv, gemma4-tokenizer-golden.provenance.jsonscripts/gen_tokenizer_fixture.py ... gemma4 (--golden-only rewrites only these)DiffusionGemma tokenizer ids against llama-tokenize from the ggml-org/llama.cpp#24423 branch at the commit scripts/build_dg_ref.sh pins; the provenance file records that commit, the model file's sha256 and the command
golden-generation.tsvUPDATE_GOLDEN=1 build/tests/test_goldentiny LLaDA/Dream token sequences, forward-pass counts, early stopping and per-step commits
src/mojo_dllm/tokenizer/unicode_tables.mojoscripts/gen_unicode_tables.pycode point classes from llama.cpp's own tables
moe-q5.bin, moe-q8.bin, moe-q4-q5.bin, moe-q4-q8.binscripts/gen_moe_fixture.pydeterministic raw expert tensors for scalar MoE comparisons, including mixed activation block sizes
GGUF/source tensor shape reportscripts/check_weights.py (--self-test)LLaDA and Dream name mapping and safetensors header shapes; run with uv run --only-group fixtures python scripts/check_weights.py MODEL.gguf INDEX.json (the second argument may be one .safetensors file)

tests/gpu/test_gpu.mojo holds the GPU tests: each kernel against the CPU function it replaces, and the GPU forward pass on both tiny models against the f32 reference.

The tiny model is 2 layers of hidden size 256 with random K-quant blocks. Its expected logits come from a float numpy forward pass over the weights that the gguf package dequantizes, so the Mojo forward pass is checked against the reference formats, not against itself.

Sampler goldens

tests/test_golden.mojo pins current behavior on the tiny fixtures. These outputs come from mojo-dllm itself: they detect changes, not independent correctness. The numpy forward-pass tests and real-model parity runs provide that evidence. No model download or GPU is needed.

Run from the repository root:

bash scripts/build.sh tests/test_golden.mojo build/tests/test_golden
build/tests/test_golden
# Only after reviewing an intentional behavior change:
UPDATE_GOLDEN=1 build/tests/test_golden

Only UPDATE_GOLDEN=1 writes the file, and scripts/run-tests.sh drops that variable from the environment (and says so) unless you run scripts/run-tests.sh --update-golden: the test rewrites the golden and then compares against what it wrote, so an exported variable would make every run pass. Normal runs reject missing, empty, duplicate, unknown or incomplete cases and compare tokens and schedule metadata exactly. A mismatch names the case, the first differing token position (including the prompt), both token lists and both complete rows. A golden diff in a PR needs a sentence explaining why behavior changed, and test_model.mojo must still pass. Do not regenerate merely to silence a failure on another machine; inspect the differing logits and select a case without an unstable near-tie.

The 14 cases cover LLaDA block geometry, uneven transfer counts, serial commits, random remasking, temperature and early stop, plus Dream's three confidence rules, uneven timesteps, nondefault eps and early stop. LLaDA uses prompt [5, 17, 300]; Dream uses [448, 50, 134] because the original prompt gave identical all-49 outputs for D1 and D5. To avoid duplicate token lists, L5 uses seed 1, D2 uses eps=0.2, D3 uses a 12-token canvas and D4 uses eps=0.1. D1/D5 retain the same prompt, length and steps and differ only in eps. L8/D6 use fixed stop ids 347/49 taken from the baseline outputs, never selected at runtime. All cases disable threshold and dual-cache acceleration.

Heavy jobs

A job that holds more memory than the machine can spare makes the kernel's global OOM killer pick a victim. With systemd's default OOMPolicy=stop, systemd then stops every process in the victim's scope, and a job started from a terminal lives in that terminal's scope: the shell, the editor session and every other session in the window end with it. A measured run of DiffusionGemma-26B on the CPU peaked at about 25.6 GiB resident, and on a 30 GiB machine with a 2 GiB swapfile that run ended in exactly this way.

scripts/capped.sh starts a command in a scope of its own, under a hard cap, so only that scope can fail:

scripts/capped.sh --mem 26G -- build/mojo-dllm run --model model.gguf --prompt "hello"

It sets four things, each for a reason you can check in the documentation of the kernel's cgroup v2 memory controller and of systemd's resource control:

SettingEffect
MemoryMax=26Gthe hard cap (memory.max); the kernel OOM killer runs inside the scope when it is reached and kills nothing outside it
MemorySwapMax=0no swap, so a job over its cap dies instead of dragging the desktop through swap
OOMPolicy=killone OOM kill ends the whole scope (memory.oom.group), not half of it
CPUQuota=N00%only with --cpus N, so a job leaves cores for the desktop

It never sets MemoryHigh. Below the cap, MemoryHigh throttles a process that has nothing to reclaim until it stands still (state D, no progress); a job that needs its memory should be killed, not frozen.

The script also refuses to start a job in two cases:

  • /proc/meminfo reports less MemAvailable than --need (default: --mem). A cap larger than the available memory does not protect the machine, because the global OOM killer still runs. Exit status 75.
  • The systemd user manager cannot enforce a memory limit (no cgroup v2, or the memory controller is not delegated to it). It never runs the command uncapped. Exit status 69.

--dry-run runs both checks and prints the systemd-run command without running it. When a capped job dies with exit status 137, the cap was most likely reached; journalctl -k | grep oom-kill tells the cases apart: constraint=CONSTRAINT_MEMCG is the cap working, while constraint=CONSTRAINT_NONE with global_oom means the whole machine ran out of memory.

The test suite checks the limits the kernel reports for a capped scope (memory.max, memory.swap.max, memory.oom.group). The test that kills a process at its cap runs only with CAPPED_TEST_KILL=1, because GNOME's gsd-housekeeping answers every such kill with an "Application Stopped" notification.

For a container, set the same two limits on the container: podman run --memory 12g --memory-swap 12g (--memory-swap counts memory plus swap, so equal values mean no swap).

What does not isolate a job on its own:

  • ulimit -v limits address space, which counts mapped files such as a 16 GB model that is read lazily, and it does not stop a process that stays below the limit but takes the machine's last free page.
  • systemd-oomd reacts to memory pressure and swap use over seconds, and a job that fills memory in one allocation burst can reach the kernel's killer first.
  • A swapfile only delays the kill and slows the machine while it lasts.

References: the kernel's Control Group v2 memory interface, systemd's resource control and scope units (OOMPolicy=).

Benchmarks

uv run --no-project python bench/run_bench.py --config bench/config.json --reps 3
uv run --no-project python scripts/bench_table.py --write

bench/config.json runs every CPU runtime in one pass and names the model files and the llama.cpp and diffuse-cpp checkouts. The committed CPU benchmark tables came from bench/config-main.json, bench/config-diffuse.json and bench/config-dream.json; the thread sweep came from bench/config-fair-cpu.json. scripts/bench_table.py refuses a runtime that appears in two result files, so move the files a new run replaces into bench/results/archive/ before --write. The harness waits for an idle CPU before each run, interleaves the runtimes, switches the power profile to performance for the duration, and writes a result file with every raw run. On a laptop it measures on mains power only: unplugged, it pauses before the next run and prints that it is waiting for the charger, and a run the charger left midway is repeated. The eval, the quality table and scripts/gpu_profile.py do the same through bench/power.py; --allow-battery measures on battery instead and records it. The benchmark harness kills any process named pytest while it runs, because test suites from other work on the development machine kept starting mid-benchmark. Set "kill_pytest": false in a config to turn that off; the GPU configs (bench/config-gpu.json, bench/config-dream-gpu.json) do, since a GPU run leaves the CPU nearly idle. They compare --device gpu with llama.cpp's diffusion example built with CUDA in build-cuda/:

cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build-cuda --target llama-diffusion-cli

GPU kernels

scripts/gpu_profile.py traces one run --device gpu with Nsight Systems (nsys --trace=cuda, which ran here without root although the driver keeps hardware counters for admins) and writes the device time per forward pass of each kernel group to bench/profiles/. It does not synchronize the stream, unlike MOJO_DLLM_GPU_PROFILE=1. --root points it at another checkout's build for before/after comparisons, and it reports a median-based total next to the plain sum, because a laptop GPU's clocks move under sustained load. --runtime llama.cpp --seq-len N traces llama.cpp's CUDA llama-diffusion-cli on the same prompt instead (it replays its forward pass as a CUDA graph, so the trace expands graph nodes), with the same kernel groups. bench/bench_gpu_gemm.mojo times each LLaDA weight shape alone, adding up the per-forward total from the tensor types in the GGUF.

On the development laptop the GPU is capped at 30 W and its clocks drop under sustained load (bench/checks/2026-10-12-gpu-power-under-load.txt), so compare kernels with interleaved runs (A, B, A, B) in one session rather than against a number from another day.

Comparing GPU dispatch changes

bench/compare_gpu.py compares two already-built checkouts through run_bench.run_mojo, interleaving A/B and B/A rounds on one assigned GPU. It excludes a warmup pair per generation length, records every raw run, and rejects failures or mismatched forward counts and canvas sizes. It leaves the system power profile and other processes alone. Keep each pair on the same GPU; another GPU can run a different model's independent comparison.

Both tools, and scripts/compare_logits.py, refuse a base and a new that are the same checkout, the same clean commit or byte-identical binaries, and record the SHA-256 of each build/mojo-dllm in the evidence, because a stale build carries the label of whatever commit is checked out. compare_gpu.py sets CUDA_DEVICE_ORDER=PCI_BUS_ID so that --gpu (an index or a GPU- uuid) names the same card to CUDA and to nvidia-smi, records the index and the uuid read back from nvidia-smi, and runs the isolation preflight and a window around each run of bench/isolation.py (--allow-noisy continues after a failed preflight). check_gpu_dispatch.py records CUDA_VISIBLE_DEVICES and CUDA_DEVICE_ORDER as it found them, so export CUDA_DEVICE_ORDER=PCI_BUS_ID there too.

python3 bench/compare_gpu.py --base ../base --new . --model MODEL.gguf \
  --gpu GPU-UUID --lengths 128,256,512 --reps 3 --prompts 3 \
  --out bench/results/dispatch/run.json
CUDA_VISIBLE_DEVICES=GPU-UUID python3 bench/check_gpu_dispatch.py \
  --base ../base --new . --model MODEL.gguf --arch llada \
  --out bench/checks/dispatch-exact.json

Use --arch dream for Dream. The exact check uses that model's mask token, compares all finite logit values byte for byte at several canvas sizes, and compares complete generated token lists. It checks preservation of the base GPU computation, not correctness against an independent runtime. The timing comparison runs without the synchronizing MOJO_DLLM_GPU_PROFILE mode.

Quality

bench/eval_gsm8k.py answers a fixed 200-problem GSM8K subset (bench/eval/gsm8k-200.jsonl, see bench/eval/SOURCE.md) and compares two arms problem by problem: accuracy difference with a 95% interval and McNemar's exact test. Any change that is not exact (a sampler that commits more per step, a K/V cache) needs a comparison against the one-token-per-step baseline before it becomes a default.

uv run --no-project python bench/eval_gsm8k.py run --model M.gguf --arm baseline
uv run --no-project python bench/eval_gsm8k.py run --model M.gguf --arm candidate -- --some-flag
uv run --no-project python bench/eval_gsm8k.py compare baseline candidate
uv run --no-project python bench/eval_gsm8k.py compare baseline candidate --limit 50

compare writes bench/eval/<date>-<a>-vs-<b>.json, and scripts/bench_table.py turns those files into the accuracy table in Benchmarks. Time per problem depends on what else the machine runs, so --limit N compares each arm's first N problems in run order and writes a -firstN file; the table takes its times from that file. Run that round with nothing else on the machine.

Toolchain

Mojo is pinned to 1.1.0 and max-core to 26.6.0 in pyproject.toml, locked in uv.lock; both come from PyPI. Bump them together by editing the two pins, running uv lock, and committing that on its own with a green run: Mojo 1.1 removed fn, alias and @parameter closures, and each release has moved APIs this code uses. max-core supplies max.algorithm.parallelize and the GPU API (max.gpu, max.gpu.host). The Mojo wheels need glibc 2.34 or newer.

The Python scripts run through uv too. uv run --no-project python script.py runs a script that uses only the standard library. The fixture generators and bench/parity.py need numpy and gguf, pinned in the fixtures group: uv run --only-group fixtures python scripts/gen_fixtures.py.

Releasing

A release is a version bump, a dated CHANGELOG section, a tag and a GitHub release. Pushing the tag and creating the release need write access to the repository. X.Y.Z stands for the new version.

  1. Check that CI is green on the commit of main that you ship.

  2. In CHANGELOG.md, rename ## [Unreleased] to ## [X.Y.Z] - YYYY-MM-DD and add an empty ## [Unreleased] above it.

  3. Set the version in the three places that carry it: version under [project] in pyproject.toml, comptime VERSION in src/main.mojo, and uv.lock. Run uv lock for the third; its diff is one line, the version of the mojo-dllm entry. Result files under bench/ record the version of the binary that produced them; leave them.

  4. Run scripts/ci-local.sh. On a machine with an NVIDIA GPU also run scripts/run-gpu-tests.sh, because CI builds the GPU suite and does not run it.

  5. Commit the files that steps 2 and 3 changed (CHANGELOG.md, pyproject.toml, src/main.mojo, uv.lock) with the subject Release X.Y.Z, together with any other file that still names the old version or milestone as current (for example docs/plan.md). Get the commit onto main.

  6. Tag that commit. The existing tag v0.1.0 is annotated and not signed:

    git tag -a vX.Y.Z -m "mojo-dllm X.Y.Z"
    

    For a signed tag use git tag -s vX.Y.Z -m "mojo-dllm X.Y.Z" and check it with git tag -v vX.Y.Z.

  7. Push the tag with git push origin vX.Y.Z. Check from the remote side that it arrived: git ls-remote origin 'refs/tags/vX.Y.Z^{}' prints the commit that git rev-parse 'vX.Y.Z^{commit}' prints.

  8. Create the GitHub release with the CHANGELOG section as its notes. Skip this step once the release workflow exists (#131): pushing the tag in step 7 then creates the release.

    notes="$(mktemp)"
    awk -v v="X.Y.Z" '/^## \[/ { p = (index($0, "## [" v "]") == 1); next } p' CHANGELOG.md > "$notes"
    test -s "$notes"
    gh release create vX.Y.Z --verify-tag --title "mojo-dllm X.Y.Z" --notes-file "$notes"
    

    test -s stops the sequence when the section is missing or empty. gh release view vX.Y.Z shows the result.