Development plan

Written 2026-10-03 from the draft spec (docs/spec.md) and three research passes: the Mojo 1.1 toolchain, the diffusion-LLM landscape, and the house style of the author's other repositories. Where research contradicted the spec, the correction is recorded here and the code follows the correction. The plan predates the code: where the code later moved, the text below carries the as-built wording, and Architecture maps what shipped.

Corrections to the spec

Spec saysReality (verified)Consequence
Q4_K_M support, optionally Q8_0/F16A Q4_K_M GGUF of LLaDA-8B holds Q4_K (incl. token_embd), Q6_K (attn_v/ffn_down in half the layers, output) and F32 normsv0.1 needs F32 + Q4_K + Q6_K
mojoproject.tomlmagic is gone; Mojo installs from PyPIpin mojo==1.1.0, max-core==26.6.0 in pyproject.toml, commit uv.lock
mojo testremoved; TestSuite, each test file built with mojo build and run as a binary; an empty suite exits 0CI pins the test count
llama.cpp as generation referenceits diffusion sampler skips CFG/alg_temp, noises one row, inverts entropy, samples at temp 0llama.cpp is the logit reference; token parity uses the LLaDA generate.py algorithm re-implemented over llama.cpp logits
diffuse-cpp as drop-in comparisonown diffuse GGUF schema, no tokenizer, unverified paritybenchmarked with its own file, on token IDs
RoPEGGUF converter un-permutes Q/K: GGUF uses adjacent-pair (NORM) RoPENORM RoPE when reading GGUF
UnsafePointer, fn, parallelize in std1.1 removed fn/alias; parallelize moved to max.algorithmwrite 1.1 Mojo only

Rows 3, 6 and 7 record assumptions from the research passes. docs/spec.md uses fn, and names none of mojo test, UnsafePointer, parallelize or a RoPE variant.

Model facts used (LLaDA-8B-Instruct, GGUF llada)

32 layers, d = 4096, 32 heads × 128 (MHA), FFN 12288 SwiGLU, RMSNorm ε = 1e-5, RoPE θ = 500 000, vocab 126 464, mask 126 336, BOS 126 080, EOS 126 081, EOT 126 348, non-causal, untied output. Tokenizer: byte-level BPE (gpt2), pre-tokenizer bailingmoe.

Compute design

By default a denoising step runs the full transformer over the whole canvas (prompt + generation, L ≈ 100 to 160 tokens), so linear layers are GEMMs, not GEMVs. Cost per step ≈ 2 × 7.0·10⁹ × L FLOPs, plus 2 × 0.5·10⁹ per row whose logits are computed (LLaDA-8B): compute-bound, not bandwidth-bound.

  • Activations are quantized per token to int8 with one f32 scale per 256 values (the Q8_K scheme), plus the sum of each group of 16 values as a float.
  • At load, each Q4_K and Q6_K matrix is repacked so that a 256-bit register holds 4 consecutive weights of each of 8 rows. The two types have their own layouts and tile functions; The quantized GEMM has both.
  • The inner product uses vpdpbusd (AVX-VNNI) when the CPU has it, and pmaddubsw plus pmaddwd otherwise. One vpdpbusd multiplies 8 rows by 4 activation bytes, and no horizontal sum is needed until the row ends.
  • The LM head runs only on the masked positions the sampler can commit in the step: the current block for LLaDA (at most --block-length rows, 32 by default) and every masked position of the canvas for Dream (at most --max-tokens rows), not all L rows. This is a diffusion-specific saving that a generic runtime does not take.
  • Threads split output rows into many small chunks so the 4 P-cores and 4 E-cores of a hybrid CPU balance.

Correctness ladder

  1. Unit: GGUF parse (synthetic files generated by scripts/gen_fixtures.py and committed under tests/fixtures/), Q4_K/Q6_K decode against hand-built blocks, quantized dot vs f32 dot, RMSNorm, RoPE, softmax, attention, scheduler transfer counts against generate.py.
  2. Tokenizer: golden token IDs produced by llama-tokenize on a fixed corpus.
  3. Logits: a small C++ tool over libllama dumps logits for fixed token sequences; mojo-dllm is compared on cosine similarity, relative RMS error and argmax agreement. Correctness has the measured values.
  4. Generation: the same tool runs LLaDA's generate.py greedy algorithm over llama.cpp logits; the tokens are compared one by one, and a run that diverges reports the first position that differs.

Benchmark protocol

Same machine, same prompts, same thread count, release builds, model load excluded from generation time and reported on its own. Configuration: gen 128, block 32, steps 32 (4 tokens per step, 32 forward passes), plus per-step latency. Three runs each, median reported, raw numbers in bench/results/*.json. The README table is rendered from that JSON and a gate fails when they disagree. No estimated numbers are published.

Repository gates

Sized to what this repo can break, copied from sysknife's reasoning:

GateGuards against
staged secret + employer-name scan (pre-commit)a key or the employer name in history
mojo format then git diff --exit-codeunformatted code (there is no check mode)
build with -Werrordeprecated APIs piling up across Mojo releases
tests with a pinned counta suite that selects 0 tests and still exits 0
shellcheck, yamllint, markdownlintbroken scripts and workflows
action pins verified against tagsa floating @v4 or a mislabelled SHA
README numbers equal bench/resultshand-typed benchmark claims
mdBook build + internal link checkbroken docs

Development lists the full set, including the launcher tests, the GPU suite, the no-GPU check and two self-tests.

Local first: scripts/ci-local.sh runs everything; pre-commit runs the fast set, pre-push the full set. GitHub Actions runs the same gates on push and PR, except that it builds the GPU suite without running it, split into three jobs (.github/workflows/ci.yml). .github/workflows/docs.yml deploys the book to GitHub Pages on pushes to main that touch docs/, book.toml, theme/ or assets/.

Milestones

Deliverable
M0repo, CI, logo, docs skeleton
M1mojo-dllm inspect model.gguf
M2Q4_K/Q6_K decode, quantized GEMM, tests, microbenchmark
M3RMSNorm, RoPE, attention, SwiGLU
M4LLaDA forward, logit parity with llama.cpp
M5tokenizer, scheduler, mojo-dllm run
M6threading and SIMD tuning, benchmark report vs llama.cpp and diffuse-cpp
M7NVIDIA backend (--device gpu), shipped in 0.2.0
M8Dream-7B, shipped in 0.1.0
laterblock diffusion; inter-step caching beyond --dual-cache

M0 to M6 and M8 shipped in 0.1.0. M7 shipped in 0.2.0.

The repository went public after M6, with parity documented and the license of shipped binaries reviewed (Modular runtime libraries are not Apache; NOTICE records the terms).