MSc thesis · University of Amsterdam · July 2026
A diffusion language model does not write left to right. It starts from a fully masked sequence and reveals tokens over a fixed number of steps, choosing at each step which position to fill. That choice is a second thing you can train. This thesis asks whether post-training should change the model’s weights, the order it writes in, or both.
Supervisor dr. B. Wouters. Second reader Prof. dr. C. Diks. Econometrics, Data Science track. Trained on the Dutch national supercomputer under a fixed compute grant.
Train a LoRA adapter with reinforcement learning, then merge it. A decoding policy that was fitted against the pre-merge weights does not survive the move: HumanEval pass@1 falls from 46.6 to 43.3 under direct transfer. Re-fit the same policy on the merged weights and it reaches 61.8. The thesis calls the combined procedure co-adaptation, and the drop under direct transfer is the evidence that the policy is tied to a particular set of weights rather than to the task.
Two controls limit how much of that can be claimed. A learned schedule decodes with more network function evaluations than the vanilla configuration, so a like-for-like comparison has to hold the decoding budget fixed; once it does, only Dream keeps a positive margin, and LLaDA does not. A continuation control attributes part of the recovery to the additional training the re-fit involves rather than to re-adaptation alone. Both appear in the results tables.
GRPO needs a probability ratio between the current policy and the one that generated the sample. For an autoregressive model that ratio is a product of per-token conditionals and is read straight off the forward pass. A masked diffusion model has no such factorisation: the exact sequence likelihood is intractable, because the model defines a distribution over denoising trajectories rather than over left-to-right continuations.
diffu-GRPO works around this with a one-step estimator of the sequence log-probabilities under
random prompt masking. That estimator is the load-bearing approximation of the whole method,
and it is the reason results from the autoregressive RL literature cannot be assumed to carry
over. The advantage itself follows the variant that drops the standard-deviation denominator,
so it reduces to ri − r̄, and the optimiser step is skipped
entirely for any group whose advantages are all zero.
That last detail is where the thesis’s main diagnostic comes from.
Call a group effective when the rewards inside it are not all equal, so
σr > 0 and the advantage is non-zero. Define
This is the fraction of sampled groups that can produce a reward-driven gradient at all. It costs nothing to log and it separates three failure modes that otherwise look identical from the loss curve.
| Condition | Effect on ρeff | Why |
|---|---|---|
| Binary reward, early training | Small | Almost every group is uniformly wrong, so every completion scores zero and the spread is zero. |
| Partial-credit reward | Should rise | Test-pass fraction and AST similarity break ties before any completion is exactly correct, so a group of near-misses still carries signal. |
| Already RL-tuned model | Small, even on problems it cannot solve | The advantage vanishes when rewards agree, which happens when every completion is wrong and when every completion is nearly the same. A sharpened policy samples with too little diversity to disagree with itself. |
The third row is the finding. DiffuCoder-cpGRPO arrives having already been through coupled-GRPO, and it resists every method tried here. Its within-group reward spread runs 0.07 to 0.16 against 0.20 to 0.41 for the reinforcement-learning-naive models, leaving 11 to 15 percent of groups effective. Running the identical recipe on its supervised-only sibling gives 2.4 times the spread and 2.6 times the effective groups, which points at the prior RL stage rather than at the architecture, since the two checkpoints share one.
The autoregressive literature documents a related phenomenon under the name entropy collapse, where group-relative optimisation sharpens a policy until its samples agree and its exploratory range narrows. The thesis notes the resemblance and then declines to claim it: the relation to the accuracy outcome is associative, was not established by intervention, and one model family cannot settle whether it generalises.
Generation runs from a fully masked sequence back to clean text. Two quantities are free. The schedule, meaning which positions to commit and in what order, and the number of reverse steps, counted as network function evaluations.
They are coupled, and the coupling is the whole inference-time problem. Committing one position per step makes the model’s conditional-independence assumption exact and spends the most NFEs. Committing many positions at once empties the mask set in far fewer NFEs but forces an independence that need not hold, and the errors from prematurely co-committed tokens are what degrade the output. The schedule is therefore a direct trade of compute against the accuracy lost to committing too early, which is why every comparison in the thesis has to be budget-matched before it means anything.
An autoregressive model has no equivalent choice, and this is the degree of freedom that makes post-training a diffusion model a different problem rather than the same problem on unusual hardware.
The experiments ran on the Dutch national supercomputer under a fixed compute grant, plus rented H100 and H200 nodes when the grant ran low. A subset of evaluations ran on A100 and B200. None of that is a large cluster by industry standards, and the constraints shaped the experiment design more than the method did.
The H100 partition has four GPUs per node. Any run wanting eight has to cross a node boundary,
which turns a single-machine job into a distributed one and puts the gradient all-reduce on
the interconnect instead of NVLink. The submission script asks for
--nodes=2 --gpus-per-node=4 and one task per node, then launches four processes
per task.
The eight ranks talk over NCCL, which is what PyTorch selects on CUDA, so the gradient all-reduce crosses the InfiniBand fabric between the two nodes rather than staying on NVLink. Before any of that, the eight processes need a meeting point in order to form the group at all. The first hostname in the Slurm allocation becomes the rendezvous address, and the port is derived from the job ID rather than fixed, because two of my own jobs landing on the same node with the same hard-coded port will collide and hang. Each process learns which machine it is from Slurm rather than being told.
A login node can reach the network. The compute nodes cannot. Any code that lazily downloads a checkpoint, a tokenizer or a dataset on first use will sit and fail after the job has already spent queue time. So everything is staged onto the shared scratch filesystem first and the libraries are told, explicitly, that they are offline. Experiment tracking writes to disk and syncs later.
Jobs are gang scheduled: you get every node you asked for at once, or you wait. Asking for
two nodes rather than one means waiting longer for both to be free at the same time. The wall
clock is twelve hours, so anything longer has to checkpoint and resubmit, and a run that dies
at hour eleven with nothing written has cost eleven hours of a grant that does not refill.
The job prints nvidia-smi on every node before it launches anything, so a run
that is about to fail on a bad card fails in the first ten seconds instead of the last hour.
A fixed grant makes experiment design an allocation problem. Three seeds per cell rather than ten. Several conditions run at a single seed and labelled exploratory in every table rather than quietly averaged in. The factorial was pruned to the cells that could answer the question, and the thesis says which cells are thin instead of hiding them.
| Precision | Weights loaded in bfloat16. The trainer’s own mixed-precision flag is off, so this is bf16 throughout rather than bf16 compute with an fp32 master copy. With LoRA only the adapter parameters receive updates, which limits the exposure, but it is a real difference from the standard recipe and a real difference from the standard recipe. |
| Attention | flash_attention_2 where the model supported it, sdpa where it did not. The attention score matrix is never written to memory in full. |
| Adaptation | LoRA adapters through PEFT, so a fraction of a percent of the parameters are trained and an adapter can be merged into the base weights afterwards. Merging is what creates the stale-policy problem described above. |
| Optimiser | AdamW, betas 0.9 and 0.99, weight decay 0.1, gradient norm clipped at 0.2, learning rate 3e-5 on a cosine schedule with 100 warmup steps. |
| GRPO | Eight generations per prompt. The group mean is the baseline, so there is no critic network to keep stable. Completions capped at 256 tokens. |
| Data movement | Datasets and the model cache staged on shared scratch before the job starts. Checkpoints written to scratch, not to a home directory with a quota. |
Six primary contrasts were tested. After Holm correction, two survive: the HumanEval gains on Dream and on the DiffuCoder sibling, at p = 0.003. The LLaDA HumanEval gain does not survive correction, at p = 0.136, and is reported as suggestive. Every MATH-500 movement is null. The thesis states this in one sentence and then explains each one.
The model family is niche. Three of the mechanisms are not.
An adapter and anything fitted against it are one object. Merging a LoRA adapter into base weights is routine at serving time. Anything calibrated against the pre-merge model, a decoding policy here, but equally a router or a threshold, is silently invalidated by the merge. It cost 3.3 points of pass@1 and produced no error.
Group-relative methods need disagreement, not just difficulty. The advantage is computed against the group mean, so it vanishes when the samples agree. That happens when every completion is wrong and equally when every completion is nearly identical, which is what a previous round of RL produces. ρeff costs one counter to log and would have flagged the dead checkpoint before it consumed a share of the grant.
Denser supervision did not become better supervision. Partial credit raised the fraction of groups carrying a usable advantage, which was the mechanism it was introduced for, and the benchmark still did not move. Density is a property of the reward function. Whether the extra information is learnable is a property of the model and the data, and the two are measured separately.