Built to Be Read, Not Just Run
As MarkTechPost reports, Molt measures roughly 8.6K lines of RL code, counted by tracing the import graph from each framework's RL entry point. The same method tallies about 62K lines for verl, 25K for slime, and 7.2K for OpenRLHF. That deliberate compactness is the project's central pitch: agentic RL research is constant algorithm modification — new estimators, new pipeline stages, new rollout schemes — and in mainstream frameworks every change threads through layers of trainer, distributed-backend, and rollout glue. Molt aims to remove that friction.
The framework is Apache 2.0 licensed and ships with launch codes, Slurm scripts, and a prebuilt container. NVIDIA is clear, however, that the accompanying research paper positions Molt as research infrastructure rather than a production training service, and hardware is the real gatekeeper. The shipped recipes assume 2 nodes of 8 H100 GPUs, split 8 for training and 8 for rollout. That puts it within reach of frontier and frontier-adjacent labs, well-funded startups doing post-training, enterprise AI research groups, and academic groups with multi-node H100 or H200 access.
Composed, Not Forked
Molt's architecture composes three existing tools without forking any of them, so upstream improvements arrive as a container pin rather than a painful rebase. It uses Ray for placement and asynchronous queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for training. The runtime consists of an agent pool, a set of vLLM engines behind a request router, and a single trainable policy actor. A streaming pool keeps prompt groups in flight so the engines never drain while the actor trains.
A feature called partial rollout is central to efficiency. When the policy updates, Molt pauses the engines, broadcasts the actor's updated shards over NCCL directly to each engine, and resumes the retained requests rather than discarding them. That avoids the wasteful pattern of throwing away in-progress rollouts every time the model changes.
One Module, Two Agent Forms
An RL run in Molt names a single Python module that exports an AgentRunner, and everything else — including the reward function — is ordinary code. The framework supports two forms. With Env, the framework owns the LLM loop inside a Gymnasium-aligned `step()`, which fits the familiar reinforcement-learning pattern. With ChatAgent, the user owns the loop through a stock OpenAI or Anthropic SDK, making it straightforward to wrap an existing agent.
To bridge both, Molt launches a loopback server that speaks both wire protocols, and every request is decoded server-side into one token-exact accumulation. When a long-horizon agent compacts its context and rewrites the prefix — a common operation for agents that run for many turns — the server seals the current segment and opens a fresh one automatically. This is the kind of plumbing detail that determines whether an agentic RL run is correct in practice or just looks correct.
Three Correctness Invariants
Molt's design is organized around three correctness invariants that address the subtle failures of asynchronous agentic training. Token identity means sampled token ids define the trajectory, not a retokenized transcript — important because stitching text back into tokens can silently change the data. Policy-version semantics ensures trainable tokens keep their behavior-policy log-probabilities, with asynchronous use corrected per token behind a sequence-level gate. Forward consistency requires that the rollout engine and the training actor agree on model semantics.
That last invariant matters most for mixture-of-experts (MoE) policies, which are increasingly common. The rollout and training routers select experts independently, and small numerical differences can flip top-k choices, producing a mismatch between what was sampled and what is being trained on. Molt addresses this with rollout routing replay: vLLM returns its per-token expert ids, and the training forward pass replays those exact routing decisions rather than re-deriving them.
What It's For
The shipped recipes and use cases point to where NVIDIA expects Molt to be adopted: multi-turn tool-use agents, code-execution agents, vision-language environments (the framework includes a shipped geo3k recipe), LLM-as-judge reward loops, and on-policy distillation onto a smaller student model. These are precisely the workloads that have driven the surge in agentic RL across the industry, and each one is notoriously fiddly to get right under the partial-rollout, async-sampling conditions that real training imposes.
The broader signal is that NVIDIA is investing not just in the GPUs that train agents but in the software stack researchers use to build them. By keeping the codebase small and readable — and explicitly designing it so that an AI coding assistant can modify it — Molt reflects a bet that the future of agentic RL tooling will be co-developed with the models that use it.
Stay Ahead of AI
For more on the frameworks reshaping how frontier agents are trained, follow our hub for the latest AI developments.
Read more AI news →
