A Small-But-Wide Mixture-of-Experts Design
According to a detailed breakdown in MarkTechPost, Instella-MoE-16B-A3B is a decoder-only MoE model with 16 billion total parameters that activates only 2.8 billion per token. Each of its 27 layers uses 2 shared experts plus 6 routed experts selected from a pool of 64, with a hidden size of 2048, 16 attention heads, and a 128,896-token vocabulary. A Multi-Token Prediction objective is used during both pre-training and mid-training.
That sparse architecture — where a tiny slice of the network "wakes up" for each token — is the same efficiency logic behind the cheap-to-run open models that have reshaped the market over the past year. AMD trained the model on 7.1 trillion tokens drawn from open corpora including Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360, then ran a mid-training stage on Dolma3 Dolmino 100B across three data variants that were merged by weight averaging. A long-context stage stretches the window from 4K to 64K tokens using YaRN.
Two Systems-Level Innovations
The release carries two structural choices that AMD highlights as meaningful engineering wins. The first is Gated Multi-head Latent Attention (Gated MLA), which adds a lightweight learned output gate to Multi-head Latent Attention. A dedicated linear projection derives an input-conditioned gate that is applied multiplicatively before the output projection.
The second, FarSkip-Collective, is a connectivity scheme that passes outdated and partial activations into the MoE and attention layers, overlapping expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to a 39.2% reduction in time to first token when serving with expert parallelism. For a company pitching its hardware on price-performance, those are exactly the numbers a buyer wants to see.
Strong Benchmarks for a Fully Open Model
On the base checkpoint, Instella-MoE averages 76.7 across evaluations, which AMD calls the strongest result among fully open models. That puts it ahead of Moonlight-16B-A3B (76.2), SmolLM3-3B-Base (70.5), OLMo-3-7B (70.1), and OLMoE-1B-7B (61.9), though it still trails Qwen3.5-4B-Base (79.5). It leads on WinoGrande with a score of 86.5 and posts 65.7 on HumanEval+. For long-context tasks, it averages 41.5 on HELMET and 79.4 on RULER.
Post-training sharpens the model further. Average scores climb from 71.58 after supervised fine-tuning to 72.67 after DPO and 73.22 in the "Think" configuration, beating Olmo3-7B-Think (71.97), Gemma-4-E4B Think (70.47), and Qwen3.5-4B (69.73). On instruction-following, IFEval rises from 77.08 to 83.70.
The post-training recipe is itself notable. After SFT on Dolci-Think-SFT-7B and Nemotron mixtures, AMD runs DPO with router bias updates and the auxiliary load-balancing loss disabled to avoid degradation. Reinforcement learning then proceeds inside AMD's Miles framework: 1,400 steps of instruction-following RLVR, followed by Multi-Teacher On-Policy Distillation that folds the gain back in without sacrificing math or code performance.
The Licensing Catch
There is a caveat that matters for commercial users. The model weights ship under a ResearchRAIL license, which restricts use to academic and research purposes only, so Instella-MoE is not a drop-in model for production deployments. The training codebase, however, is MIT licensed, and that is arguably the more reusable asset — it gives other labs a concrete blueprint for training on AMD silicon.
AMD has published the full weights from every training stage, the data mixtures, the training configurations, and the inference code across a Hugging Face collection, a GitHub repository, and its ROCm blog. That level of openness — training-stage-by-training-stage, not just a final checkpoint — is what distinguishes a "fully open" release from a typical open-weight one.
Why This Matters Beyond the Benchmarks
The subtext of the release is hardware competition. Nvidia's CUDA ecosystem and H100-class GPUs have been the default substrate for frontier model training, and challengers have spent years trying to prove their accelerators can handle the full training stack. Instella-MoE is a credible artifact that AMD's Instinct line — already deployed at scale by hyperscalers and national labs — can do more than inference workloads. If AMD can keep producing competitive open models trained on its own hardware, it strengthens the case for buyers looking to break Nvidia's grip on the AI compute market.
For researchers, the appeal is the transparency. Fully open releases that document the data mix, the mid-training blend, the distillation strategy, and the RL curriculum remain rare, and they let the community audit and reproduce claims rather than take a lab's word for it. Instella-MoE joins a small but growing roster — including AMD's own earlier Instella dense models — pushing that standard forward.
Stay Ahead of AI
Want this kind of deep technical coverage as it happens? Bookmark our hub for the latest AI developments and never miss a release.
Read more AI news →
