An independent developer has demonstrated that DeepSeek's V4 Flash model, a 304-billion-parameter mixture-of-experts system, can run in production on a single AMD MI300X GPU, achieving 168.6 tokens per second in single-stream decoding without any weight quantization or offloading. The work, published on GitHub and surfacing to the top of Hacker News on August 4, 2026, offers the clearest evidence yet that AMD's hardware can serve a frontier-scale open model without relying on Nvidia.

The repository, maintained by developer Ryan Zhou, includes a complete Docker Compose stack, pinned file overlays, and tuning tables for running the DeepSeek-V4-Flash-0731 checkpoint on one MI300X. For readers tracking the breaking AI news on the chip war between AMD and Nvidia, the result is significant because it challenges the assumption that frontier inference requires Nvidia's latest and most expensive hardware.

The Numbers

The benchmarked performance, measured on a pinned software stack using the ROCm nightly build of vLLM, tells a compelling story about what a single AMD accelerator can do:

  • Single-stream decode: 168.6 tokens per second, fast enough for real-time chat and agentic workloads.
  • Prefill throughput: approximately 7,900 to 8,500 tokens per second with tuned kernels, meaning long prompts are processed in well under a second.
  • Eight concurrent streams: 542 tokens per second aggregate, with 90.3 tokens per second per stream.
  • 64-stream burst: 830 tokens per second aggregate, with no out-of-memory errors and no engine failures.
  • Context length: 256,000 tokens validated in testing, with the architecture supporting up to one million.

The entire model occupies 156.67 gigabytes of high-bandwidth memory, fitting entirely within the MI300X's 192-gigabyte HBM3 pool. No additional quantization or weight offloading was needed, which preserves the model's full accuracy.

Why the MI300X Works

The AMD MI300X possesses two attributes that make single-GPU deployment of a model this large possible. It has 192 gigabytes of HBM3 memory, 2.4 times the capacity of an Nvidia H100 SXM5, and 5.3 terabytes per second of memory bandwidth. A separate write-up by a developer identified as Fergus Finn, which laid the groundwork for this deployment, estimated that the MI300X costs roughly half as much as comparable Nvidia hardware at list price.

For this particular 304-billion-parameter checkpoint, the memory capacity is the decisive factor. The model fits entirely in on-chip memory, leaving room for a 20-gigabyte GPU key-value cache pool and a 96-gigabyte CPU tier for evicted prefix-cache entries. One card can handle two to eight typical concurrent streams and bursts of up to 64 streams, which is enough for many production inference workloads.

The Engineering Hurdles

Running DeepSeek V4 Flash on MI300X was not straightforward. The official vLLM recipe covers Nvidia hardware and newer AMD GPUs such as the MI325X and MI355X, but not a single-MI300X production configuration for the July 31 checkpoint.

The repository documents several engineering problems that had to be solved. The MI300X uses a variant of the FP8 number format that differs from the standard used by newer AMD chips, and a kernel that assumes the wrong semantics can produce results off by a factor of two. The developer also had to fix mixture-of-experts routing at high concurrency, causal speculative verification, and CPU key-value synchronization, several of which remain unfixed in upstream vLLM.

The repository adds correctness overlays, a validated serving configuration with speculative drafting, AITER GEMM tuning tables for chip shapes that the packaged tables were missing, and a hybrid key-value strategy that combines a GPU cache with CPU offload.

What It Means for the Chip War

The demonstration arrives at a pivotal moment for AMD's ambitions in AI. Nvidia controls the overwhelming majority of the AI accelerator market, buttressed by its CUDA software ecosystem, which many developers see as a moat that AMD has struggled to cross. SemiAnalysis examined that question directly in a July 2026 analysis titled "Can AMD break the CUDA moat?" and the answer remains contested.

But open-source projects like this one chip away at the perception gap. If a single MI300X can serve a frontier-scale model at competitive speeds, the cost argument for AMD becomes harder to dismiss. AMD has also been building out its own open model portfolio, releasing Instella-MoE-16B, a fully open model trained from scratch on its own Instinct GPUs, and partnering with Zyphra on a 15-megawatt MI355X platform.

Meanwhile, DeepSeek itself has been driving down the cost of frontier AI. The V4 Flash model was already the cheapest well-known model to run according to research firm analysis, matching GPT-5.6 Luna at 60 percent lower cost. Combined with cheaper AMD hardware, the economics of serving large models outside the Nvidia ecosystem are improving rapidly.

The Open Source Advantage

The repository underscores a broader theme in 2026 AI development: open-weight models and open-source inference tooling are making it possible for independent engineers to replicate and optimize deployments that were once the exclusive domain of well-funded labs. By publishing the full configuration, tuning tables, and the specific bugs fixed along the way, the work lowers the barrier for anyone considering AMD hardware for serious AI workloads.

Stay Ahead of AI

The battle between AMD and Nvidia for AI inference is intensifying, and open-source contributions like this deployment are quietly shifting the competitive landscape. For the latest AI developments in hardware, open models, and infrastructure, stay with our coverage.

Read more AI news →