Last week, NVIDIA published a technical post about running mixture-of-experts models faster on DGX Spark. The engineers chose Ant’s Ling-3.0-flash for the test and worked with the Ant Ling team on the deployment recipe.
The headline number was a 42.5% increase in decode throughput.
Ling went from 37.81 to 53.89 tokens per second on the same DGX Spark. Mean response latency also dropped from 15.21 to 11.49 seconds. Accuracy on the GSM8K math benchmark stayed almost identical, moving from 96.89% to 96.82%.
You can read the comprehensive research and test results here:
https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark
NVIDIA changed three parts of the inference setup:
The kernel that runs Ling’s experts
The precision of its output layer
The way it predicts upcoming tokens
The result gives local AI developers a practical example of how much speed can still be found after a model already fits on the hardware.
Ling has enough capacity for coding, research, and agent workflows, yet only a small part of the model runs for each token. Ant Ling describes the idea as getting “more useful work per token.” NVIDIA’s experiment shows what that looks like once the model meets a serving stack tuned for its architecture.
This is an incredibly important piece of information because it shows how small these models can get while still maintaining strong reasoning and coding capabilities. I’ve kept an eye on nearly every coding model that can run locally, including Qwen3.8–27B, DeepSeek-V4-Flash, GLM-5.3-Flash, and many other open-weight models.
So, it made me wonder why NVIDIA’s researchers chose Ling-3.0-flash for this experiment.
Why NVIDIA Chose Ling-3.0-Flash
NVIDIA’s 42.5% result came from a close match between Ling-3.0-flash, the GB10 chip inside DGX Spark, and several changes in SGLang. The model choice explains a lot of what happened next.
Take note that even though Ling-3.0-flash behaves like a much smaller model during inference, its 124 billion parameters are still crazy big on disk and in memory.
I checked the model’s download files on Hugging Face, and the total size is around 255 GB. Ant’s design gets around the compute problem through a mixture-of-experts, or MoE, architecture.
The Ling model card lists 512 routed experts. Every token passes through eight of those experts and one shared expert. That leaves only 5.1 billion active parameters per token, a small fraction of the full model.
This setup gives Ling access to a large pool of learned weights while keeping each generation step relatively light.
The entire expert pool still has to sit somewhere in memory, though. Sparse activation reduces the amount of computation, while quantization and DGX Spark’s unified memory handle the physical size of the checkpoint.
Ling also comes with features that suit long agent runs. It has a native 256K context window, reasoning that can be turned on or off, and training aimed at coding, tool calls, deep research, and multi-step tasks.
Its built-in NEXTN layer can predict upcoming tokens, which later became the largest source of acceleration in NVIDIA’s test.
NVIDIA did not say it compared Ling with other models, but my guess is that it offered a better test for DGX Spark.
Ling gave the engineers three clear targets in one model:
Narrow MoE layers
A huge vocabulary layer
A built-in NEXTN speculative decoding
Its FP4 checkpoint also fits within DGX Spark’s unified memory. A smaller dense model would have been easier to run, but it would not have tested as many parts of the inference stack.
DGX Spark Can Hold the Whole Model
NVIDIA DGX Spark is a compact desktop AI system built around the GB10 Grace Blackwell Superchip. The 128 GB version has enough unified memory to hold models that would overwhelm the VRAM on most consumer GPUs.
This is what the actual chip looks like:
The official DGX Spark page lists up to 1 petaFLOP of FP4 AI performance, a 20-core Grace CPU, and 128 GB of unified system memory. NVIDIA rates that configuration for inference with models up to 200 billion parameters.
Ant Ling’s optimized checkpoint is much smaller than the full-precision model. The deployment notebook puts the MXFP4 weights at roughly 60 GB.
Step 3: Download Ling-3.0-flash FP4 (MXFP4) Model Weights
Pre-quantized MXFP4 weights (~60GB) are hosted on:
ModelScope Ling-3.0-flash-fp4
Hugging Face Ling-3.0-flash-fp4
Download model weights to ~/models/Ling-3.0-flash-fp4 using ModelScope:
!source .venv/bin/activate && uv pip install -U modelscope
!source .venv/bin/activate && uv run modelscope download --model inclusionAI/Ling-3.0-flash-fp4 --local-dir ~/models/Ling-3.0-flash-fp4That leaves room for the server, caches, compiled kernels, and other memory needed during generation.
The shared memory pool is the reason this 124B model can run on a single box. There’s no need to split its experts across several GPUs or send traffic through an interconnect.
For someone building a private coding assistant or local research agent, that makes the whole setup easier to manage.
It also creates a different performance problem. DGX Spark can hold the weights, but it still has to move them quickly. During single-request decoding, NVIDIA found that Ling spent much of its time reading weights from memory. Extra compute power won’t help much while the processor is waiting for those reads to finish.
Where Ling Was Losing Time and What NVIDIA Changed
NVIDIA tested Ling-3.0-flash at batch size one, similar to how someone might run a local assistant. The profiling showed that decoding was limited mainly by memory bandwidth. Most of the delay came from repeatedly reading model weights.
Two parts were especially expensive. The first was Ling’s MoE layer, which routes every token through eight experts and one shared expert. The second was its LM head, which scores outputs across a vocabulary of 157,184 tokens.
Ling’s built-in NEXTN layer offered another opportunity. It can draft several upcoming tokens and let the main model verify them together, reducing the number of decoding rounds.
NVIDIA turned these findings into three optimizations:
Humming kernels tuned for Ling’s expert shapes
FP8 quantization for the large LM head
NEXTN speculative decoding to generate several tokens per round
The first two made each decoding step cheaper. The third reduced the number of steps needed to generate an answer. Together, they produced the 42.5% throughput improvement.
Let’s discuss them one by one.
1. A Better Kernel for Ling’s Experts
The baseline ran Ling’s experts through the FlashInfer MXFP4 backend. NVIDIA replaced it with Humming, a kernel library that compiles GPU code around the model’s exact dimensions, data types, and hardware.
This reduced the time per output token from 27.71 milliseconds to 25.35 milliseconds, raising throughput by 9.3%.
This optimization fits local inference especially well. A personal workstation often runs the same checkpoint repeatedly, which makes a one-time compilation step easy to justify.
The model does not become smaller or smarter. The software simply becomes better at running its unusual expert shapes.
It is also a reminder that model format is only part of local performance. Two machines can load the same quantized weights and still produce different speeds because their inference kernels are different.
2. Compressing the Output Head
Ling’s output layer, or LM head, maps 2,560 hidden values across a vocabulary of 157,184 tokens. Its weights were stored in BF16 and had to be read for every generated token.
NVIDIA converted this layer to FP8 during loading, cutting its weight traffic roughly in half. Together with Humming, throughput reached 43.94 tokens per second, a 21.7% improvement over the original setup.
Here’s an illustration to make things easier to understand.
One thing developers can easily miss is that quantization is not always applied evenly across a model. Ling’s main weights were already compressed, while the LM head remained at higher precision. Targeting that one layer produced a noticeable gain without rebuilding the entire checkpoint.
This also shows how vocabulary size affects inference. A larger vocabulary may help a model cover more languages, code syntax, and special tokens, but the final projection becomes heavier.
3. Predicting Several Tokens at Once
Ling already contains this prediction layer, so the setup doesn’t require a second draft model or another large checkpoint. That keeps the memory footprint reasonable on a single DGX Spark.
Remove upgrade prompts from your unlocked stories in your settings.
Checking those predictions still costs compute because one normal decode step activates eight routed experts. Verifying four positions can activate up to 32 experts, and NVIDIA measured each verification round at about 2.26 times the cost of standard decoding. The draft has to predict enough correct tokens to recover that extra work.
On a synthetic 32K-input and 2K-output test, NEXTN accepted an average of 3.90 tokens per round. The complete stack reached 102.88 tokens per second, 185.1% above the baseline.
That huge number didn’t carry over to normal prompts. Synthetic output tends to repeat itself, which makes the next few tokens easier to guess. The full GSM8K run gave NVIDIA a more sensible estimate.
What the 42.5% Result Covers
NVIDIA ran the optimized setup across all 1,319 questions in GSM8K. Throughput increased from 37.81 to 53.89 tokens per second, while mean latency dropped from 15.21 to 11.49 seconds. Accuracy stayed almost identical at 96.82%.
The published results were strong without looking suspiciously perfect:
Decode throughput: 37.81 tokens per second increased to 53.89, up 42.5%.
Time per output token: 26.45 milliseconds dropped to 18.56, a 29.8% reduction.
Mean latency: 15.21 seconds dropped to 11.49, a 24.5% reduction.
P95 latency: 50.20 seconds dropped to 38.60.
GSM8K accuracy: 96.89% changed to 96.82%. NVIDIA found no statistically detectable accuracy difference.
Time to first token: 0.207 seconds changed to 0.214 seconds.
The time to first token barely changed, moving from 0.207 to 0.214 seconds. NVIDIA made generation faster, not prompt processing.
A long coding response should benefit more than a short chat reply, and retrieval or tool calls will still take the same amount of time. A 42.5% increase in decoding speed does not mean an agent will finish its entire task 42.5% sooner.
GSM8K contains grade-school math word problems. It provides a consistent test of multi-step reasoning, but it says little about editing a repository, calling tools, or searching through documents. The stable accuracy shows that NVIDIA preserved Ling’s math performance during this test.
Run Ling-3.0-Flash Locally
Ant Ling’s official cookbook includes recipes for SGLang, vLLM, and llama.cpp on one DGX Spark. The SGLang MXFP4 Humming notebook is the closest match to NVIDIA’s optimized setup.
The local setup can stay fairly direct:
Start with the 128 GB DGX Spark. The optimized recipe targets the model’s roughly 60 GB FP4 checkpoint and about 121 GB of available unified memory.
Follow the tuned SGLang branch. The cookbook currently points to InclusionAI’s Ling-specific branch, which contains the Humming adapter, FP8 output-head conversion, and module-sharing fix for NEXTN.
Precompile the kernels. Compiling FlashInfer kernels while the model is already loaded can exhaust unified memory. The notebook frees the model first and runs the compiler with the machine’s memory available.
Enable the changes one at a time. Record a plain baseline, switch to Humming, turn on the FP8 head, and add NEXTN last. Check the startup log after every change.
Benchmark your own prompts. Track time to first token, decode speed, total latency, accept length, memory pressure, and task success. Include short replies, long answers, and tool-calling runs.
The notebook recommends setting SGLang’s static memory fraction to 0.68. A value of 0.75 can cause out-of-memory errors or driver lockups under heavier unified-memory loads.
SGLang serves the model through an OpenAI-compatible API, so many existing tools only need a different base URL. Replace the sample API key, restrict network access, and pin the model and server revisions before treating the endpoint as a shared internal service.
How Developers Take Advantage of Small Models?
As I said, the discussion is huge around small models running locally and used for agentic tasks and coding. Imagine your agents running 24/7 on your system without any interruptions. No need for subscriptions or buying credits/tokens.
Talking about the current state of small models running locally, Ling makes a broad point. A lower active parameter count can make an AI system easier to keep running throughout the day, especially when one workflow sends dozens of model calls before it finishes.
The word “small” is relative here.
Ling still needs around 60 GB for the quantized checkpoint. Its 5.1B active parameters are the part that gives it small-model behavior during generation.
Tiny models can run on laptops and phones, while sparse models such as Ling bring more capacity to a workstation without activating the whole network on every token.
Developers can bring that idea into several everyday workflows:
Private coding assistants: A local model can read internal repositories, search documentation, suggest patches, and explain test failures without sending source code to a hosted API.
Document and research agents: Teams can place reports, contracts, spreadsheets, or customer records beside the model and keep processing on their own hardware.
Routine steps inside larger agents: Classification, extraction, tool selection, query rewriting, and short summaries rarely need the largest available model. A smaller local model can handle those calls, while difficult reasoning tasks go to a stronger remote model.
Background automation: Overnight code reviews, document tagging, local search indexing, and report generation become easier to budget on hardware you already own.
Faster iteration on the serving stack: Local deployment gives you control over quantization, context length, caching, sampling, and speculative decoding. NVIDIA’s Ling test shows how much those choices can change the experience before the model itself changes.
Smaller models also make agent loops feel less sluggish.
One reply that arrives three seconds sooner is pleasant. Ten serial calls with the same saving can remove half a minute from a task. Coding and research agents often make more calls than that, which is why per-token efficiency becomes noticeable at the workflow level.
A model earns its place in a developer stack through sufficient quality, predictable latency, and low enough cost for repeated calls. Local models are especially attractive for high-volume routine work and sensitive data.
Larger cloud models can remain available for prompts that require broader knowledge or harder reasoning.
Ling’s reasoning mode can also be disabled per request. Code repair and research synthesis may deserve the extra thought. Routing, extraction, and short tool calls can run through a faster profile. A local API lets one checkpoint support both styles without downloading separate models.
Where Ling-3.0-Flash Fits Best
Ling is strongest in workflows that generate many tokens across several connected steps.
Coding agents, research systems, office automation, and private knowledge assistants all fit that description. They benefit from long context, tool calling, and the lower active compute of the MoE design.
Ant Ling is clear about the model’s limits.
Its release post mentions weaker broad knowledge coverage and occasional Chinese-English mixing during long or stressful conversations. Teams still need their own evaluations for code quality, tool accuracy, hallucinations, and language consistency.
A 256K context window also carries a cost.
Large prompts take longer to process and consume more cache memory. Retrieval can often give the model a smaller and cleaner set of documents, which is usually easier to serve on a workstation.
The NVIDIA result only covers batch size one. A shared endpoint with many simultaneous requests may behave differently once continuous batching enters the picture. Local teams planning multi-user service should repeat the test at their expected concurrency.
DGX Spark also moves operational work onto your desk. Model updates, authentication, monitoring, backups, and uptime become your responsibility.
That trade can be worthwhile for private or high-volume workloads, but a hosted API remains easier for occasional requests and quick access to many models.
Final Thoughts
NVIDIA’s Ling-3.0-flash experiment is one of the more convincing local inference studies I have seen because the engineers worked with the same model and hardware from start to finish.
They found speed in three ordinary places: the expert kernel, the output layer, and the decoding strategy.
Ling brought the right ingredients to the test.
Its 124B parameter count provides a large expert pool, while 5.1B active parameters keep each token manageable. The built-in NEXTN layer gave NVIDIA another way to reduce generation time without training a separate draft model.
The 42.5% improvement belongs to NVIDIA’s GSM8K setup, so I would take that figure with a grain of salt. The better takeaway is the process of profiling the actual model, fixing the expensive operations, and measuring the full workload again.
For developers, the goal is a model that can stay close to the work at a cost that allows repeated calls. Ling-3.0-flash shows how a sparse model, enough local memory, and careful serving code can turn one compact desktop system into a capable agent machine.
Let me know what you think in the comments!
Hi there! Thanks for making it to the end of this post! If you enjoyed this content and would like to support my work, consider becoming a paid subscriber. Your support means a lot!