DeepSeek Open Sources DeepInfer Inference Framework

Oct 20, 2026
DeepSeek released its production inference framework DeepInfer under the Apache License 2.0, with support for Nvidia and Huawei hardware.

DeepSeek open sources DeepInfer, the inference framework used for its large scale production services, as detailed in the project's GitHub repository. The initial release supports the DeepSeek-V4.1-Flash model on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs.

DeepInfer includes optimized kernels, parallel processing for attention and mixture of experts layers, asynchronous scheduling, structured outputs and a decoding method that uses draft tokens and confidence estimates. Batch invariant inference and deterministic sampling are available only on Nvidia hardware.

The release provides offline inference but does not include a KV cache system, separate prefill and decode services or load balancing across data parallel replicas. DeepSeek released the framework under the Apache License 2.0, with Linux, GCC 13.0 or later and Python 3.12 among its requirements.

The framework was originally built largely on vLLM, with ideas from SGLang and TensorRT-LLM. No outside party has yet benchmarked its efficiency gains, Bloomberg reported, according to Quartz.

We hope you enjoyed this article

Consider subscribing to one of our newsletters like Silicon Brief or Daily AI Brief.

Also, consider following us on social media:

AI Chips & Datacenters AI Brief AI Brief (X)
Free newsletter

Silicon Brief

Weekly coverage of AI hardware developments including chips, GPUs, cloud platforms, and data center technology.

AI Token Webinar

Whitepaper

Tensordyne Napier: What If One Rack Could Do the Work of Nine?

Tensordyne

This Tensordyne whitepaper presents Napier, an inference-focused AI processor and rack-scale system based on the company’s TDN Math logarithmic number system. It examines infrastructure requirements for large mixture-of-experts and agentic models, compares major inference architecture approaches, and details the TDN AIP processor, TDN72 pod, TDN Link fabric, and Napier Ultra configuration. The paper reports simulation-based performance, cost, and accuracy-validation results, including Tensordyne’s projected comparison of one Napier rack with a nine-rack Nvidia Rubin plus Groq deployment; the chip is reported as taped out and in fabrication.

Read more

You may also like

Free, six days a week

Daily AI Brief: the AI news that matters, in your inbox.