DeepSeek DeepInfer: what it is and is it free?

DeepInfer is the software DeepSeek uses to serve its own models through its API, and as of October 20 it is open source. The code is free under the Apache 2.0 license, but running it takes data-center hardware: Nvidia's SM100 GPUs (the B200 and GB200 class) or Huawei's Ascend 950 chips.

DeepSeek announced it in a thread on X. @deepseek_ai called it "our in-house inference framework, now fully open source," and said it powers DeepSeek-V4.1-Flash at scale. The stated goal is to help others "lower compute costs, reduce energy use, and make AI more affordable." The GitHub repo has passed 11,000 stars since then.

What DeepInfer does

A model's weights are a big file of numbers. The inference engine turns that file into a working service: it loads the model across many chips, batches requests from many users, and decides how to generate tokens as fast and cheaply as possible. Most companies use open engines like vLLM or SGLang. The big labs keep their own engines private, since a lot of their cost advantage lives there.

The second post in DeepSeek's thread lists what is inside: optimized kernels, distributed inference, speculative decoding, a feature called SWA Bounded Replay, and structured outputs. Translated for people who don't write CUDA:

• Speculative decoding drafts several tokens ahead and checks them in one pass, so answers come out faster.
• Structured outputs force a reply into a set format, like a JSON schema, which other software can then read without breaking.
• Deterministic sampling (Nvidia only) returns the same answer to the same prompt every time. That's handy when you're testing.
• The parallelism settings spread one huge mixture-of-experts model across dozens of GPUs.

According to the README, the project grew mainly out of vLLM and builds on DeepSeek's earlier open libraries, FlashMLA, DeepGEMM and DeepEP. Right now it supports exactly one model: DeepSeek-V4.1-Flash, defaulting to the 1017 update that shipped last week. DeepSeek says code for future models will land in the same repo.

Is DeepInfer free?

The software is. Apache 2.0 lets you use it commercially, change it and ship it. The hardware is where it stops being free. The README asks for Linux, Python 3.12, and CUDA 13 on Nvidia or CANN 9.2 on Ascend. DeepSeek's own benchmark charts were measured on 4 SM100 GPUs for prompt processing and 64 for generating answers. You won't be running that on a Mac Studio.

The release is also not the full production stack. The README says it leaves out a KV cache system, the split between prompt processing and answer generation onto separate machines, and load balancing across replicas. Those pieces are a big part of what keeps a service cheap under real traffic, so a hosting company picking this up still has work to do. The speed claims are DeepSeek's own, and Bloomberg noted no outside testing has been published yet.

What people on X are saying

Most of the reaction was about cost. @kimmonismus called it a big deal because "model weights alone don't make AI cheap to run," and pointed out that it runs on Huawei chips as well as Nvidia. I think the Huawei part is the bigger story. DeepSeek put Ascend on equal footing with Nvidia's data-center GPUs in the README, not as an afterthought.

@krishnanrohit read it as strategy, arguing DeepSeek's plan is simpler than any American lab's: "build models, then open source models and infrastructure, and provide inference to make money." Put another way: give away the weights and the engine, then sell the hosted version to everyone who doesn't want to run 64 GPUs.

My read

If you run a small business, you won't install DeepInfer. You'd feel it secondhand, if at all. Cloud providers that host DeepSeek models can now borrow the methods DeepSeek uses, which should push the price of hosted open models down and narrow the gap with the API DeepSeek runs itself. If a US provider hosts V4.1-Flash with these tricks, you could get close to DeepSeek's pricing without sending prompts to a company in China.

The data question is real. As TNW reported, the US ordered civilian federal agencies to remove DeepSeek products a day before this release. Open code you can inspect and run with your own provider is not the same as DeepSeek's app, but if you work in a regulated industry, you should know which one you're using.

What it means for a business

Cheaper inference makes it easier to put an AI model behind a routine job, like reading inbound emails, pulling fields from invoices, or drafting replies. What's still hard is deciding which of your jobs are steady enough to hand off and which data has to stay put. New Face Design's free process audit starts there: we map where your team's hours go and show which tasks are worth automating first.