DeepInfer for DeepSeek

What is DeepInfer?

DeepInfer is the inference framework behind DeepSeek’s large-scale production services. The public repository exposes an offline engine and examples for DeepSeek-V4.1-Flash. It is the software that runs the model; the model weights and the surrounding service are separate parts of a deployment.

Start with the deepseek-ai repository below when searching for “deepinfer deepseek.” The deployment Q&A explains the architecture, while the examples document the runnable entry points.

DeepInfer deployment requirements

Check the accelerator and software stack before building. The reviewed README lists the following targets. A benchmark’s GPU count describes that experiment; it does not establish a minimum card count for every offline configuration.

Build and run offline inference

Activate a fresh Python 3.12 environment on a supported machine with the matching toolkit already available. This NVIDIA example follows the upstream source-build workflow; it is intended for a prepared inference machine. Review the repository’s build script and record the checked-out revision for reproducibility.

git clone --recurse-submodules https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
git rev-parse HEAD

export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py --help
python examples/offline_inference.py

Choose the model files and parallel configuration

The examples default to deepseek-ai/DeepSeek-V4.1-Flash-1017. On the first run, a model ID is downloaded to the Hugging Face cache. If the files are already present, point DEEPINFER_HF_MODEL_DIR at the local checkpoint directory. An explicitly supplied --model_name takes priority over that environment variable.

For Ascend, use DEEPINFER_ACCELERATOR=ascend and the corresponding toolkit. The examples document --ep, --dp and --cp configurations. Choose them for the devices and topology you actually have; the command above does not promise that one accelerator has enough memory.

export DEEPINFER_HF_MODEL_DIR=/absolute/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py

What is needed for an online service?

The public engine is one part of the production architecture described in DeepSeek’s deployment Q&A. The release identifies several service-level pieces that are still needed around it. Treat the offline run as an engine check before integrating those pieces.

DeepInfer vs vLLM and SGLang

Compare the workflow you need, the exact model revision and the supported hardware. The official documentation provides different entry points: DeepInfer’s released offline examples, vLLM’s HTTP serving guide, and SGLang’s server-and-request tutorial. Those entry points alone do not establish a speed ranking.

Evaluate performance on your workload

DeepSeek publishes separate prefill and decode benchmarks with specific hardware and parallel settings. Keep that context attached to any reported result. The examples also explain how draft-token settings affect decode-throughput accounting; accepted output tokens and speculative work need to be distinguished.

For your own comparison, hold the checkpoint, precision, input/output lengths, concurrency and stopping conditions constant. Repeat runs with both cold and warm caches, then check output quality alongside latency. An engine that looks fast on short prompts may behave differently on your actual request mix.

Count the tokens your deployment actually sees

Use the tokenizer and conversation format for the exact deployed checkpoint. DeepSeek Recipe provides request conversion, prompt rendering and token encoding with an attached tokenizer for DeepSeek V4 and V4.1. The caller still supplies model inference and HTTP transport.

A plain-text count helps inspect the text layer. A complete request can also contain roles, tools, images and generation markers. Record the final serialized input token IDs or request-level usage when checking context limits and throughput. If two frameworks receive different serialized prompts, their performance results are not directly comparable.

Tiktokenizer’s related DeepSeek V4 and R1 pages describe their own tokenizer targets. They are useful references, but their counts are not a verified DeepSeek-V4.1-Flash-1017 request total. This guide does not run a DeepInfer server in your browser.

Tiktokenizer Reference Sources

Tokenizer behavior and model limits change. Verify production decisions with current provider documentation:

• DeepSeek: DeepInfer repository and build requirements
• DeepSeek: Inference deployment Q&A
• DeepSeek: Offline inference and benchmark examples
• DeepSeek Recipe: Request conversion and token encoding
• vLLM: Online serving
• SGLang: Sending a request