A few weeks ago,
during Open Source Week,
we open-sourced several libraries.
The response from the community has been incredibly positive – sparking inspiring collaborations, productive
discussions, and valuable bug fixes.
Encouraged by this, we’ve decided to take another step forward: contributing our internal inference engine back to the
open-source community.
We are deeply grateful for the open-source ecosystem, without which our progress toward AGI would not be possible.
Our training framework relies on PyTorch, and our inference engine is built
upon vLLM,
both of which have been instrumental in accelerating the training and deployment of DeepSeek models.
Given the growing demand for deploying models like DeepSeek-V3
and DeepSeek-R1, we want to give back to the community as much as we can.
While we initially considered open-sourcing our full internal inference engine, we identified several challenges:
- Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago. Although structurally
similar, we’ve heavily customized it for DeepSeek models, making it difficult to extend for broader use cases. - Infrastructure Dependencies: The engine is tightly coupled with our internal infrastructure, including cluster
management tools, making it impractical for public deployment without significant modifications. - Limited Maintenance Bandwidth: As a small research team focused on developing better models, we lack bandwidth to
maintain a large open-source project.
Considering these challenges, we’ve decided to collaborate with existing open-source projects as more sustainable alternatives.
Moving forward, we will work closely with existing open-source projects to:
- Extract Standalone Features: Modularize and contribute reusable components as independent libraries.
- Share Optimizations: Contribute design improvements and implementation details directly.
We are profoundly grateful for the open-source movement – from operating systems and programming languages to machine
learning frameworks and inference engines. It’s an honor to contribute to this thriving ecosystem and to see our models
and code embraced by the community. Together, let’s push the boundaries of AGI and ensure its benefits serve all of
humanity.
Note
To clarify, this article outlines our approach to open-sourcing of our DeepSeek-Inference-Engine codebase only.
Regarding future model releases, we maintain an open and collaborative stance towards both the open-source community
and hardware partners.
We commit to proactively synchronizing inference-related engineering efforts prior to new model launches, with the
goal of enabling the community to achieve state-of-the-art (SOTA) support from Day-0. Our ultimate aim is to foster a
synchronized ecosystem where cutting-edge AI capabilities can be seamlessly implemented across diverse hardware
platforms upon official model releases.
