DeepSeek-Coder-V2: Open-Source Code Intelligence

Introduction

The field of code intelligence has seen remarkable advancements through the open-source community, with models like StarCoder, CodeLlama, and DeepSeek-Coder making significant strides. However, these models have yet to reach the performance levels of their closed-source counterparts such as GPT4-Turbo and Claude 3 Opus. Enter DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model designed to bridge this gap. Built on the foundation of DeepSeek-V2, DeepSeek-Coder-V2 undergoes further pre-training with an additional 6 trillion tokens, significantly enhancing its coding and mathematical reasoning capabilities while supporting 338 programming languages and extending context length to 128K tokens.

Enhanced Capabilities

DeepSeek-Coder-V2 stands out with its substantial improvements in various code-related tasks, achieving superior performance compared to closed-source models like GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro. This model excels in benchmarks such as HumanEval, MBPP+, MATH, and GSM8K, demonstrating its prowess in both coding and math tasks. The extensive pre-training dataset, comprising 60% source code, 10% math corpus, and 30% natural language corpus, has been meticulously curated and expanded, resulting in significant accuracy improvements in benchmarks.

Training and Alignment

The training process of DeepSeek-Coder-V2 involves a combination of Next-Token-Prediction and Fill-In-Middle (FIM) objectives, particularly for the 16B parameter model. The FIM approach structures content reconstruction in a specific sequence, enhancing training efficacy and model performance. Additionally, the alignment phase incorporates Group Relative Policy Optimization (GRPO) to align the model's behavior with human preferences, using compiler feedback and test cases to optimize the model's responses for correctness and user satisfaction.

Contributions and Evaluations

DeepSeek-Coder-V2's contributions to the field of code intelligence are manifold. It introduces the first open-source hundred-billion-parameter code model, demonstrating significant advancements over state-of-the-art closed-source models. With a permissive license, DeepSeek-Coder-V2 is publicly available for both research and unrestricted commercial use, promoting further innovation and development in the field. Evaluation results highlight its superiority in code generation and mathematical reasoning, rivaling top closed-source models and setting new benchmarks in various evaluations.

Conclusion

The introduction of DeepSeek-Coder-V2 marks a significant milestone in the evolution of open-source code intelligence. With its enhanced capabilities, extensive training, and public availability, DeepSeek-Coder-V2 paves the way for further advancements in the field, providing a powerful tool for developers and researchers alike. As open-source models continue to close the gap with their closed-source counterparts, DeepSeek-Coder-V2 stands as a testament to the potential of collaborative innovation in the realm of code intelligence.