DeepSeek's Next Move: What V4 Will Look Like

The Evidence Trail

DeepSeek doesn't announce roadmaps. Instead, the company communicates through a steady cadence of research papers, each one a puzzle piece that, when assembled, reveals the contours of their next model. Between late 2025 and early 2026, DeepSeek published three major papers — on Engram, mHC, and DSA — while simultaneously pushing revealing commits to their open-source repositories.

This is the same pattern that preceded DeepSeek-V3: a series of architectural innovations published months before the model dropped. The difference this time? The ambition is significantly larger. The papers don't just optimize transformers — they propose fundamental changes to how attention, memory, and inference work.

What follows is not official confirmation. It's an analytical reconstruction based on public evidence: peer-reviewed papers, open-source code, and hardware signals. Where the evidence is strong, we say so. Where it's speculative, we flag it clearly.

Native Multimodal Architecture

Current multimodal models — including DeepSeek's own Janus — bolt vision capabilities onto a language model through adapters. It works, but it's architecturally inelegant: the vision and language components don't truly share representations.

V4 is expected to change this. Multiple signals point toward a unified encoder-decoder architecture where text, image, and video tokens flow through the same transformer backbone from the start. This isn't just a quality-of-life improvement — it fundamentally changes what the model can reason about.

A natively multimodal V4 could reason across modalities in a single forward pass: reading a chart, understanding the text label, interpreting the visual trend, and generating a written analysis — all without handing off between separate systems. This is the direction Google's Gemini took, and DeepSeek appears to be following suit with their own architectural twist.

DSA: Dense Sparse Attention

Standard Mixture-of-Experts (MoE) architectures make a binary choice: route each token to a subset of experts. It's efficient but crude. DSA (Dense Sparse Attention) takes a more nuanced approach by operating at the attention level rather than the expert level.

The core idea: for each token, the model dynamically decides whether to apply full dense attention (expensive but thorough) or sparse attention (fast but selective). Complex reasoning tokens — those in the middle of a multi-step mathematical proof, for example — get the full treatment. Routine tokens — articles, prepositions, predictable continuations — get the sparse path.

The result is a model that can be both large and fast. Dense attention handles the hard problems; sparse attention prevents the compute bill from exploding on the easy ones. Early benchmarks in the paper show significant inference speedups with minimal quality loss on reasoning tasks.

mHC: Manifold-Constrained Hyper-Connections

Training instability is one of the dirty secrets of large language models. As models scale, gradients can vanish, explode, or oscillate — causing training runs to crash or produce suboptimal weights. DeepSeek's mHC paper proposes a mathematically elegant solution.

mHC constrains the information flow between transformer layers to lie on a learned manifold. Think of it as giving the gradient signal a highway to travel on: instead of bouncing unpredictably between layers, information follows smooth, geometrically constrained paths. The paper reports 6–7% training efficiency improvements — not just speed, but actual convergence quality.

For V4, mHC likely serves as the training backbone. It won't be visible in the final model's outputs, but it's what makes it possible to train a model of V4's expected scale without burning through billions of dollars in failed runs.

Independent researchers have called this a "striking breakthrough" — not because it changes what the model can do, but because it changes what's economically feasible to train.

Engram: External Memory at O(1)

Here's the fundamental problem with transformers: every piece of knowledge the model "knows" must be re-derived from the weights during each forward pass. There's no distinction between static knowledge (the capital of France) and dynamic reasoning (solving a novel math problem). Both cost the same compute.

Engram fixes this by introducing a conditional external memory module with O(1) lookup. Static knowledge — facts, definitions, established relationships — gets stored in a persistent memory bank. The model learns when to query this memory instead of recalculating.

The implications are significant. A model with Engram can be smaller (fewer parameters dedicated to memorization) while being more knowledgeable (the memory bank can be updated without retraining). It also opens the door to domain-specific memory banks: plug in a legal Engram for contract analysis, a medical Engram for diagnostic support.

The paper, signed by DeepSeek founder Liang Wenfeng himself, positions Engram as solving a "fatal flaw" in transformer architecture. That's not modest language from a company known for understatement.

DualPath Inference Strategy

Modern AI doesn't just answer questions — it runs multi-step tasks. Coding agents, research assistants, and automated analysts all need to maintain context over long, branching conversations. Standard transformer inference handles this poorly: the attention cost grows quadratically with context length.

DualPath splits inference into two parallel tracks: a fast path for immediate, short-context responses and a deep path for long-context agentic reasoning. The model dynamically routes requests based on complexity, context length, and task type.

For a simple Q&A, the fast path delivers sub-second responses. For a 50-step coding task with 100K tokens of context, the deep path kicks in with optimized attention patterns, Engram lookups, and sparse computation. The user doesn't see the routing — they just experience a model that's both fast and capable.

Hardware Optimization: Blackwell & FP8

Code doesn't lie. Recent commits to DeepSeek's open-source inference engine reveal NVIDIA Blackwell-specific optimizations, including FP8 KV cache support and FlashMLA (Multi-Level Attention) hooks. These aren't experimental branches — they're being merged into main codepaths.

FP8 KV cache is particularly telling. By halving the precision of cached key-value pairs, the model can maintain twice as much context in the same GPU memory. Combined with FlashMLA — a custom attention kernel optimized for DeepSeek's multi-level attention patterns — this suggests V4 is being designed from the ground up for Blackwell hardware.

This matters because it signals DeepSeek's access to (or expectation of accessing) next-generation NVIDIA hardware, despite ongoing US export controls. It also suggests that V4's performance won't just come from better algorithms — it'll come from hardware-software co-design.

Confirmed vs. Speculative