DeepSeek Engram — V4 Memory Architecture Explained
A groundbreaking paper titled "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models," signed by Liang Wenfeng, proposes the Engram module — a new conditional memory architecture that provides O(1) knowledge lookup, potentially fixing Transformer's fundamental memory limitations and hinting at DeepSeek V4's architecture.
📊 Key Metrics at a Glance
Engram Module
New conditional memory layer for O(1) knowledge lookup
U-Shaped Scaling Law
20-25% memory allocation optimal for performance
27B Parameters
Outperforms MoE baseline under equal conditions
<3% Overhead
100B parameters offloadable to CPU/SSD
Reasoning Boost
Frees early layers for deeper reasoning
2.9k+ GitHub Stars
Strong community reception and interest
Table of Contents
📌 Quick Summary
DeepSeek, in collaboration with Peking University, has released a 33-page paper introducing Engram — a new sparse memory module that complements Mixture of Experts (MoE). While MoE solves "how to calculate less," Engram solves "don't calculate blindly" by providing deterministic O(1) knowledge lookup for static patterns like entity names and fixed phrases. The paper discovers a Sparsity Allocation Law: 20-25% of sparse parameters should go to memory, with the rest to computation.
The Problem: Transformer's Memory Limitation
Currently, Mixture of Experts (MoE) has become the mainstream architecture for large language models, powering systems like DeepSeek V3, Mixtral, and others. However, it's fundamentally still based on the Transformer architecture, which lacks a native "knowledge lookup" mechanism.
This means that many tasks that should be solved in O(1) time — like retrieving factual information — have to be "simulated" through extensive computations. The model essentially has to compute its way to remember things that could simply be looked up.
💡 Example: To identify the entity "Diana, Princess of Wales," an LLM has to consume multi-layer attention and FFN to gradually combine features. In theory, this process could be completed through a single knowledge lookup operation — like looking up a word in a dictionary.
The paper argues that this is fundamentally wasteful: using expensive compute cycles to repeatedly "rediscover" static knowledge that could be retrieved in constant time.
Enter Engram: The "Electronic Brain" Module
The term Engram originates from neurology, meaning "memory trace" — an extensible and retrievable memory unit. DeepSeek's implementation modernizes the classic "hashed N-gram embedding" technique and transforms it into an extensible lookup table module inserted into Transformer's middle layers.
Two Types of Tasks in Language Modeling
The paper clearly divides language modeling into two types of subtasks:
🧠 Combination & Reasoning
Requires dynamic computation and attention:
- • Context relationships
- • Long-range dependencies
- • Logical reasoning
- • Chained reasoning
- • Novel combinations
📚 Pattern Retrieval
Can be solved with O(1) lookup:
- • Entity names
- • Fixed collocations
- • Common phrases
- • Grammar fragments
- • Idiomatic expressions
Engram transfers these "local static patterns" to a cheap knowledge lookup primitive, quickly providing candidate information through deterministic lookup tables. The context then decides whether to adopt it through a gating mechanism.
Core Architecture Deep Dive
The Engram module processes each position in two functional stages: Retrieval and Fusion.
1. Sparse Retrieval via Hashed N-grams
2. Context-Aware Gating
Retrieved embeddings serve as context-independent prior information but are susceptible to noise from hash collisions or polysemy. The team adopted a context-aware gating mechanism inspired by attention:
- Current hidden state acts as dynamic Query
- Retrieved memory provides Key and Value projections
- RMSNorm ensures gradient stability
- Short depth-causal convolution expands receptive field
| Aspect | MoE (Mixture of Experts) | Engram (Conditional Memory) |
|---|---|---|
| Core Problem Solved | "How to calculate less" | "Don't calculate blindly" |
| Mechanism | Conditional computation via routing | Conditional memory via O(1) lookup |
| Complexity | O(experts activated) | O(1) deterministic |
| Best For | Dynamic reasoning, novel combinations | Static pattern recall, entity recognition |
| Offloading Potential | Limited (compute-bound) | Excellent (memory-bound) |
The Optimal Memory Allocation (20-25%)
One of the paper's most significant discoveries is the Sparsity Allocation Law: under a fixed sparse parameter budget, the optimal split is approximately 20-25% memory (Engram) and 75-80% computation (MoE).
🎯 The 20-25% Rule
When allocating sparse parameters between memory and computation:
- ✓ Too little memory (<20%): Model wastes compute rediscovering patterns
- ✓ Optimal (20-25%): Best balance of retrieval and reasoning
- ✓ Too much memory (>25%): Diminishing returns, starves reasoning capacity
This finding is critical for future model design: it suggests that simply adding more compute isn't always the answer — sometimes you need dedicated memory capacity for pattern storage.
The U-Shaped Scaling Law Discovery
Through "Sparsity Allocation" modeling, the team discovered an unexpected U-shaped scaling law between MoE and Engram. This means the resource ratio between the two needs careful adjustment to find an optimal trade-off between computation and static memory.
🎯 Key Result
Following this law, after expanding Engram to 27B parameters, it outperforms the MoE baseline under strict equal-parameter and equal-FLOPs conditions. The gains appear across multiple benchmarks including reasoning, coding, and math tasks.
"MoE only solves the problem of 'how to calculate less', while Engram directly solves the problem of 'don't calculate blindly'."
Why Reasoning and Math Improve (The Surprising Finding)
Perhaps the most counterintuitive finding: Engram doesn't just help with pattern recognition — it significantly improves reasoning, math, and coding abilities. Why?
💡 The Key Insight
By offloading local pattern work to memory lookup, early Transformer layers finish their "pattern work" faster. This means early layers in Engram models behave like much deeper layers in MoE-only models.
"Engram doesn't make models smarter by adding facts — it makes them smarter by freeing compute."
The paper provides evidence that reasoning improvements come from this "compute liberation" effect: when the model doesn't have to spend cycles on pattern matching, those cycles become available for deeper reasoning chains.
Long-Context Benefits
Engram provides substantial benefits for long-context processing:
Hardware Cost Implications
Scaling memory-enhanced models is often limited by GPU high-bandwidth memory (HBM) capacity. However, Engram's deterministic retrieval mechanism naturally supports decoupling parameter storage from computing resources.
System Efficiency Metrics
- 100B parameter memory tables can be offloaded with less than 3% inference overhead
- Deterministic indices: Memory indices depend only on input tokens, not activations — enables prefetching
- Asynchronous prefetching: Memory retrieval can overlap with computation
- CPU/SSD offloading: Parameters can live in system RAM or even SSD, not expensive HBM
This has massive implications for inference cost: if memory tables can be stored in cheap system RAM instead of expensive GPU HBM, the cost per parameter drops dramatically. This could enable much larger effective model sizes without proportional hardware cost increases.
Cross-Language Generalization
Visualization of the gating scalars shows an obvious selective pattern. The gating mechanism consistently activates when dealing with local, static patterns:
🇬🇧 English Examples
- • "Alexander the Great"
- • "the Milky Way"
- • "By the way"
- • "Princess of Wales"
- • "United States of America"
🇨🇳 Chinese Examples
- • "Four Great Inventions" (四大发明)
- • "Zhang Zhongjing" (张仲景)
- • Idiomatic expressions (成语)
- • Historical entities
- • Classical poetry phrases
This demonstrates that Engram's benefits generalize across languages and domains, recognizing that all languages have static patterns worth storing rather than computing.
Could This Be DeepSeek V4?
With Liang Wenfeng — DeepSeek's founder — as one of the paper's authors, and the breakthrough nature of this research, speculation is mounting that Engram could be integrated into DeepSeek's next flagship model.
🔮 What This Means for V4
- ✓ Better knowledge retrieval without massive parameter increases
- ✓ Improved reasoning, coding, and math abilities from freed compute
- ✓ More efficient compute allocation with the 20-25% memory ratio
- ✓ Drastically reduced hardware costs through memory offloading
- ✓ Potential new standard for hybrid sparse LLMs
DeepSeek has a track record of publishing research before integrating it into production models (as seen with MoE and MLA). Engram could follow the same pattern.
GitHub Repository & Resources
The full 33-page paper and demo code are available on GitHub:
deepseek-ai/Engram
The paper title is: "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models"
Frequently Asked Questions
What is DeepSeek Engram?
Engram is a new conditional memory module proposed by DeepSeek and Peking University that provides O(1) knowledge lookup capabilities, solving the memory limitations of traditional Transformer architectures. It's named after the neurological term for "memory trace."
Is Engram the architecture for DeepSeek V4?
While not officially confirmed, the paper's innovations and Liang Wenfeng's involvement suggest Engram could be integrated into DeepSeek's next flagship V4 model. DeepSeek has historically published research before integrating it into production models.
How does Engram differ from MoE?
While MoE solves "how to calculate less" through conditional computation, Engram solves "don't calculate blindly" by providing deterministic knowledge lookup with O(1) complexity for static patterns. They're complementary — the paper recommends using both together.
What is the optimal memory-to-compute ratio?
According to the paper, 20-25% of sparse parameters should go to memory (Engram), with the rest going to computation (MoE). This is called the Sparsity Allocation Law.
Can Engram be offloaded to CPU/SSD?
Yes! Engram's deterministic retrieval mechanism supports offloading up to 100B parameters to CPU or SSD with less than 3% inference overhead. This is possible because memory indices depend only on input tokens (not activations), enabling asynchronous prefetching.
Does Engram help with reasoning tasks?
Yes, surprisingly. By offloading local pattern work to memory lookup, early Transformer layers finish faster and behave like much deeper layers. This frees compute for deeper reasoning chains, improving math, coding, and logical reasoning abilities.
Does Engram replace attention or MoE?
No, Engram augments them. It's inserted into select Transformer layers alongside existing attention and MoE modules. The paper's optimal configuration uses all three together.
What is the Sparsity Allocation Law?
The U-shaped scaling curve discovery showing that model performance varies based on how sparse parameters are allocated between memory (Engram) and computation (MoE). The optimal allocation is approximately 20-25% memory, 75-80% computation.
Related Reading
DeepSeek R1
Learn about DeepSeek's reasoning model
DeepSeek V3.1 Silent Revolution
How V3.1 improved without fanfare
What is DeepSeek AI?
Complete overview of DeepSeek
DeepSeek vs ChatGPT
How DeepSeek compares to OpenAI