DeepSeek Engram

Context: N-Gram Models

"engram"? → right arrow Embedding vector itself is the "n-gram"

N-gram Models illustrate higher-order correlations between N words (unigram, etc.)

We can scale Embedding tables >10billionparameters is greater than 10 billion parameters. This is used in Ads and Pinterest. State of the art for modeling clicks, conversions, etc. Evolved to perform regression, memorize patterns, and train on trillions of examples and huge QPS inference.

We chunk the query ("reebok new running shoes") into 3 3-grams. The query is Lemmatized, which reduces words to their base or dictionary form, called a "lemma" (i.e. "running" turns into "run" and "shoes" into "shoe" to reduce cardinality). We then hash the string and look into the sparse embedding table to fetch and sum embeddings. The embeddings are then fed into the neural networks.

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

It goes back to the original string and actually tokenizes is differently. The Engram block grabs level two grams and three grams and uses that to produce the lookups for the embeddings. Then they also use the hidden like the residual backbone. And they use that to perform like a gating also. The lookups happen directly from the input string to the transformer. and then a gating function is the only thing that looks at the hidden state.

So when it's training, they are taking the input IDs like the input sequence. They can in parallel, start computing the transformer from the unigrams and then they're fetching that and the embeddings from like the lookup and then they're injecting them into the later parts of the architecture. So you can like overlap this computation so you can be sending parsing and sending the message.

They do the same canonicalization with stripping space and lowercasing to get a 23% 23 percent reduction in the vocabulary projection layer. Vocabularies in transformers go up to 50thousandwords 50 thousand words or 200thousandwords 200 thousand words To be more precise: if you process a 128thousandtoken 128 thousand token tokenizer with a fixed budget, the number of symbols decreases by 23% 23 percent and for 2 2-grams, this would be a 0.77×0.77=59.29% 0.77 times 0.77 equals 59.29 percent or a 40.71% 40.71 percent reduction due to quadratic nature.

Mathematics: Context-aware Gating (From Paper)

The retrieved embeddings et e sub t serve as context-independent priors. Being static, however, they inherently lack contextual adaptability and may suffer from noise due to hash collisions or polysemy. They employ a context-aware gating mechanism via a hidden state ht h sub t— which has aggregated global context via preceding attention layers—as a dynamic Query, while the retrieved memory

et e sub t serves as the source for both Key and Value projections:

kt=WK⁢et k sub t equals W sub K e sub t, vt=WV⁢et v sub t equals W sub V e sub t

where WK,WV W sub K comma W sub V are learnable projection matrices. To ensure gradient stability, we apply RMSNorm RMSNorm to the Query and Key before computing the scalar gate αt∈(0,1) alpha sub t is a member of open paren 0 comma 1 close paren:

αt=σ⁢(RMSNorm⁢(ht)⊤⁢RMSNorm⁢(kt)d) alpha sub t equals sigma times open paren the fraction with numerator RMSNorm times open paren h sub t close paren raised to the down tack power times RMSNorm times open paren k sub t close paren and denominator the square root of d close paren

The gated output is defined as v~t=αt⋅vt v tilde sub t equals alpha sub t times v sub t. This design enforces semantic alignment: if the retrieved memory et e sub t contradicts the current context ht h sub t, the gate αt alpha sub t tends toward zero, effectively suppressing the noise.

Finally, to expand the receptive field and enhance the model’s non-linearity, we introduce a short, depthwise causal convolution. Let V~∈ℝT×d V tilde is a member of r-T times d denote the sequence of gated values. Using a kernel size w w (set to 4), dilation δ delta (set to the max N N-gram order) and SiLU SiLU activation, the final output Y Y is computed as:

Y=SiLU⁢(Conv⁢1⁢D⁢(RMSNorm⁢(V~)))+V~, Y equals SiLU times open paren Conv 1 D times open paren RMSNorm times open paren V tilde close paren close paren close paren plus V tilde comma

The Engram module is integrated into the backbone via a residual connection: H(ℓ)←H(ℓ)+Y the script l-th power of H left arrow the script l-th power of H plus Y, followed by the standard Attention and MoE.

Impact: Scaling and Performance

For a fixed parameter count, the sweet spot is around 20% 20 percent of the parameters being used for the embeddings. The second graph shows that with the non-scaling O⁡(1) O of 1 overhead of Engram, the curve follows a strict power law (linear in log-space), indicating that Engram provides a predictable scaling knob. Regarding scaling efficiency: while the direct averaging approach of OverEncoding benefits from larger memory tables, Engram unlocks much larger scaling potential from the same memory budget.

Moreover, the throughput hit for adding a 100billionparameter 100 billion parameter Engram on the CPU is small even when adding to a 4billionparameter 4 billion parameter or 8billionparameter 8 billion parameter models.

This Engram blocks gives it a shortcut to avoid complicated computations from the Transformer blocks to get the multi-token lookup. The underlying argument: we are separating out the components of the fact-storing versus the overall semantic meaning of a sentence and we can take advantage of cheap RAM + CPU cycles VS expensive VRAM + GPU cycles.

Questions

  • "Gated residual connections, make sense. So now do the authors also compare gated residual connections within the network between the layers to just these gated connections to the embedding table and do they justify that specifically?"