<aside> 💡

If you find any issues or logical errors in this article, feel free to leave a comment or contact me at [email protected]. Published: November 12, 2025 Last modified: November 12, 2025

</aside>

1. Background

As large language models (LLMs) increasingly need to process long contexts, traditional softmax attention faces major challenges when handling extremely long sequences because both its computation and memory complexity scale as $O(N^2)$ with sequence length. To address this problem, linear attention models have returned to the research spotlight. Among them, DeltaNet [1] proposes a Delta Rule-based linear attention mechanism. By introducing a recurrent state update with a forgetting mechanism, DeltaNet maintains linear complexity while achieving performance comparable to, or even better than, traditional Transformers.

However, when processing extremely long sequences, such as 100K or even 1M tokens, the activation memory often cannot fit on a single GPU. Moreover, even within one GPU, we still need to split long sequences and distribute the work across different compute units in order to fully utilize all streaming multiprocessors (SMs). This motivates the need for context parallelism (CP) [2][3].

This article discusses how to design context parallelism for DeltaNet. The parallelization logic described here applies both to different SMs within a single GPU, that is, chunk-wise parallel acceleration, and to distributed context parallelism across multiple GPUs, such as Ring Attention [4] or communication-based CP.

2. Core Computation in DeltaNet

For DeltaNet, the core attention computation can be decomposed into three main steps:

  1. WY representation computation: compute local projection matrices such as Query, Key, and Value.
  2. Recurrent state update: update the hidden state $\mathbf{S}$ in a recurrent manner.
  3. Output computation: compute the final output $\mathbf{O}$ from the updated state $\mathbf{S}$.

Among these steps, step 1 and step 3 do not contain recurrent dependencies across time steps, so different chunks of the sequence can be computed fully in parallel. The real obstacle to parallelization lies in step 2.

Following the derivation in Dr. Songlin Yang’s blog post DeltaNet Explained (Part II) [5], the chunk-wise state update of DeltaNet is:

$$ \begin{align*} \mathbf{S}{[i+1]} &= \mathbf{S}{[i]} (\mathbf{I}-\mathbf{W}{[i]}^\top \mathbf{K}{[i]}) + \mathbf{U}{[i]}^\top \mathbf{K}{[i]} \\ &= \mathbf{S}{[i]} + \left(\mathbf{U}{[i]} - \mathbf{W}{[i]}\mathbf{S}{[i]}^\top\right)^\top \mathbf{K}_{[i]} && \in \mathbb{R}^{d\times d}
\end{align*} $$

Here, $\mathbf{S}{[i]} := \mathbf{S}{iC} \in \mathbb{R}^{d \times d}$ denotes the initial state of the $i$-th chunk. We can see that computing $\mathbf{S}{[i+1]}$ requires $\mathbf{S}{[i]}$ to be available first. This strong dependency is the main obstacle to context parallelism.

3. Mathematical Derivation of Context Parallelism

To break this serial dependency, we need to decouple the computation using properties of linear systems.

Suppose we want to apply context parallelism to DeltaNet. Consider a simple example: the total sequence length is 8192, and the DeltaNet chunk size is set to 64. The full sequence is therefore split into 128 chunks, whose initial states are $\mathbf{S}{[0]}, \mathbf{S}{[1]}, \dots, \mathbf{S}_{[127]}$.

Without context parallelism, these 128 states are computed serially on the same SM: $\mathbf{S}{[1]}$ is computed from $\mathbf{S}{[0]}$, then $\mathbf{S}{[2]}$ is computed from $\mathbf{S}{[1]}$, and so on.

Now suppose we introduce context parallelism with parallel degree 2, namely CP=2, and distribute the computation across 2 SMs or 2 GPUs. We evenly split the sequence into two subsequences, each containing 64 chunks, or 4096 tokens: