Linear Transformer Model

Switch language: Linear Transformer 模型

M7 is the linear-attention alternative to the standard Transformer.

Key motivation

Standard attention has quadratic dependence on sequence length. For genomic sequences, that can become a serious bottleneck. The linear Transformer replaces the softmax attention form with a kernelized approximation so that attention can be computed in linear-time style form with respect to sequence length.

Project implementation highlights

• Input: one-hot sequence tensor projected into model space

• Positional encoding: sinusoidal

• Attention: ELU+1 kernel feature map

• Pooling: mean pooling over sequence positions

The key approximation replaces softmax attention with feature maps:

\[\mathrm{Attn}(Q, K, V) \approx \frac{\phi(Q)\big(\phi(K)^\top V\big)} {\phi(Q)\big(\phi(K)^\top \mathbf{1}\big)}\]

where \(\phi(\cdot)\) is a positive kernel feature map such as ELU+1. This reorders the computation so sequence length scaling becomes linear in style.

Why it matters for EPI

This model asks an important question: can we keep the global interaction flavor of attention while scaling better to long DNA sequences than a standard Transformer?

That question matters directly for genomic inputs because long-range regulatory dependencies are scientifically relevant, but full quadratic attention becomes costly exactly in the length regime where those dependencies matter most.

Strengths

• better scaling behavior than quadratic attention;

• no mandatory CNN token compression step;

• useful for long-range sequence modeling studies.

Computational complexity

• Time: linear-attention style evaluation reduces dependence on sequence length to approximately \(O(T \cdot d^2)\) or similar implementation-dependent linear form, avoiding full \(T^2\) attention maps.

• Memory: more favorable than standard attention because full pairwise token matrices are not materialized.

• Best-fit regime: attractive for long windows where global-style interaction is desired but quadratic attention becomes impractical.

Limitations

• linearized attention is an approximation, not a drop-in perfect substitute for full softmax attention;

• quality depends on whether the approximation preserves the interactions most relevant to the task.