mLSTM Model
Switch language: mLSTM 模型
M5 implements the matrix-memory LSTM route inspired by xLSTM-style design.
What makes mLSTM different
Unlike a conventional LSTM that stores vector-valued hidden memory, mLSTM uses a matrix-style memory update. In this repository, the design is motivated by the idea that richer recurrent state may capture more structured sequence dependencies than standard recurrent cells.
Core idea
query, key, and value projections are formed at each sequence step;
memory is updated as a matrix rather than a simple vector state;
gating is stabilized in log space;
a bidirectional wrapper is used so the encoder sees context from both sides.
One abstract view of the memory update is:
where \(C_t\) is matrix-valued memory, \(v_t \otimes k_t\) denotes a structured outer-product write, and \(i_t, f_t\) control write and retain behavior. Compared with vector memory, this allows the state to preserve richer interaction structure.
Why it is interesting for genomic sequences
Genomic regulatory sequences often involve combinational dependencies rather than isolated motifs. Matrix-memory recurrence is attractive because it can, in principle, preserve richer interaction structure across positions than a simple hidden vector.
That becomes relevant when one motif changes the role of another or when small groups of motifs behave as interacting units rather than independent pattern matches.
Strengths
richer recurrent state than vanilla LSTM;
explicit recurrence without relying on quadratic attention;
useful as a modern recurrent alternative in long-sequence benchmarking.
Computational complexity
Time: still sequential in length, but with heavier per-step state updates than a vanilla LSTM because matrix-style memory is maintained.
Memory: higher recurrent-state cost than BiLSTM, since the model preserves a richer internal memory object rather than only vector hidden states.
Best-fit regime: valuable when sequence order and structured interaction memory matter more than raw throughput.
Tradeoffs
more specialized and harder to reason about than plain BiLSTM;
still sequential in spirit, even if conceptually more expressive;
more implementation complexity than simpler baselines.
Project role
M5 is the project’s advanced recurrent representative. It helps position the
benchmark beyond classical LSTMs and gives the documentation a bridge between
legacy sequence models and newer state-space or attention families.
