MAE Pretraining Model
Switch language: MAE 预训练模型
M14 is the self-supervised pretraining route of the repository.
Core idea
The model uses a Masked Autoencoder style workflow:
mask a large fraction of sequence patches;
encode the visible subset;
reconstruct the masked content;
reuse the learned encoder for supervised EPI prediction.
The reconstruction objective can be written as:
where \(\mathcal{M}\) is the masked patch set and \(\ell\) measures how well the decoder recovers hidden sequence content.
Why this is different from M13
M13 imports external pretrained genomic knowledge.
M14 learns a task-adjacent representation directly from the project’s own
data distribution through self-supervision.
Why this matters
For regulatory genomics, labels are valuable and often limited. A self-supervised route is attractive because it can:
extract structure from unlabeled or weakly labeled sequence data;
adapt the representation to the local data distribution;
provide a middle path between scratch training and large external foundation models.
Implementation logic
Encoder: Transformer-style sequence encoder
Pretraining objective: masked reconstruction
Finetuning objective: paired enhancer-promoter classification
This means the encoder is first optimized to model sequence structure without labels, then repurposed as a supervised feature extractor for the downstream interaction task.
Project role
This model gives the documentation a full representation-learning ladder:
scratch baselines;
advanced sequence architectures;
external foundation models;
in-project self-supervised pretraining.
Computational complexity
Time: pretraining is substantially more expensive than direct supervised training because reconstruction must be learned before downstream finetuning.
Memory: encoder-decoder pretraining and later finetuning make this route heavier than a simple task-only baseline.
Best-fit regime: appropriate when unlabeled sequence volume is available and a richer in-project representation is worth extra training cost.
