Transformer-based Arch.

Transformer-XL G- a) Compressive Transformer

— 2019

| | |

“— 2020

an

Multi-Query Attention

Linear-complexity Arch.

zauryt

e/a

RoPE

2021 ——_—____

ne

Hybrid Arch.

LongFormer a | _ ~

Memformer

I~

Position Embedding

Linformer

2022

rf | Alibi

| | |

Performers G Hi-Transformer

an

\W—_ G&G RMT

RetNet

=

an

2023

one ERNIE-SPARSE

RWKV-4

\

Jamba

1-8

sek Transformer-VQ ae

Group-Query Attention G

Zamba

|

zyph

MOA

9-12

« ‘@) Mamba

il,

o~ StreamingLLM

Cy ma

TO /\

2024

| |

Megalodon

O,8)

an

yoco

YARN NOUS

\

OX) Br

Infini-Attention

G

1-4

|_| Samba

RWKV-5/6

NsA @Y

5-12

ReMamba

2025

Griffin G

MoBA 4 |

ms

MoM

1-4

Recurrent Gemma

G

Minimax-01 afin
