Add torch scaled dot product attention (FlashAttention) - #1798

Draft

dorian-K wants to merge 38 commits into

Commits on Dec 19, 2025

WIP scaled dot product attention
dorian-K
committed
b814a5f
wip
dorian-K
committed
b69bbfd
wip
dorian-K
committed
1194018
wip
dorian-K
committed
9a2cae2
fix formatting
dorian-K
committed
e5f636c
more
dorian-K
committed
a0627ec
more
dorian-K
committed
90316e9
fix tests
dorian-K
committed
b91ea9d
more
dorian-K
committed
54beb8f
more
dorian-K
committed
f9adbf3
fix
dorian-K
committed
eada973
fix pycharm
dorian-K
committed
62f825e

Commits on Jan 9, 2026

Merge branch 'master' into doriank-sdpa
dorian-K
committed
6c8fd3f
remove debug prints, update
dorian-K
committed
c8bd06b
more
dorian-K
committed
f08d16a
use is_causal=True
dorian-K
committed
2de26f8
add back a test
dorian-K
committed
ecb79ed
more tests
dorian-K
committed
a1b6c5d
fix test
dorian-K
committed
6eb0906

Commits on Jan 16, 2026

more tests
dorian-K
committed
c8b7c11
Merge branch 'master' into doriank-sdpa
dorian-K
committed
b31e9b8
fix formatting
dorian-K
committed
d6e25f9
fix import
dorian-K
committed
8480c3d
remove some formatting only changes
dorian-K
committed
6b35b43
more sdpa vs fallback tests
dorian-K
committed
ef324be
Add benchmarks
dorian-K
committed
4480aa5

Commits on Jan 23, 2026

Merge branch 'master' into doriank-sdpa
dorian-K
committed
5469e1e
force flash att
dorian-K
committed
547d139
also upgrade cross attention
dorian-K
committed
4962ced
add back att_dropout_broadcast
dorian-K
committed
f91e5be
convert some asserts to exceptions
dorian-K
committed
33a045b
fix private member
dorian-K
committed
322112c
oops
dorian-K
committed
96470b4
reduce line length
dorian-K
committed
6d3a585

Commits on Jan 30, 2026

Make tensors contigous to enable fused kernels
dorian-K
committed
0162ef3
_embed_dim -> _feat_dim
dorian-K
committed
7da3b66
pycharm
dorian-K
committed
315fd86
Merge branch 'master' into doriank-sdpa
dorian-K
committed
a388048