Repository navigation
fla: add kdn kernels - #1010
fla: add kdn kernels#1010
Conversation
| from .modules.fused_norm_gate import rms_norm_gated | ||
|
|
||
|
|
||
| class FusedRMSNormGated(nn.Module): |
There was a problem hiding this comment.
This is for the future, e.g. we can use this for qwen 3.5
There was a problem hiding this comment.
If it's for the future, does it need to be present in this PR?
There was a problem hiding this comment.
Yes, confirmed that I will need this internally. I will check a bit if this is the correct variation before merge tho
| python-depends = ["einops"] | ||
|
|
||
| [general.hub] | ||
| repo-id = "kernels-community/fla" |
There was a problem hiding this comment.
What do we think about asking the partner if we could host the kernels under their namespace, similar to what we did for Minimax? Cc: @LysandreJik
For the build, we can push to kernels-community and then have one of the admins transfer it to their namespace. But after that, we can update this to the partner namespace (once there is agreement).
There was a problem hiding this comment.
So you mean under the FLA organization? Imo, that sounds good to me; wanted to credit them in shape or form in any case
There was a problem hiding this comment.
Yup. So something like FLA/fla or whatever their org namespace is.
| @@ -58,6 +58,7 @@ | |||
| "aiter-kernels": "https://github.com/ROCm/aiter", | |||
| # Too much stuff happening in vLLM, not worth reporting as MoE freshness. | |||
| "vllm-moe": "", | |||
| "fla": "github.com/fla-org/flash-linear-attention", | |||
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Let's get the nix formatting fixed.
|
Checking whether to add causal conv1d here as well so withholding for a bit until I've settled on that |
drbh
added
backend: cuda
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks. Let's try to transfer the kernel repo to the actual org if possible after it's been merged.
We're working on huggingface/kernels#694 which will allow us to open PRs to these repos (where we don't have write access) through our CI automatically :)
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Let's wait till the build is successful?
|
/kernel-bot merge-and-upload fla |
|
Build request processed. Command: Merge result: PR is already merged. Continuing with build/upload. Dispatched (1): |
3 participants
As per title, starts to add the kernels from FLA. Since these are mostly implemented in triton there is no extra deps for the most part. There are some tilelang implementations but would ignore them for now. Same kernels also over here for example: https://huggingface.co/kernels/AntonV/fla
I only focused on gdn and kdn but would like to gradually update this. Should this be version 0 then maybe.