Basic

基础数学原理

• kl-divergence
• negative-log-likelyhood

Transformer 模型结构

Embedding

• rope
• rope-length-extrapolation
• over-encoding

Attention

• attention-basic

• flash-attention

  • flash-attention-varlen
  • flash-attention-v2
  • ring-attention

• paged-attention

• Full Attention

  • multi-head-attention
  • gated-attention
  • deepseek-mla
  • sliding-window-attention

• Sparse Attention

  • native-sparse-attention
  • moba
  • deepseek-sparse-attention

• linear-attention

  • delta-net
  • gated-delta-net
  • lightning-attention
  • mamba

• flex-attention

MLP

• swiglu

MoE

• moe
• deepseek-moe
• deep-ep

VLM

• vlm

Huggingface

• huggingface-transformers

Training

N-D 并行

• fsdp
• fsdp2
• torchtitan
• dtensor
• torch-tp
• pipeline-parallel
• ulysses-sp
• ring-attention
• context-parallel
• fsdp2-ep-tp
• deep-ep
• vocab-tp
• ib2nvlink
• device-mesh

Comm-Compute Overlap

• flux
• triton-distributed
• comet
• deep-ep
• bumi-ep
• ring-ep

低精度训练

• mixed-precision2
• mixed-precision-fp8

负载均衡

• sequence-packing
• rl-balance
• eplb

LongContext

• long-context-llm
• ulysses-sp
• ring-attention

Dataload & Ckpt

• byte-checkpoint
• streaming-dataloader

Inference

Decode

• spec-decode
• mtp
• medusa
• looking-ahead-decoding
• eagle

KV-Cache

• kv-cache-basic
• kv-cache-optimize
• kv-cache-long-context
• kv-cache-split-or-not

PD 分离/合并

• prefill-decode-disaggregate
• chunked-prefill
• orca

Quant

• quant

Kernel

• triton
• deep-ep

RL 强化学习

RL Algo

• rl-math
• rl-llm
• rl-async
• traininig-inference-mismatch
• routing-replay

RL System

• rl-colocate
• rl-math
• rl-llm
• rl-agent
• rl-balance
• rl-off-policy

Post-Training Recipes

• on-policy-distill
• mixed-rl
• post-training
  • sft
  • lora

MaaS

• tinker

Hardward

• tpu
• nvidia-gpu-mig
• 910c
• cambricon
• tpu
• dojo
• colossus
• mi300x
• mlperf

通信

• nccl-explained
• alltoall
• torch-tensorstore
• nvshmem

Misc

• moe

• streaming

• flight-recorder

• gb300

• rlhf

• the-ultra-scale-playbook

• how-to-scale-your-model

• swe-agent

• deep-research

• veRL

• navit

• infer-serving

• multimodal

• robotics

• alphafold

• pathways

• dl-based-nlp

• llama3

• diffusion

• diffusion-model

• image-generation

Kernels

• softmax
• layernorm
• rmsnorm
• dropout