Microsoft’s SAMBA Model Redefines Long-Context Learning for AI

SAMBA is a hybrid neural architecture that effectively processes very long sequences by combining Sliding Window Attention (SWA) with Mamba, a state space model (SSM). SAMBA achieves speed and memory efficiency by fusing the exact recall capabilities of attention with the linear-time recurrent dynamics of Mamba. SAMBA surpasses Transformers and pure SSMs on important benchmarks like MMLU and GSM8K after being trained on 3.2 trillion tokens with up to 3.8 billion parameters.