TurboSparse Efficiency: Achieving 97% Parameter Sparsity in Mixtral-47B

Discover how TurboSparse-Mistral-7B and Mixtral-47B leverage ReLUfication to reach up to 90% neuron inactivity, reducing active parameters to just 3% per MoE layer.