Soofi 聯盟發布 Soofi S 30B-A3B：一個開放混合 Mamba-Transformer MoE 基礎模型，支援德語與英語

A German research consortium has published the pretraining report for Soofi S 30B-A3B. It is an open base model for German and English. Training ran end to end on Deutsche Telekom’s Industrial AI Cloud in Munich. Preview weights are on Hugging Face. It is worth noting that among some of the fully open base models tested, Soofi S records the highest English and German aggregate scores. What is Soofi S 30B-A3B? Soofi S is a Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model. It totals ~31.6B parameters and activates ~3.2B per token. As a base model, it has no instruction tuning, alignment, or safety tuning. The KI Bundesverband coordinates the consortium, funded by the German Federal Ministry for Economic Affairs and Energy. Participants include Fraunhofer IAIS, DFKI, TU Darmstadt, ellamind, and Merantix Momentum. How the architecture works? The efficiency claim starts with the layer stack. The network holds 52 layers. That is 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers, and 6 Grouped-Query Attention (GQA) layers. Only those 6 GQA layers maintain a KV cache. Each MoE layer holds 128 routed experts, activates 6 per token, and adds 2 shared experts. Other details: model dimension 2688, squared ReLU, RMSNorm, and no positional embeddings. Soofi S adopts the Nemotron 3 Nano reference design without modification. The research team gives three reasons for that choice. Those are deployability on stacks such as vLLM, serving efficiency, and scientific control. Because the backbone is fixed, Nemotron 3 Nano becomes an architecture-identical baseline. The data recipe is the only moving part. #mtp-soofi-embed{margin:22px 0;} #mtp-soofi-embed hr,#mtp-soofi-embed p:empty,#mtp-soofi-embed del,#mtp-soofi-embed s{display:none!important} #mtp-soofi-frame{width:100%!important;border:0!important;display:block!important;background:#0E0E0E!important;border-radius:14px!important;overflow:hidden!important;color-scheme:dark!important} @media(max-width:640px){#mtp-soofi-embed{margin:16px 0}} (function(){ var f=document.getElementById('mtp-soofi-frame'); if(!f) return; window.addEventListener('message',function(e){ if(e&&e.data&&typeof e.data.sfHeight==='number'&&e.data.sfHeight>200){ f.style.height=Math.round(e.data.sfHeight)+'px'; } },false); })(); The training recipe: ~26.68T consumed tokens in three phases That recipe follows a Warmup–Stable–Decay (WSD) schedule with a minus_sqrt decay segment. Phase 1 consumed ~20T tokens on a diverse, quality-tiered mixture at a 1e-3 plateau. Phase 2 consumed ~6.58T tokens of high-quality annealing data. It decays 1e-3 to 1e-5, then continues at a constant 1e-5. Phase 3 consumed ~0.10T tokens at a 1,048,576-token sequence length. It extends the usable context window up to 1M tokens. German is the deliberate variable. It rises from 7.2% of Phase 1 effective tokens to 15.32% in Phase 2. The reference Nemotron 3 Nano mixture allocates about 5% to all non-English languages combined. German sources include HPLT v3 and v4, German Commons, German FinePDFs, and FineWiki. Genios adds 193M articles from 916 newspaper and trade-press archives, commercially licensed. Infrastructure follows the same sovereignty logic. The run used up to 512 NVIDIA B200 GPUs, from 24 March to 13 May 2026. It consumed ~253,000 B200 GPU-hours. Performance Those choices show up in the evaluation. Soofi S ran against 16 other open base models. All used the same lm-evaluation-harness pipeline, prompts, and few-shot settings. Benchmark (%)Soofi S 30B-A3BOlmo 3 32BApertus 70BEuroLLM 22BAlia 40BEnglish aggregate70.167.362.461.259.0German aggregate79.169.272.870.668.4Held-out (EN / DE)41.4 / 41.833.1 / 36.227.6 / 33.530.8 / 33.928.0 / 29.4HumanEval (pass@1)73.863.030.239.323.8MBPP-DE (pass@1)84.270.850.959.445.6LBPP (pass@1)31.032.16.410.78.6GSM8K86.180.765.425.165.4Minerva MATH-DE56.048.529.028.412.9INCLUDE-DE61.248.250.451.143.9GPQA-Diamond43.433.327.330.329.8GLP-DE88.873.081.278.265.4 Against its architecture-identical reference, Soofi S gains 1.8 points on the English aggregate. German gains 4.2, and held-out English 6.7. That isolates the data recipe from the backbone. The picture changes against larger open-weight models. Qwen3.5 35B-A3B holds the highest English, German, and held-out means. Soofi S scores 70.1 English against 70.3 for Gemma 3 27B and Ministral 3 14B. On German it leads both, 79.1 to 78.4 and 78.3. Running the base model Reproducing any of this starts with the weights. The base repo is a gated preview, and it ships custom modeling code. Copy CodeCopiedUse a different Browser# pip install -U transformers accelerate torch # hf auth login # base repo is gated: accept the terms on the model page first from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Soofi-Project/Soofi-S-Base" tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, dtype="auto", device_map="auto" ) # Base model: plain text completion. No chat template, no system prompt. prompt = "AI sovereignty is the idea that" inputs = tok(prompt, return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=128) print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) The same repo serves through vLLM: Copy CodeCopiedUse a different Browservllm serve "Soofi-Project/Soofi-S-Base" Where it fits? Together, the numbers suggest three deployment shapes. First, German document work: GLP-DE 88.8 and INCLUDE-DE 61.2 suit an insurer fine-tuning on policy PDFs. Second, bilingual code assistance: MBPP-DE 84.2 suits teams prompting in German against Python tasks. Third, high-concurrency long-context serving: a support-ticket RAG system at batch 32 and 40K context matches the measured regime. For that case, test retrieval against the RULER and NaturalQuestions gaps. Key Takeaways Soofi S activates 3.2B of 31.6B parameters; only 6 of 52 layers hold a KV cache. It leads fully open base models: 70.1 English aggregate, 79.1 German aggregate. German hits 15.32% of the Phase 2 mixture, versus ~5% multilingual in Nemotron. Decode measures 8–9× dense 14–24B models at 40K context, flat from 4K to 256K. Open gaps: RULER extraction at long inputs, factual recall, gated preview weights, unfinalized license. Check out the Pretraining report, Project page and Hugging Face . Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Soofi Consortium Releases Soofi S 30B-A3B: An Open Hybrid Mamba-Transformer MoE Foundation Model For German And English appeared first on MarkTechPost.