使用 NVIDIA srt-slurm、SLURM 配方、參數掃描與帕累託分析驗證分散式 LLM 服務基準測試

In this tutorial, we explore NVIDIA’s srt-slurm framework and learn how we use srtctl to convert declarative YAML configurations into reproducible SLURM benchmark workflows for distributed LLM serving.

We set up the project in Google Colab, inspect its internal architecture, define a cluster configuration, dry-run built-in and custom recipes, and model a disaggregated prefill-and-decode deployment for DeepSeek-R1.

We also generate parameter sweeps, interact with the typed Python API, validate expanded configurations, and analyze simulated benchmark results through a throughput-versus-latency Pareto frontier.

Although Colab does not provide a real SLURM environment, we use it as a practical development workspace to understand, validate, and prepare production-grade benchmark recipes before we submit them to an actual GPU cluster.

Copy CodeCopiedUse a different Browserimport os, sys, subprocess, textwrap, json, shutil, importlib from pathlib import Path def run(cmd, check=True, quiet=False): """Run a shell command, stream output.""" print(f"\n$ {cmd}") r = subprocess.

run(cmd, shell=True, text=True, capture_output=True) out = (r.stdout or "") + (r.stderr or "") if not quiet: print(out[-6000:]) if check and r.returncode != 0: raise RuntimeError(f"Command failed ({r.

returncode}): {cmd}") return out def section(title): print("\n" + "═"*78 + f"\n {title}\n" + "═"*78) section("1.Install srt-slurm") REPO = Path("/content/srt-slurm") if Path("/content").exists() else Path.cwd()/"srt-slurm" if not REPO.exists(): run(f"git clone --depth 1 https://github.

com/NVIDIA/srt-slurm.git {REPO}", quiet=True) run(f"{sys.executable} -m pip install -q -e {REPO}", quiet=True) sys.path.insert(0, str(REPO / "src")) importlib.invalidate_caches() os.

chdir(REPO) run("srtctl --help") We prepare the Colab environment by importing the required modules and defining reusable helper functions for command execution and section formatting.

We clone the NVIDIA srt-slurm repository, install it in editable mode, and expose its source directory to the active Python runtime.We then switch to the repository directory and verify that the srtctl command-line interface is installed correctly.Copy CodeCopiedUse a different Browsersection("2.

Repository architecture") print(textwrap.dedent(""" src/srtctl/ cli/ submit.py (apply/dry-run/preflight/monitor), do_sweep, interactive core/ schema.py (typed config), sweep.py, slurm.py (sbatch gen), validation.py, health.py, topology.py, fingerprint.py backends/ sglang.py, trtllm.py, vllm.

py, mocker.py ← engine adapters frontends/ Dynamo / router frontends templates/ Jinja2 → sbatch + orchestrator scripts recipes/ ready-made benchmarks per platform (gb200-fp4, h100, b200-fp8, qwen3-32b, dsv4-pro, mocker smoke tests, ...

) analysis/ srtlog (log parsers) + Streamlit dashboard (Pareto, latency...) docs/ sweeps.md, profiling.md, analyzing.md, config-reference.md """)) for d in ["recipes", "docs"]: print(f"{d}/ →", ", ".join(sorted(p.name for p in (REPO/d).iterdir()))[:300]) section("3.Cluster configuration (srtslurm.

yaml)") (REPO/"srtslurm.yaml").write_text(textwrap.

dedent("""\ cluster: "colab-demo" default_account: "demo-account" default_partition: "gpu" default_time_limit: "01:00:00" gpus_per_node: 4 use_gpus_per_node_directive: true use_segment_sbatch_directive: true containers: dynamo-sglang: "/containers/dynamo-sglang.sqsh" lmsysorg+sglang+v0.5.5.post2.

sqsh: "/containers/sglang-v0.5.5.sqsh" model_paths: deepseek-r1: "/models/DeepSeek-R1" """)) print((REPO/"srtslurm.yaml").read_text()) We inspect the repository structure to understand how srtctl organizes its command-line tools, schemas, backends, templates, recipes, and analysis components.

We then create a local srtslurm.yaml file containing simulated cluster defaults, container aliases, GPU settings, and model paths.We use this configuration to resolve recipe references in Colab without requiring access to an actual SLURM cluster.Copy CodeCopiedUse a different Browsersection("4.

Dry-run: mocker smoke test → generated sbatch script") run("srtctl dry-run -f recipes/mocker/agg.yaml", check=False) section("5.Custom disaggregated recipe (prefill/decode split)") (REPO/"my-disagg.yaml").write_text(textwrap.

dedent("""\ name: "colab-disagg-demo" model: path: "deepseek-r1" container: "lmsysorg+sglang+v0.5.5.post2.

sqsh" precision: "fp8" resources: gpu_type: "gb200" gpus_per_node: 4 prefill_nodes: 1 decode_nodes: 2 prefill_workers: 1 decode_workers: 2 backend: prefill_environment: { PYTHONUNBUFFERED: "1" } decode_environment: { PYTHONUNBUFFERED: "1" } sglang_config: prefill: served-model-name: "deepseek-ai/DeepSeek-R1" model-path: "/model/" trust-remote-code: true kv-cache-dtype: "fp8_e4m3" tensor-parallel-size: 4 disaggregation-mode: "prefill" decode: served-model-name: "deepseek-ai/DeepSeek-R1" model-path: "/model/" trust-remote-code: true kv-cache-dtype: "fp8_e4m3" tensor-parallel-size: 4 disaggregation-mode: "decode" benchmark: type: "sa-bench" isl: 1024 osl: 1024 concurrencies: [64, 128, 256] req_rate: "inf" """)) run("srtctl dry-run -f my-disagg.

yaml", check=False) We dry-run the built-in mocker recipe to examine how srtctl validates configurations and generates SLURM submission artifacts without executing a real benchmark.

We then define an advanced DeepSeek-R1 recipe that separates prefill and decode workloads across independent node and worker pools.We validate this disaggregated SGLang configuration through another dry run and inspect how the serving parameters are translated into job scripts.

Copy CodeCopiedUse a different Browsersection("6.Parameter sweep (grid search) — dry-run + expansion on disk") run("srtctl dry-run -f examples/example-sweep.yaml", check=False) sweep_dirs = sorted((REPO/"dry-runs").

glob("example-sweep_sweep_*")) if sweep_dirs: latest = sweep_dirs[-1] print("Per-job configs generated by the sweep expander:") for p in sorted(latest.rglob("config.yaml")): print(" ", p.relative_to(REPO)) section("7.Programmatic use of srtctl's Python API") import yaml from srtctl.core.

config import load_config from srtctl.core.sweep import generate_sweep_configs, expand_template from srtctl.core.schema import BenchmarkType, Precision, GpuType cfg = load_config("my-disagg.yaml") print(f"Loaded : {cfg.name}") print(f"Model : {cfg.model.path} ({cfg.model.precision}) on {cfg.

resources.gpu_type}") print(f"Layout : {cfg.resources.prefill_nodes}P + {cfg.resources.decode_nodes}D nodes, " f"{cfg.resources.gpus_per_node} GPUs/node") print(f"Bench : {cfg.benchmark.type} isl={cfg.benchmark.isl} osl={cfg.benchmark.osl} " f"concurrencies={cfg.benchmark.

concurrencies}") print(f"Enums : benchmarks={[b.value for b in BenchmarkType]}") print(f" precisions={[p.value for p in Precision]}, gpus={[g.value for g in GpuType]}") raw_sweep = yaml.safe_load(Path("examples/example-sweep.yaml").

read_text()) jobs = generate_sweep_configs(raw_sweep) print(f"\nSweep expands to {len(jobs)} jobs:") for job_cfg, params in jobs: pf = job_cfg["backend"]["sglang_config"]["prefill"] print(f" {params} → chunked-prefill-size={pf['chunked-prefill-size']}, " f"max-total-tokens={pf['max-total-tokens']}") print("\nTemplate substitution:", expand_template({"flag": "{x}", "n": "{y}"}, {"x": 4096, "y": 2})) We execute the example parameter sweep and inspect the individual job configurations created from its Cartesian search space.

We load our custom recipe through the typed Python API and examine its model, precision, GPU topology, benchmark settings, and supported enumeration values.We also programmatically expand sweep templates and verify how each parameter combination affects the generated backend configuration.

Copy CodeCopiedUse a different Browsersection("8.Analysis: Pareto frontier from (simulated) benchmark results") import numpy as np, matplotlib.pyplot as plt rng = np.random.default_rng(0) def simulate(variant, base_tps, base_itl): rows = [] tps_gpu = base_tps * c / (c + 90) * rng.uniform(.97, 1.

03) itl = base_itl * (1 + c/220) *