**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**

Back to Articles Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization Enterprise + Article Published September 23, 2026 Upvote 1 Francesco fciannella Follow nvidia Maryam Motamedi maryameee Follow nvidia Taejin Park taejinp Follow nvidia Ivan Medennikov imedennikov Follow nvidia Adi- margolin Amargolin Follow nvidia Turn overlapping conversations into speaker-aware data with one open-weight, 100M-parameter model - ranked #1 in Voice Arena's initial Diarization-Bench results.

Why speaker diarization matters Every conversation carries two layers of information: what was said and who said it.Speech recognition captures and transcribes the words.Speaker diarization classifies who spoke when, helping applications connect what was said to the right participant.

Consider a transcript from a meeting, customer call, or podcast in which every sentence is correct but none is attributed to a speaker.You can read the words, but you cannot reliably tell who made a commitment, who raised an objection, or which participant interrupted.

Search, summaries, action items, conversation analytics, and voice-agent memory all become less useful.Speaker diarization identifies the time intervals during which each speaker is active, including intervals where people talk over one another.

Those speaker timestamps can then be combined with automatic speech recognition (ASR) to create a speaker-attributed transcript.NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on Voice Arena's Diarization-Bench leaderboard with a 14.

72% Diarization Error Rate (DER).Supporting up to eight speakers across live and recorded conversations, it handles overlapping speech, chunked processing for flexible recording lengths, and customizable streaming latency.

Earlier models like NVIDIA Streaming Sortformer established this approach for four-speaker diarization, including the streaming diarstreamingsortformer4spk-v2.1 checkpoint used as the baseline below.

Nemotron 3 Diarization expands support to eight speakers and improves the accuracy and throughput measured in the following evaluations.How Nemotron 3 Diarization works One model for offline and streaming conversations Diarization systems must solve two related problems.

First, they must detect speech and assign it to the correct speaker.Second, they must preserve that assignment throughout the conversation, even after silence, interruptions, or long gaps between a speaker's turns.Streaming makes the second problem harder.

An offline model can inspect an entire recording at once.A streaming system receives only a small chunk of new audio and limited context.Without an effective memory mechanism, the speaker assigned to one output channel in the current chunk can be assigned to a different channel in the next.

Nemotron 3 Diarization follows the Sortformer approach of ordering output speakers by when they first appear.The first new voice becomes the first speaker channel, the next new voice becomes the second, and so on.

This arrival-time ordering makes the model's generic speaker labels stable and removes the need to solve a new speaker permutation for every chunk.

Nemotron 3 Diarization was trained using public and licensed speech data, including multispeaker-annotated, real-world conversations licensed from David AI.Additional licensed David AI audio provided source material for large-scale simulated English and multilingual mixtures spanning 21 languages.

Adding David AI data to our training decreased compound diarization error rate (DER) by 0.77 absolute points, from 11.19% to 10.42%, at both the offline-style and ultra-low-latency operating points.The model supports as many as eight speaker channels.

These are anonymous labels, not real-world identities: the model can report that speaker2 spoke from one timestamp to another, but it does not determine that speaker2 is a particular person.

Downstream applications can map these anonymous channel IDs to explicit speaker identities by pairing timestamps with meeting metadata, user profiles, or active speaker verification models.

From audio to speaker activity Architecture flow for Nemotron 3 DiarizationThe model accepts 16 kHz, single-channel audio and converts it into Mel-spectrogram features with a 10 ms frame step.

It stacks those features by a factor of eight, producing 80 ms frames for a 31-layer Transformer encoder with rotary positional embeddings (RoPE).Figure 1.Nemotron 3 Diarization converts audio into arrival-ordered speaker-activity probabilities.AOSC and FIFO context support streaming inference.

Above the Transformer, a Conv1D layer upsamples the predictions to the input-feature resolution.The default output is a [T, 8] floating-point tensor: T time steps by eight possible speaker channels.Each value is the probability that a speaker is active at that time.

The default stride is 10 ms and can be configured to another multiple of 10 ms.This representation handles overlap naturally.If two people speak at the same time, two channels can be active in the same frame.

Postprocessing converts these probabilities into generic speaker labels with start and end timestamps.

During streaming inference, two forms of memory provide context: The Arrival-Order Speaker Cache (AOSC) retains useful information about speakers observed in earlier chunks, organized according to their arrival-ordered channels.

A first-in, first-out (FIFO) queue supplies recent frame context before the current chunk.The input buffer also includes right context, which is audio immediately after the current chunk.

More right context can help the model interpret speaker transitions, while less right context reduces the time it must wait before producing a result.Together, the current chunk, right context, FIFO queue, and speaker cache enable one model to operate at several latency points.

Chunked inference removes a fixed model-imposed maximum audio duration.Performance can still degrade on unusually long recordings or audio with severe noise, reverberation, far-field capture, or domain shift.

Diarization and Speaker Attributed ASR (Multi-speaker ASR) are different tasks Standalone diarization produces speaker activity and timestamps, not the words being spoken.ASR produces text but does not necessarily preserve speaker attribution.

A speaker-attributed transcription pipeline combines both outputs: Figure 2.End-to-end speaker-attributed transcription pipeline combining audio diarization timestamps with automatic speech recognition (ASR) to map spoken words to specific speakers.That separation matters when designing a system.

Diarization errors include missed speech, false speech detections, incorrect speaker assignments, and boundary errors.ASR errors affect the words.Applications should evaluate both components and the combined pipeline on their intended audio.

Balance latency and accuracy The same model supports recommended input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds.Shorter buffers let the system respond sooner, while more context generally improves accuracy and throughput.

These values measure the audio buffered before inference; computation, networking, ASR, and application processing add to end-to-end latency.Although the model can technically use an 80 ms input buffer, 0.32 seconds is the lowest recommended configuration.

The configuration table in "Get started" shows how to select an operating point.

Benchmark results: Ranked #1 in Voice Arena's initial Diarization-Bench In Voice Arena's initial Diarization-Bench results, NVIDIA Nemotron 3 Diarization ranked first among 12 systems and 17 total system configurations evaluated across 139 English-language conversations totaling approximately 22 hours.

With overlapping speech scored, system-generated speech activity detection, and no boundary collar, Nemotron 3 Diarization achieved a 14.72% diarization error rate (DER), compared with 19.3% for the next-ranked system - a ~24% relative reduction.

It also ranked first with 100 ms and 250 ms collars and across both in-person and online recordings.These initial results may change as Voice Arena completes its Version 1 evaluation and paired statistical analysis.Figure 3.

Voice Arena Diarization-Bench v1 results (English, 0 ms collar, DER lower is better), showing NVIDIA Nemotron 3 Diarization ranking #1 with 14.72% DER.The figure below uses the Nemotron 3 Diarization evaluation results.

They compare the model with NVIDIA's previous four-speaker Streaming Sortformer under the disclosed scoring and throughput settings.Figure 4.Diarization Error Rate (DER) performance across eight public benchmarks at 1.

04-second output latency, showing Nemotron 3 Diarization state-of-the-art results compared to the previous NVIDIA Sortformer baseline.Measuring diarization accuracy The primary metric used to evaluate the model is diarization error rate (DER).

It combines three kinds of error: Missed speech: a reference speaker was active, but the system detected no corresponding speech.False alarm: the system marked a speaker as active when the reference contained no corresponding speech.

Speaker confusion: the system detected speech at the right time but assigned it to the wrong speaker.Figure 5.DER adds missed speech, false alarms, and speaker confusion, then divides by total reference speaker time.Overlapping reference speakers each contribute to the denominator.

Benchmark settings can materially change DER, so the evaluation protocol is part of the result.

The Nemotron 3 evaluation contains 901 condition-specific recordings spanning multilingual telephone speech, meetings, near-field and far-field microphones, multi-microphone capture, and difficult acoustic environments.Overlapping speech is scored in every evaluation.

DIHARD III, AliMeeting, AMI, and NOTSOFAR1 use a zero-second collar, meaning no boundary tolerance is excluded from scoring.CALLHOME-Part2 uses a 0.25-second collar.Results were generated with the NeMo e2ediarizespeech.py evaluation script.The comparison below uses diarstreamingsortformer4spk-v2.

1, NVIDIA's previous four-speaker streaming Sortformer, as the baseline.It uses the final Nemotron-3-Diarization values rather than preview-model results.An average 40% relative DER reduction at 1.04-second latency Figure 6.Full-set DER for the final model and previous NVIDIA baseline at 1.

04-second input-buffer latency.Lower is better.At 1.04-second input-buffer latency, Nemotron 3 Diarization reduces DER on all eight listed evaluation conditions.The relative reductions range from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM.

The unweighted mean of the eight per-dataset relative reductions is 41.0%.In other words, this is an average of relative improvements across evaluation conditions; it is not a pooled DER computed by combining every recording into one score.The improvement is also consistent across operating points.

At each latency shared by the two models (30.4, 1.04, and 0.32 seconds), the final model has lower full-set DER on every evaluated dataset.Figure 7.Full-set DER across input-buffer latency settings.The previous baseline does not have a 0.64-second configuration.

Improvements increase in higher-speaker-count conditions Support for eight speakers enables meetings and group conversations with more than four participants.The benchmark advantage also widens in the higher-speaker-count subsets of DIHARD III, CALLHOME-Part2, and NOTSOFAR1.Figure 8.DER at 30.

4-second input-buffer latency, broken out by speaker count.Shaded regions contain more than four speakers.The figure also preserves an important nuance: on the two-speaker CALLHOME subset, the final model records 5.98% DER compared with 5.68% for the previous baseline.

Across the full CALLHOME-Part2 evaluation, however, DER improves from 10.32% to 9.10%, with larger gains in the higher-speaker-count subsets.DIHARD III groups recordings