--- title: Silero VAD Explained: Modern Voice Activity Detection, Stateful On-Device Inference, and Real-Time Audio Streaming Pipelines publishedAt: 2026-08-30 summary: An architectural deep dive into Silero VAD: recurrent neural network foundations, stateful hidden tensor lifecycles, lock-free streaming audio pipelines, and hysteresis state machines. --- # Silero VAD Explained: Modern Voice Activity Detection, Stateful On-Device Inference, and Real-Time Audio Streaming Pipelines Voice Activity Detection (VAD) is a critical gateway component in modern conversational AI, real-time speech recognition (ASR), and WebRTC audio processing pipelines. Silero VAD addresses the limitations of traditional energy- and heuristic-based detectors by utilizing a compact, deep learning-based recurrent architecture that operates statefully on streaming audio chunks. Building a production-grade, low-latency streaming pipeline with Silero VAD requires rigorous stream synchronization, lock-free thread isolation, zero-allocation memory management, and dual-threshold hysteresis post-processing. ``` +---------------------------------------------------------------------------------------------------+ | REAL-TIME SILERO VAD PIPELINE | | | | [Audio Input] ---> [Resampler] ---> [SPSC Ring Buffer] ---> [Inference Thread] ---> [Hysteresis] | | (48kHz PCM) (16kHz PCM) (Lock-Free Frame) (ONNX + Recurrent (State | | Hidden State) Machine) | +---------------------------------------------------------------------------------------------------+ ``` --- ## What Is Silero VAD? Architecture & Recurrent Foundations Silero VAD is an open-source, lightweight deep learning-based Voice Activity Detector built on an acoustic feature encoder combined with a compact recurrent neural network (RNN / LSTM / GRU cells) optimized for speech/non-speech classification. By learning temporal speech representations across sequential frames rather than relying on hand-crafted spectral heuristics, the model maintains high boundary precision and robust speech discrimination even in non-stationary acoustic noise. The official model implementation is maintained in the [Silero VAD Official Repository](https://github.com/snakers4/silero-vad) and distributed via [PyTorch Hub: Silero VAD](https://pytorch.org/hub/snakers4_silero-vad/) and standalone ONNX packages. ``` +-----------------------------------------------------------------------+ | SILERO VAD CORE TOPOLOGY | | | | 1D Float32 PCM Chunk | | [x_0, ..., x_511] | | | | | v | | +---------------------------------------------------------------+ | | | Acoustic Feature Encoder (Temporal Convolutions / Subsampling)| | | +---------------------------------------------------------------+ | | | | | v | | +---------------------------------------------------------------+ | | | Recurrent Neural Network (LSTM / GRU Blocks) | | | | Hidden State Input: h_{t-1}, c_{t-1} | | | | Hidden State Output: h_t, c_t | | | +---------------------------------------------------------------+ | | | | | v | | +---------------------------------------------------------------+ | | | Linear Classification Head + Sigmoid Activation | | | +---------------------------------------------------------------+ | | | | | v | | Scalar Probability Output: p in [0.0, 1.0] | +-----------------------------------------------------------------------+ ``` ### Acoustic Feature Extraction & Recurrent Modeling Traditional voice activity detectors isolate speech by analyzing statistical signal characteristics such as short-time spectral energy, zero-crossing rates, and low-order Gaussian Mixture Models (GMMs). While these approaches are computationally lightweight, they degrade rapidly in the presence of dynamic acoustic noise (e.g., keyboard clicks, background babble, ambient transit noise). Silero VAD models the temporal dynamics of human phonation through a two-stage neural pipeline: 1. **Acoustic Feature Encoder**: The raw time-domain PCM signal is mapped into intermediate spectral representations through 1D convolutional layers with subsampling. This step reduces the dimensionality of the incoming raw waveform while retaining spectral formants essential for identifying human vocal tract resonance. 2. **Compact Recurrent Engine**: The extracted acoustic embeddings are fed into recurrent layers (LSTM or GRU blocks). The recurrent cells maintain temporal context across successive audio windows, enabling the network to differentiate between abrupt transient non-speech sounds (such as door slams or mic bumps) and sustained vocal fold vibrations. 3. **Probability Projection**: The final recurrent output is projected via a linear classification layer followed by a Sigmoid activation, yielding a continuous scalar probability value $p \in [0.0, 1.0]$ representing the likelihood of active speech in the analyzed frame. --- ## Stateful Streaming Inference & Hidden State Management Streaming inference in Silero VAD is fundamentally stateful, requiring the model's internal recurrent hidden states ($h$ and $c$ tensors, typically shape `(2, batch_size, 64)` or `(2, batch_size, 128)`) to be passed sequentially from chunk $t$ to chunk $t+1$. Hidden states must be explicitly reset to zero tensors when switching audio streams, clearing a buffer, or establishing a new connection to prevent cross-stream contextual bleeding and false activations. ``` Stream Sequence: Chunk t-1 ---> [ Model Session ] ---> Probability p_{t-1} | Hidden State (h_{t-1}, c_{t-1}) | v Chunk t ---> [ Model Session ] ---> Probability p_t | Hidden State (h_t, c_t) | v Chunk t+1 ---> [ Model Session ] ---> Probability p_{t+1} -- STREAM RESET EVENT (New Call / Stream Disconnect) -- Reset: h_0 = zeros(2, batch, hidden_dim), c_0 = zeros(2, batch, hidden_dim) ``` ### Contextual Bleeding and Hidden State Lifecycles In a non-streaming offline architecture, an entire audio file is processed as a continuous tensor, allowing the recurrent network to propagate its internal memory from start to finish. In real-time streaming, audio arrives in small frames (e.g., 31.25 ms chunks). To preserve temporal continuity, the inference host must treat the recurrent state as an explicit input and output parameter: $$\{p_t, h_t, c_t\} = \text{Forward}(x_t, h_{t-1}, c_{t-1}, \text{sr})$$ Where: - $x_t$ is the current linear PCM frame tensor. - $h_{t-1}, c_{t-1}$ are the hidden state and cell state tensors from the previous step. - $\text{sr}$ is the native sample rate scalar (typically `16000` or `8000`). - $p_t$ is the scalar speech probability for the current frame. - $h_t, c_t$ are the updated hidden state tensors to be stored and forwarded to step $t+1$. ```python import torch import numpy as np class SileroVADStreamSession: def __init__(self, model_path: str, sample_rate: int = 16000): self.sample_rate = sample_rate # Load TorchScript or ONNX session self.model = torch.jit.load(model_path) self.model.eval() self.hidden_dim = 64 # Or 128 depending on model variant self.num_layers = 2 self.batch_size = 1 self.reset_states() def reset_states(self) -> None: """Explicitly zero out recurrent tensors on stream start or teardown.""" self.h = torch.zeros(self.num_layers, self.batch_size, self.hidden_dim, dtype=torch.float32) self.c = torch.zeros(self.num_layers, self.batch_size, self.hidden_dim, dtype=torch.float32) def process_chunk(self, pcm_chunk: np.ndarray) -> float: """ Process a single 512-sample float32 chunk at 16kHz. """ tensor_chunk = torch.from_numpy(pcm_chunk).unsqueeze(0) # Shape: (1, 512) with torch.no_grad(): # Model forward consumes current chunk and previous hidden states prob, self.h, self.c = self.model(tensor_chunk, self.h, self.c, self.sample_rate) return prob.item() ``` > [!IMPORTANT] > Reusing a `SileroVADStreamSession` instance across different incoming client streams without calling `reset_states()` leads to severe **contextual bleeding**. Residual hidden state activations from a previous speaker will corrupt initial speech probability scores for the next caller, creating false speech triggers or clipped sentence onsets. --- ## Audio Preprocessing: Sample Rates, Normalization, and Framing Silero VAD strictly requires 1D single-channel linear PCM audio normalized as 32-bit floating-point values in the range $[-1.0, 1.0]$ at fixed native sampling rates of 8,000 Hz or 16,000 Hz. Inbound streaming audio from standard capture pipelines (e.g., 44.1 kHz / 48 kHz WebRTC streams or arbitrary payload sizes) requires an upstream resampler and a circular accumulation buffer (ring buffer) to package exact chunk sizes before feeding tensors into the inference session. ``` Inbound Stream (48 kHz, Arbitrary Frame Sizes: 10ms, 20ms, 128 samples) | v +----------------------------+ | Linear / Polyphase Resampler| +----------------------------+ | 16 kHz Floating-Point PCM | v +----------------------------+ | Circular Accumulation Buffer| (Accumulates until >= 512 samples) +----------------------------+ | Exact 512-Sample Chunks (31.25 ms @ 16 kHz) | v +----------------------------+ | Silero VAD Inference Engine | +----------------------------+ ``` ### Supported Frame Chunk Dimensions and Window Latency The model architecture is parameterized around specific sample counts per forward pass. Passing chunk sizes that deviate from these exact dimensions will trigger runtime tensor mismatch errors within the acoustic encoder layers. | Native Sampling Rate | Chunk Size (Samples) | Window Duration ($T_{\text{chunk}}$) | Context Suitability | | :--- | :--- | :--- | :--- | | **16,000 Hz** | **512 samples** | **31.25 ms** | Ultra-low latency conversational agents / Real-time ASR streaming | | **16,000 Hz** | **1024 samples** | **64.00 ms** | Standard voice streaming / WebRTC voice pipelines | | **16,000 Hz** | **1536 samples** | **96.00 ms** | Bandwidth-constrained or batch-optimized processing | | **8,000 Hz** | **256 samples** | **32.00 ms** | Legacy telephony / Narrowband VoIP | ### End-to-End Algorithmic Latency Decomposition The total algorithmic latency ($\tau_{\text{total}}$) observed from the moment physical sound reaches a microphone to the generation of a validated VAD decision is defined by the following system components: $$\tau_{\text{total}} = \tau_{\text{accum}} + \tau_{\text{resample}} + \tau_{\text{inference}} + \tau_{\text{debounce}}$$ Where: - $\tau_{\text{accum}}$: Chunk window accumulation time (e.g., 31.25 ms for 512 samples at 16 kHz). - $\tau_{\text{resample}}$: Filter delay of the polyphase or sinc resampler converting 48 kHz $\to$ 16 kHz. - $\tau_{\text{inference}}$: Execution time of the forward neural pass inside the ONNX Runtime / TorchScript engine. - $\tau_{\text{debounce}}$: Downstream confirmation window required by the state machine to validate speech onset (e.g., consecutive speech frames). --- ## High-Performance Pipeline Architecture: Thread Isolation & Memory Management Real-time streaming architectures must isolate model forward execution (`Ort::Session::Run` or TorchScript C++ execution) from low-latency audio capture threads (e.g., CoreAudio, ALSA, WebRTC native callbacks, or WebAudio AudioWorklets). Audio callbacks should push raw PCM frames into a lock-free Single-Producer Single-Consumer (SPSC) ring buffer, allowing a dedicated worker thread to run inference without risking audio buffer underruns. ```mermaid flowchart LR subgraph AudioCaptureThread [Audio Capture Thread (High Priority / Hard Real-Time)] Hardware[Audio Hardware Callback] --> Resample[Fast Resampler 48k->16k] Resample --> PushSPSC[Push to SPSC Ring Buffer] end subgraph LockFreeQueue [Memory Barrier / Lock-Free Boundary] PushSPSC -. Lock-Free SPSC Buffer .-> PopSPSC[Pop from SPSC Ring Buffer] end subgraph InferenceThread [Inference Worker Thread (Dedicated CPU Core)] PopSPSC --> FrameAcc[Accumulate 512 Samples] FrameAcc --> PreAllocOrt[Bind Pre-allocated Ort::Value Tensors] PreAllocOrt --> OrtRun[Ort::Session::Run] OrtRun --> StateUpdate[Propagate Hidden States h_t, c_t] StateUpdate --> StateMachine[Dual-Threshold Hysteresis] end ``` ### The SPSC Ring Buffer Boundary Audio hardware capture drivers (such as macOS CoreAudio `AURenderCallback` or Linux ALSA/JACK callbacks) execute in real-time priority contexts. Performing heavy tensor math, memory allocations, mutex locking, or I/O operations inside these callbacks will cause immediate audio buffer dropouts (underruns/xruns). To guarantee thread safety without lock contention: 1. The **Audio Capture Callback (Producer)** writes resampled float32 PCM frames into a pre-allocated, bounded SPSC ring buffer using atomic head/tail index updates. 2. The **Inference Worker (Consumer)** continuously reads available samples, assembles complete 512-sample frames, and executes the neural model. ### Eliminating Hot-Path Allocations in ONNX Runtime Deployments In production [ONNX Runtime Documentation](https://onnxruntime.ai/docs/) deployments (C++, Rust, Go, C#, WASM), allocating new input/output `Ort::Value` buffers on every ~31.25 ms chunk creates severe heap fragmentation and triggers unpredictable Garbage Collection (GC) pauses in managed runtimes. High-throughput C++ pipelines optimize this execution loop by: - Pre-allocating pinned `Ort::MemoryInfo` CPU allocations for input PCM tensors, sampling rate scalars, and recurrent state tensors during engine initialization. - Reusing persistent memory addresses via `Ort::Value::CreateTensor` and updating input pointers in-place across chunk iterations. ```cpp #include #include #include class SileroVADOnnxPipeline { public: SileroVADOnnxPipeline(const wchar_t* model_path) : env_(ORT_LOGGING_LEVEL_WARNING, "SileroVAD"), session_(env_, model_path, Ort::SessionOptions{nullptr}), memory_info_(Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault)) { // Pre-allocate persistent state tensors (2, 1, 64) h_state_.fill(0.0f); c_state_.fill(0.0f); std::array state_shape = {2, 1, 64}; std::array input_shape = {1, 512}; std::array sr_shape = {1}; // Initialize reusable Ort::Value wrappers pointing to fixed memory h_tensor_ = Ort::Value::CreateTensor( memory_info_, h_state_.data(), h_state_.size(), state_shape.data(), state_shape.size()); c_tensor_ = Ort::Value::CreateTensor( memory_info_, c_state_.data(), c_state_.size(), state_shape.data(), state_shape.size()); sr_tensor_ = Ort::Value::CreateTensor( memory_info_, &sr_value_, 1, sr_shape.data(), sr_shape.size()); } float StepInference(const float* chunk_512_ptr) { std::array input_shape = {1, 512}; // Wrap input pointer directly without heap allocation Ort::Value input_tensor = Ort::Value::CreateTensor( memory_info_, const_cast(chunk_512_ptr), 512, input_shape.data(), input_shape.size()); const char* input_names[] = {"input", "sr", "h", "c"}; const char* output_names[] = {"output", "hn", "cn"}; std::array inputs = { std::move(input_tensor), std::move(sr_tensor_), std::move(h_tensor_), std::move(c_tensor_) }; // Run forward pass auto outputs = session_.Run( Ort::RunOptions{nullptr}, input_names, inputs.data(), inputs.size(), output_names, 3); float prob = outputs[0].GetTensorMutableData()[0]; // Recycle output states back into persistent tensors for chunk t+1 h_tensor_ = std::move(outputs[1]); c_tensor_ = std::move(outputs[2]); // Rebind scalar sample rate tensor std::array sr_shape = {1}; sr_tensor_ = Ort::Value::CreateTensor( memory_info_, &sr_value_, 1, sr_shape.data(), sr_shape.size()); return prob; } private: Ort::Env env_; Ort::Session session_; Ort::MemoryInfo memory_info_; int64_t sr_value_ = 16000; std::array h_state_; std::array c_state_; Ort::Value h_tensor_{nullptr}; Ort::Value c_tensor_{nullptr}; Ort::Value sr_tensor_{nullptr}; }; ``` --- ## Downstream Post-Processing: Dual-Threshold Hysteresis and Speech Padding The raw model output is a continuous scalar probability $p \in [0.0, 1.0]$. Production speech detection requires a downstream state machine with dual-threshold hysteresis (e.g., onset threshold $\approx 0.5$, offset threshold $\approx 0.35$) and configurable speech padding (`speech_pad_ms`, `min_speech_duration_ms`, `min_silence_duration_ms`) to prevent clipping unvoiced consonants, plosives, and natural inter-word breathing pauses. ``` Probability (p) 1.0 | .-------------------. | / \ 0.5 |--- Onset Threshold - - - - - - - - - - - \ - - - - - - - - - - - - - - - | / \ 0.35 |--- Offset Threshold - - - - - - - - - - - -\ - - - - - - - - - - - - - - | / \ 0.0 +--------------'-------------------------------'-------------------------> Time | | VAD State: [SILENCE] [SPEECH] [SILENCE] | | ^ +---> Debounce Confirmation +---> Speech Hangover -----+ (min_speech_duration) (min_silence_duration + pad) ``` ### The Flapping Problem in Single-Threshold Systems If a VAD pipeline evaluates raw probabilities using a single static threshold (e.g., $\text{active} = p > 0.5$), the system experiences severe **chatter/flapping** during conversational transitions. Because human speech contains natural amplitude decay, unvoiced plosives (e.g., /p/, /t/, /k/), and fricatives (e.g., /s/, /f/), speech probability values oscillate around $0.5$. This creates rapid, artificial segment fragmentation. ### State Machine Architecture & Parameters A robust downstream state machine prevents fragmentation by implementing the following state transitions: ```mermaid stateDiagram-v2 [*] --> SilenceState : Initialize / Reset SilenceState --> TriggerCandidate : p >= onset_threshold (0.5) TriggerCandidate --> SilenceState : p < onset_threshold (Spurious noise spike) TriggerCandidate --> ActiveSpeechState : Sustained speech >= min_speech_duration_ms ActiveSpeechState --> ActiveSpeechState : p >= offset_threshold (0.35) ActiveSpeechState --> HangoverState : p < offset_threshold (0.35) HangoverState --> ActiveSpeechState : p >= offset_threshold (Inter-word pause ends) HangoverState --> SilenceState : Silence duration >= min_silence_duration_ms ``` Key state machine tuning parameters include: - **`onset_threshold` ($\approx 0.5$)**: The probability required to enter the speech candidate state. - **`offset_threshold` ($\approx 0.35$)**: The reduced probability barrier required to remain in speech. Because speech has already commenced, context warrants a lower threshold to avoid clipping word endings. - **`min_speech_duration_ms` (e.g., 64–100 ms)**: Rejects brief acoustic impulses (coughs, key taps, clicks) that briefly exceed the onset threshold. - **`min_silence_duration_ms` (e.g., 200–400 ms)**: The hangover window that prevents splitting a single sentence into multiple disconnected segments during intra-sentence pauses. - **`speech_pad_ms` (e.g., 30–60 ms)**: Prepends pre-trigger audio buffers and appends post-trigger audio buffers to ensure unvoiced phonemes leading into or trailing out of vocal cord vibration are captured completely. --- ## Silero VAD vs. Traditional GMM-Based Detectors (WebRTC VAD) Compared to traditional heuristic/GMM-based detectors (such as the [WebRTC Native Audio Processing Module](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_processing/vad/)), Silero VAD provides significantly superior noise rejection and boundary accuracy in non-stationary noise, at the architectural trade-off of requiring a neural runtime engine (ONNX Runtime, TorchScript, or WebAssembly) and higher FLOP consumption per frame. ``` +-------------------------------------------------------------------------------+ | ARCHITECTURAL COMPARISON MATRIX | +--------------------------+------------------------+---------------------------+ | Dimension | WebRTC VAD (GMM) | Silero VAD (Neural/RNN) | +--------------------------+------------------------+---------------------------+ | Core Classification Model| Low-Order GMMs & | Acoustic Feature Encoder | | | Sub-band Energy | + Recurrent Neural Network| +--------------------------+------------------------+---------------------------+ | Non-Stationary Noise | Prone to false triggers| High rejection of babble, | | Robustness | on background babble | keystrokes, and ambient AC| +--------------------------+------------------------+---------------------------+ | Computational Complexity | Extremely Low | Moderate (Higher FLOP | | | (Bitwise/Integer math) | consumption per frame) | +--------------------------+------------------------+---------------------------+ | Deployment Footprint | Pure C (No runtime engine| Requires ONNX Runtime, | | | dependencies) | TorchScript, or WASM) | +--------------------------+------------------------+---------------------------+ | Temporal Context | Frame-local spectral | Multi-frame stateful | | | statistics | recurrent memory (h, c) | +--------------------------+------------------------+---------------------------+ ``` ### Algorithmic Foundations - **WebRTC VAD**: Operates by splitting the incoming signal into sub-bands using filter banks and computing log-energies. A low-order Gaussian Mixture Model estimates the probability density functions of speech and background noise. Because it relies heavily on spectral energy profiles, WebRTC VAD is computationally lighter than neural models, but it struggles to distinguish between non-stationary background noise and true speech. - **Silero VAD**: Employs a learned convolutional encoder and recurrent cells to extract complex spatial and temporal phoneme representations. While it requires a dedicated runtime engine (e.g., ONNX Runtime or TorchScript) and uses more computational FLOPs per frame than simple energy filters, it provides superior boundary precision and noise discrimination across varied acoustic environments. --- ## Frequently Asked Questions ### How do you prevent audio capture thread underruns when running Silero VAD? Audio capture thread underruns are prevented by strictly isolating the high-priority audio callback from the neural inference execution path. Audio capture drivers (such as CoreAudio or ALSA) must only push raw PCM samples into a lock-free Single-Producer Single-Consumer (SPSC) ring buffer without acquiring mutexes or allocating heap memory. A dedicated background worker thread reads from this ring buffer, packages complete 512-sample chunks, and invokes the model forward pass. ### Why does Silero VAD require hidden state tracking across streaming chunks? Because Silero VAD utilizes a recurrent neural network (RNN / LSTM / GRU cells) to model speech context over time, the hidden state tensors ($h$ and $c$ of shape `(2, batch_size, 64)` or `(2, batch_size, 128)`) encapsulate the historical acoustic context from preceding frames. In streaming mode, passing these states sequentially from chunk $t$ to chunk $t+1$ allows the model to differentiate sustained speech from transient acoustic spikes. When switching streams or handling a new audio session, these states must be explicitly reset to zero tensors to prevent contextual bleeding between callers. ### What is dual-threshold hysteresis and why is it necessary for speech detection? Dual-threshold hysteresis utilizes separate probability boundaries for speech onset (e.g., $p \ge 0.5$) and speech offset (e.g., $p < 0.35$). Because speech signals naturally decay during trailing vowels and fluctuate during unvoiced consonants and intra-sentence pauses, a single static threshold causes rapid state flapping (chatter). Dual thresholds, combined with speech padding and hangover windows (`min_silence_duration_ms`), ensure that natural breathing pauses and quiet consonant endings are not clipped. --- ### About the Author **Furkan Çetinkaya** is a Mobile-focused Software Developer specializing in React Native, native bridge integrations (Kotlin & Swift), and supporting backend services. Experienced in maintaining high-impact mobile applications and developer SDKs. - [GitHub](https://github.com/cetfu) - [LinkedIn](https://www.linkedin.com/in/cetfu)