Speech-to-Text (STT) Module - Comprehensive Testing Documentation¶

Overview¶

The STT module has been enhanced with comprehensive unit, integration, and performance tests covering all transcription implementations including MLX, ONNX, buffered transcription, WebSocket streaming, and external provider support.

Test Coverage Summary¶

1. Unit Tests¶

test_parakeet_mlx.py¶

Coverage: Parakeet MLX implementation
Key Tests:
Model loading and caching
Simple transcription
Chunked transcription
Audio preprocessing (resampling, stereo to mono)
Error handling
Progress callbacks
File path and numpy array inputs

test_parakeet_onnx.py¶

Coverage: Parakeet ONNX implementation
Key Tests:
ONNX session creation
Tokenizer functionality
Mel-spectrogram preprocessing
Chunked transcription with merge algorithms
Device selection (CPU/CUDA)
Error handling

test_buffered_transcription.py¶

Coverage: Advanced buffered/chunked transcription
Key Tests:
Chunk creation logic
Middle merge algorithm
LCS (Longest Common Subsequence) merge
Audio resampling
Progress callbacks
Memory efficiency

test_streaming_transcription.py¶

Coverage: WebSocket-based streaming
Key Tests:
AudioBuffer functionality
Voice activity detection
Buffer accumulation
WebSocket connection handling
Error handling during streaming
Concurrent stream support

test_external_provider.py¶

Coverage: External OpenAI-compatible API support
Key Tests:
Configuration validation
Provider management (add/list/remove)
Retry logic on rate limiting
Timeout handling
Different response formats (json, text, srt, vtt)
Authentication handling

2. Integration Tests¶

test_parakeet_mlx.py - Integration Section¶

Integration with Nemo module
Integration with main transcription library
Cross-module functionality

test_buffered_transcription.py - Integration Section¶

Integration with MLX backend
Integration with ONNX backend
Different merge algorithm comparisons

test_streaming_transcription.py - Integration Section¶

Full streaming session simulation
Concurrent streams handling
WebSocket protocol compliance

3. Performance Benchmarks¶

test_transcription_benchmarks.py¶

Coverage: All implementations performance testing
Key Benchmarks:
MLX scaling with audio duration
Chunking vs non-chunking performance
ONNX inference speed
Merge algorithm efficiency
Memory usage profiling
Real-world scenarios (podcast, meeting, batch processing)

Running the Tests¶

Run All Tests¶

cd tldw_server
python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Run Specific Test Categories¶

Unit Tests Only¶

python -m pytest -m "unit" tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Integration Tests Only¶

python -m pytest -m "integration" tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Performance Tests Only¶

python -m pytest -m "performance" tldw_Server_API/tests/Media_Ingestion_Modification/ -v

Run Tests for Specific Implementation¶

MLX Tests¶

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_parakeet_mlx.py -v

ONNX Tests¶

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_parakeet_onnx.py -v

Buffered Transcription Tests¶

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_buffered_transcription.py -v

Streaming Tests¶

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_streaming_transcription.py -v

External Provider Tests¶

python -m pytest tldw_Server_API/tests/Media_Ingestion_Modification/test_external_provider.py -v

Run with Coverage Report¶

python -m pytest --cov=tldw_Server_API.app.core.Ingestion_Media_Processing.Audio \
    --cov-report=html \
    --cov-report=term \
    tldw_Server_API/tests/Media_Ingestion_Modification/

Test Markers¶

Tests are organized with markers for easy filtering:

@pytest.mark.unit - Unit tests (fast, isolated)
@pytest.mark.integration - Integration tests (cross-module)
@pytest.mark.performance - Performance benchmarks
@pytest.mark.external_api - Tests requiring external APIs
@pytest.mark.slow - Slow tests (>5 seconds)
@pytest.mark.asyncio - Async tests

Mock Strategy¶

All tests use comprehensive mocking to ensure: - Fast execution - No external dependencies - Predictable results - Resource isolation

Key mocked components: - ML models (MLX, ONNX) - HTTP clients for external APIs - WebSocket connections - File I/O operations - Audio processing libraries

Performance Baseline¶

Expected performance metrics from benchmarks:

Implementation	Speed (vs real-time)	Memory Usage	Status
MLX (chunked)	100x+	<2GB	Production Ready
MLX (no chunks)	60x+	~2GB	Production Ready
ONNX	20-30x	~1.5GB	Requires Setup
Buffered (LCS)	70x	~2GB	Production Ready
External API	Varies	Minimal	Provider Dependent

External Provider Configuration¶

Setting Up External Providers¶

External providers allow forwarding transcription requests to any OpenAI-compatible Audio API.

Via Configuration File¶

Edit config.txt:

[external_providers.myapi]
base_url = https://api.example.com/v1/audio/transcriptions
api_key = your-api-key
model = whisper-1
timeout = 300
max_retries = 3

Via Environment Variables¶

export EXTERNAL_TRANSCRIPTION_MYAPI_BASE_URL=https://api.example.com
export EXTERNAL_TRANSCRIPTION_MYAPI_API_KEY=your-api-key
export EXTERNAL_TRANSCRIPTION_MYAPI_MODEL=whisper-1

Programmatically¶

from Audio_Transcription_External_Provider import (
    ExternalProviderConfig,
    add_external_provider
)

config = ExternalProviderConfig(
    base_url="https://api.example.com/v1/audio/transcriptions",
    api_key="your-api-key",
    model="whisper-1"
)

add_external_provider("myapi", config)

Using External Providers¶

# Use default external provider
result = transcribe_audio(
    audio_data,
    transcription_provider="external"
)

# Use specific external provider
result = transcribe_audio(
    audio_data,
    transcription_provider="external:myapi"
)

Test Data Requirements¶

Audio Generation¶

Tests generate synthetic audio data: - Sample rate: 16kHz (standard for speech) - Duration: 1-300 seconds based on test - Format: Float32 numpy arrays - Content: Sine waves with speech-like modulation

Real Audio Testing¶

For production validation, use actual audio files:

python test_all_transcriptions.py

This uses the sample.mp4 file extracted to WAV format.

Continuous Integration¶

GitHub Actions Configuration¶

name: STT Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: |
          pip install -e .[dev]
      - name: Run unit tests
        run: |
          pytest -m "unit" tests/Media_Ingestion_Modification/
      - name: Run integration tests
        run: |
          pytest -m "integration" tests/Media_Ingestion_Modification/

Troubleshooting¶

Common Test Failures¶

Import Errors
Ensure all dependencies are installed
Check PYTHONPATH includes project root
Async Test Failures
Requires pytest-asyncio
May need event loop configuration
Performance Test Variations
Results vary by hardware
Use relative comparisons, not absolute times
Mock Failures
Verify mock paths match actual module structure
Check mock return values match expected types

Future Testing Improvements¶

Property-based Testing
Use hypothesis for fuzzing
Test edge cases automatically
Load Testing
Concurrent transcription stress tests
Memory leak detection
End-to-End Tests
Full API endpoint testing
Real audio file processing
Mutation Testing
Verify test effectiveness
Identify untested code paths

Conclusion¶

The STT module now has comprehensive test coverage ensuring: - Reliability across all implementations - Performance validation - Easy regression detection - Safe refactoring capability - External provider extensibility

All critical functionality is tested, mocked appropriately, and benchmarked for performance.