SYSTEM CASE STUDY
AudioAware AI: Audio Authenticity Screening
Probabilistic detection of AI-generated speech and environmental audio
Introduction
AudioAware AI is a local web app that detects AI-generated speech and environmental audio using WavLM and AST models. It routes audio clips to the appropriate model based on speech content, providing probabilistic results and a spectrogram view. The app is built with React and FastAPI, using PyTorch and Hugging Face Transformers for model inference. The project includes detailed documentation and model metrics, but has limitations in its use cases and accuracy for certain audio types.
Problem
The app addresses the risk of misidentifying AI-generated speech and environmental audio, which can lead to incorrect decisions in critical scenarios. It aims to provide a probabilistic screening tool that can help users assess the authenticity of audio clips, but it does not offer definitive proof of authenticity.
Solution
The app routes audio clips to the appropriate model branch based on speech content, using WebRTC VAD to measure speech ratios. It provides probabilistic results and a spectrogram view, with display labels based on confidence thresholds. The solution includes a detailed documentation and model metrics, but has limitations in its use cases and accuracy for certain audio types.
Architecture
Frontend: React 19 with TypeScript, Vite, Lucide React, Tailwind/PostCSS Backend: FastAPI with Uvicorn, PyTorch, Hugging Face Transformers, librosa, matplotlib, WebRTC VAD Model artifacts: WavLM for speech, AST for environmental audio Audio processing: Normalization, speech ratio routing, spectrogram rendering
Key features
- Upload WAV, MP3, FLAC, OGG, or M4A audio
- Route speech-heavy clips to WavLM and environmental clips to AST
- Return real/fake probabilities and a human-friendly display label
- Show the selected branch, model version, router explanation, and model limits
- Render a spectrogram/frequency-time view for the uploaded clip
- Run fully locally once the model artifacts are available
- Provide detailed documentation and model metrics
- Include a user-friendly interface with a spectrogram view
Engineering challenges
- Model accuracy varies based on audio type and source, with limitations for non-English speech and environmental audio
- Model artifacts must be manually placed in the backend directory, requiring user intervention
- The app is designed for local use, which may limit accessibility and collaboration
- The app does not provide definitive proof of authenticity, which may be insufficient for critical decisions
- The app requires specific hardware and software prerequisites, which may be a barrier to entry
Technical highlights
- The model metrics are based on specific test sets and may not represent real-world performance
- The app's accuracy for non-English speech and environmental audio is not fully documented
- The app's use cases and limitations are not fully documented
- The app's performance on GPU vs. CPU is not fully documented
Impact summary
The app provides a probabilistic screening tool for audio authenticity, which can help users assess the authenticity of audio clips. However, it does not offer definitive proof of authenticity and has limitations in its use cases and accuracy for certain audio types.
SYSTEM COMPONENTS