Visual Speech Recognition

Problem statement

Visual speech recognition requires strict preprocessing, model serving and evaluation discipline. Lip movement must be converted into model-ready tensors without mixing UI concerns into inference code.

Engineering objective

1. Defined the preprocessing contract for frame rate, face alignment and mouth crops.
2. Separated web interface, FastAPI service and model adapters.
3. Scoped evaluation on both reference crops and full pipeline clips.
4. Designed live streaming segmentation around lip aperture rather than raw pixel motion.

Architecture and approach

A full visual speech system that decodes video frames, detects and tracks a face, aligns and crops the mouth region, runs an Auto-AVSR visual speech model, and returns transcripts with confidence and alternatives through upload and WebSocket APIs.

• MediaPipe-based face detection, tracking and mouth measurement.
• Auto-AVSR Conformer encoder with joint CTC and attention decoding.
• FastAPI upload and live WebSocket inference services.
• Next.js interface for camera, upload, transcript and debug views.

How it works

Python 3.10+, FastAPI, PyAV, MediaPipe, PyTorch.

Auto-AVSR adapter with pluggable model registration.

Next.js web app, pytest suite for preprocessing and API behaviour.

Evaluation scripts for word error rate on LRS3 fixtures.

Controls and constraints

• Uploads deleted after inference unless persistence is explicitly enabled.
• Debug exports opt-in per request with timed cleanup.
• Live camera frames held only in a short rolling buffer.

What the system includes

• Preprocessing pipeline from video to mouth tensors.
• ML API with health, transcribe, preprocess and stream endpoints.
• Web interface with debug visualisation.
• Evaluation and fixture generation scripts.

Product and system views

An end-to-end visual speech pipeline converting lip movement in video into text using modern visual speech recognition.

Evaluation and results

• Demonstrates separation of UI, service and model layers in a research-grade pipeline.
• Provides measured evaluation context rather than marketing accuracy claims.

Validation and testing

• Reference mouth crops to model: 5.7% word error rate on LRS3 test utterances used in repository evaluation.
• Full pipeline on composite clips: 25.7% word error rate in repository evaluation.
• 158 pytest tests covering preprocessing, API and queue behaviour.

Limitations

• Transcripts are model predictions, not recordings of speech.
• Measured pipeline word error rate on small composite fixtures is higher than reference-crop evaluation.
• Fast head movement can open utterances that are later abandoned without transcription.