Summary
The VoiceForge roadmap explicitly lists "Replace the placeholder models/wav2lip.onnx with a real lightweight browser Wav2Lip ONNX model" and "Implement real ONNX Runtime Web Wav2Lip inference" as TODO items. Currently, the lip-sync output relies on a fallback mouth animation rather than actual model-driven movement, which significantly reduces the credibility and utility of the platform for its target users — deaf and speech-impaired individuals on video calls.
Problem
- The
models/wav2lip.onnx file is a placeholder. No real inference is being performed.
- The lip-sync output is a fallback animation, meaning the visual channel does not actually synchronize with the generated speech audio.
- For deaf and speech-impaired users, the lip-sync accuracy is core to the product value — without it, the video output is unconvincing to call participants.
- The ONNX Runtime Web SDK is already listed in the tech stack, indicating this was always planned.
Impact
- The primary differentiator of VoiceForge (lip-synced facial video) does not actually work in the current implementation.
- Users demo the product and find the mouth animation disconnected from the audio, undermining trust.
Proposed Solution
I would like to implement the following:
-
Source a suitable lightweight Wav2Lip-compatible ONNX model — e.g., a quantized/pruned version of Wav2Lip that runs feasibly in a browser context with ONNX Runtime Web (WebAssembly backend). Recommended: the wav2lip_gan_quantized variant or a community-exported browser-optimized model from HuggingFace.
-
Replace the placeholder in models/ with the real .onnx file or a fetch-on-demand pattern (to avoid committing large binary weights to Git — instead fetch from a CDN or remote URL at runtime).
-
Implement inference pipeline in the client:
- Load the ONNX model using
ort.InferenceSession.create()
- Accept mel-spectrogram frames from the TTS audio output
- Accept face frames from the webcam canvas
- Run inference and render output frames to the canvas
-
Replace the fallback mouth animation with the model output frames.
-
Add a loading indicator during model initialization (models can be several MB).
I can start with a minimal proof-of-concept using a small quantized model and iterate from there. Could you please assign this issue to me?
Labels: enhancement, AI/ML, core feature, GSSoC 2026
Summary
The VoiceForge roadmap explicitly lists "Replace the placeholder
models/wav2lip.onnxwith a real lightweight browser Wav2Lip ONNX model" and "Implement real ONNX Runtime Web Wav2Lip inference" as TODO items. Currently, the lip-sync output relies on a fallback mouth animation rather than actual model-driven movement, which significantly reduces the credibility and utility of the platform for its target users — deaf and speech-impaired individuals on video calls.Problem
models/wav2lip.onnxfile is a placeholder. No real inference is being performed.Impact
Proposed Solution
I would like to implement the following:
Source a suitable lightweight Wav2Lip-compatible ONNX model — e.g., a quantized/pruned version of Wav2Lip that runs feasibly in a browser context with ONNX Runtime Web (WebAssembly backend). Recommended: the
wav2lip_gan_quantizedvariant or a community-exported browser-optimized model from HuggingFace.Replace the placeholder in
models/with the real.onnxfile or a fetch-on-demand pattern (to avoid committing large binary weights to Git — instead fetch from a CDN or remote URL at runtime).Implement inference pipeline in the client:
ort.InferenceSession.create()Replace the fallback mouth animation with the model output frames.
Add a loading indicator during model initialization (models can be several MB).
I can start with a minimal proof-of-concept using a small quantized model and iterate from there. Could you please assign this issue to me?
Labels:
enhancement,AI/ML,core feature,GSSoC 2026