Live Camera Virtual Try-On on a Consumer GPU
Dress one still with a diffusion try-on model, then puppet-warp it onto every live camera frame at 34 ms per frame.
Research found no open model does live garment-faithful try-on, so the design splits reference generation from real-time warping; own agnostic mask, sleeve detection, majority voting over seeds and a hallucination score; verified end to end on test clips.
- 34 ms/frame, ~20 fps on a 16 GB GPU
- Own agnostic mask and sleeve detector
- Seed majority vote and hallucination scoring
- Offline simulator for artefact diagnosis
The goal was a browser or phone camera feed where the viewer appears wearing a chosen garment, running on a single 16 GB consumer GPU. A research pass established that no open-weights model handles garment-faithful try-on on a live stream, so the system generates a reference with CatVTON on one frame and puppet-warps that reference onto every subsequent frame using body landmarks, refreshing the reference every couple of seconds.
Making it hold up took specific engineering: an agnostic mask built from a segmentation model plus landmark capsules without DensePose, a sleeve detector using shoulder-width ratios and cuff gaps, a majority vote across up to three seeds for the first reference because the model invents shrugs or sleeves on roughly half of some frames, a hallucination score, refinement of later references from the nearest previous one for consistency, and rigid arm pieces with a torso spline.
Measured on test clips: 34 ms per frame server-side, about 20 fps display, first dress in four to eight seconds, refresh in two, 3.7 GB of VRAM. Dresses and sleeved tops work. An offline simulator dumps crop, init, mask and result per reference so artefacts are diagnosed rather than guessed. The reference model is non-commercial; an Apache-licensed backend is identified as the next step for commercial use.
How it works
- Camera frames stream to the server over HTTPS
- Landmarks and segmentation build the agnostic mask
- Try-on model dresses a reference frame (voted over seeds)
- Each live frame is puppet-warped from the nearest reference
- Reference refreshes periodically for consistency