How Does Live Virtual Try-On With a Webcam Work? (Built on One GPU)

Live virtual try-on with a webcam works by splitting the problem in two. A garment try-on model, which is slow but faithful to the clothing, dresses a single still frame of the person to produce a reference. A fast geometric step then takes that dressed reference and warps it onto every subsequent camera frame using body landmarks, so the garment follows the person's shoulders, arms and torso in real time. Every couple of seconds a new reference is generated to absorb pose and lighting changes. The viewer sees themselves wearing the garment at roughly 20 frames per second on a single 16 GB consumer GPU.

That is the architecture I arrived at after a research pass showed that no open-weights model handles garment-faithful try-on on a live stream directly. The rest of this post explains why the split is necessary, what breaks when you try it naively, and the specific engineering that made it hold up on test clips at 34 milliseconds per frame.

Why can't a try-on model just run on every frame?

Diffusion-based try-on models produce a realistic dressed image in roughly a second or more per frame on consumer hardware. Real-time video needs a frame every 50 milliseconds. Even if you could afford the compute, consecutive frames would be generated independently and the garment would flicker, change texture and occasionally change shape from one frame to the next. Temporal consistency is not a property these models have.

The puppet-warp approach sidesteps both problems. The expensive generation happens once per reference; the per-frame work is a warp, which is cheap and deterministic. Because the same reference is reused across many frames, the garment is stable by construction. Flicker only appears at reference changes, and that is managed by refining each new reference from the previous one.

How does the system know where the body is?

A pose model gives body landmarks (shoulders, elbows, wrists, hips) per frame, and a segmentation model separates the person from the background and labels regions such as upper clothes and arms. From these the pipeline builds what try-on research calls the agnostic mask: the area of the person that the new garment is allowed to replace. Most research pipelines build this with a dense body-surface model that is heavy to run; mine builds it from the segmentation plus capsule shapes around the landmark skeleton, which runs fast and is good enough for tops and dresses.

A separate sleeve detector decides whether the person's current clothing has sleeves, using shoulder-width ratios and the gap at the cuff. This matters because the mask must include existing sleeves when the target garment is sleeveless, and must not erase bare arms when it is not.

What does the try-on model get wrong, and how do you catch it?

On roughly half of some frames, the try-on model invents things: a shrug over a sleeveless dress, sleeves that were not in the garment image, a different neckline. A single generation cannot be trusted as the reference for the next two seconds of video. Two mechanisms handle this:

• Seed majority vote: the first reference is generated with up to three different random seeds, and the outputs are compared. The version that agrees with the majority on sleeve presence and silhouette becomes the reference.
• Hallucination score: each generated reference is scored against the garment image and the mask for signs of invented structure. A high score triggers a regeneration rather than being shown.

Later references are refined from the nearest previous one instead of being generated from scratch, which keeps the garment's appearance consistent across refreshes and makes the hallucination rate drop after the first dress.

How is the warp done so arms and torso move naturally?

The dressed reference is split into pieces that move differently. The torso is handled by a smooth spline deformation anchored to shoulders and hips, so it bends and leans without tearing. Each arm is treated as rigid pieces (upper arm, forearm) that rotate about the elbow and shoulder landmarks, because arm fabric mostly rotates rather than stretches. Combining a soft torso with rigid arm pieces gave the most convincing motion for sleeved tops; a single global warp made arms look like rubber.

Per frame, the pipeline computes the landmark deltas between the reference frame and the live frame, applies the piecewise warp, composites over the live background using the segmentation, and sends the result back to the browser.

What performance is realistic on consumer hardware?

Measured on test clips on a 16 GB card:

• 34 milliseconds per frame server side, which gives about 20 frames per second displayed.
• First dress appears four to eight seconds after the person steps into frame (that is the voted reference generation).
• Reference refresh every two seconds.
• About 3.7 GB of GPU memory in steady state, leaving room for the browser and other work.

Dresses and sleeved tops work. Trousers, layered outfits and garments with strong patterns that must align across the body are the open problems; patterns in particular show the warp seams.

Can this be used commercially?

The try-on model I used for the reference generation is released under a non-commercial licence, so the system as built is a research and demonstration piece. I identified an Apache-licensed backend as the next step for a commercial deployment; the architecture does not change, only the model that dresses the reference frame. The camera path (browser or phone to server over a secure connection) and the warp engine are my own code and carry no such restriction.

How I built this: live camera virtual try-on on a consumer GPU

The Live Camera Virtual Try-On project is the implementation of everything above. Its workflow is:

1. Camera frames stream from the browser or phone to a Python server over a secure connection.
2. Per frame, a pose model and a segmentation model produce landmarks and a person mask; the agnostic mask and sleeve detection are built from them.
3. For the first reference, the try-on model dresses one frame across up to three seeds; the majority vote and hallucination score choose the winner.
4. Every live frame is puppet-warped from the nearest reference: torso spline plus rigid arm pieces, composited over the live background.
5. A new reference is refined from the previous one every two seconds to track pose and lighting.
6. An offline simulator replays recorded clips and dumps the crop, initial frame, mask and result for every reference, so artefacts are diagnosed from evidence rather than guessed at.

Design decisions: generate rarely and warp often, build the mask without a heavy dense-pose model, vote across seeds because the model hallucinates, and keep an offline simulator so every visible artefact has a reproducible cause. The outcome is a working end-to-end system at 34 milliseconds per frame on a single consumer GPU, with the commercial-licence path mapped. Like my talking head pipeline, it runs entirely locally and keeps the camera feed on the user's own hardware.

FAQ

Does the camera feed leave the user's device?

Frames are processed by a server, which in my setup is the same machine as the GPU on a local network. For a hosted product the frames would travel to the server over an encrypted connection and should not be stored. The architecture does not require any cloud model.

Why does the first dress take several seconds?

Because the first reference is generated up to three times and voted on. After that, references are refined from the previous one, which is faster and more consistent. The wait is a deliberate trade for a reference that does not show invented sleeves.

Can it handle more than one person in frame?

The current pipeline tracks one person. Two people would need separate masks, landmarks and references, which roughly doubles compute and halves the frame rate on the same card.

What would it take to add trousers or full outfits?

A lower-body agnostic mask, leg landmarks in the warp model, and a try-on model trained for full-body garments. The pipeline structure stays the same; the mask builder and the warp pieces grow.

If you want a camera-based try-on or similar real-time vision system prototyped, tell me about it on the contact page.