Video editing direction · 06 Nov to 12 Nov

Levon Khachatryan · follow-up to the 05.11 deck

H3 video editing: face swap and style transfer

The two use cases we focused on this week: swapping the people in trending videos, and turning a video into another style.

1 · Face swap on trending videos

3 viral clips, the whole 15 s in one pass; H3 arms and the new Motion Transfer LoRA

2 · Video style transfer

114 cases (real, animated and art styles), 9 methods

Where we are

We focus on two video-editing use cases

Based on the previous week's investigation (h3-video-editing.pages.dev) and the discussion in our meeting on 5 November, we decided to focus on two use cases for video editing for now:

Use case 1

Face swap for trending videos

Replace the people in a viral clip with new people from a picture.

Use case 2

Video style transfer

Turn a video into another look (anime, clay, sketch, live action and more) while keeping its people, motion and timing.

Part 1

Face swap on trending videos

Three viral clips, the first 15 s of each, new people from GPT Image 2 pictures, every method in one pass.

Setup

Three trending videos, five H3-based methods

Dance trend

3 women in a row become 3 grandmothers; 3 pictures, vertical clip.

youtube.com/shorts/-Q6Ul5UVsxk

Love of the Dead

A movie-style couple, each in 2 looks (house and field); 4 pictures.

youtube.com/watch?v=u1kD1kkraTs

Migos lobby challenge

2 performers become a new couple; 2 pictures.

youtube.com/watch?v=Minu7oTBJJc

H3 Ref2VA

base H3, FastH3, Pika H3 Flex: source as <Video 1> plus the pictures, who-is-who in a shot-by-shot prompt.

New: Motion Transfer LoRA

Community LoRA for H3 in two versions, TARA and Guide Latents; commercial use allowed (credit the author above $1M revenue).

All at 768p, 15 s (362 frames), one pass with all pictures. We also tried two community character-swap and head-swap LoRAs: they take one person per run, so multi-person clips need chained passes; that did not work well, so we leave them out for now.

Swap · dance trend

Three women, three grandmothers

What to see

Both Motion Transfer versions swap each woman at her moment and follow the dance. base H3 and FastH3 show the grandmothers, but not in the source’s order and timing; Pika keeps the originals.

Swap · dance trend

Motion Transfer TARA

Condition pictures | result, one pass with all three pictures.

What to see

The swap holds for the whole 15 s, each grandmother replaces the right woman, and the parking lot and camera of the source stay.

Swap · dance trend

Motion Transfer Guide Latents

Same pictures, the other Motion Transfer version, one pass.

What to see

The same three swaps and the same dance; the setting is redrawn (a petrol station instead of the open parking lot). Also usable.

Swap · dance trend · 2nd example

Three grandfathers: TARA

New GPT Image 2 pictures, same clip and roles, one pass.

What to see

Each grandfather appears in his own shot and keeps his look: the bearded dancer in the tracksuit, the man with glasses taking the selfie, the man in the flat cap. No women remain.

Swap · dance trend · 2nd example

Three grandfathers: Guide Latents

Same pictures, the other Motion Transfer version.

What to see

The same three swaps with identities kept; as before, the setting is redrawn as a petrol station.

Swap · movie scene

Love of the Dead

A couple, each in two looks: the man in the house and in the field, the woman as a zombie and in the field.

What to see

The hardest clip: many short shots and two looks per person. Motion Transfer follows the source shots; base H3 and FastH3 invent their own shots.

Swap · movie scene

Motion Transfer Guide Latents

Result left, the four pictures right; one pass.

What to see

Both new people appear in their shots and the cuts and poses follow the source.

Swap · movie scene

Motion Transfer TARA

Same pictures, the other Motion Transfer version, one pass.

What to see

TARA keeps the source rooms and light closely; check per shot whether each new person replaces the original actor.

Swap · movie scene · 2nd example

A Black couple: Guide Latents

Distinctive faces, so identity is easy to judge; each person in two looks.

What to see

Correct in every shot: the man keeps his face, twists and beard; the woman keeps her afro, also in the zombie look.

Swap · movie scene · 2nd example

A Black couple: TARA

Same pictures, the other Motion Transfer version.

What to see

Identities hold as with Guide Latents, except one short shot: in the second zombie close-up (about 9 s) the original actress leaks through.

Swap · limitation

A dog in a human role does not work

A Chinese man and a husky (normal and zombie look) in place of the couple; TARA shown.

What to see

The man is right in every shot. The dog appears only in a few shots; elsewhere the original woman stays or a second man appears. Guide Latents is worse. A dog works when the motion is person-like (the upright dog on Migos), not in human poses and actions.

Swap · music challenge

Migos lobby challenge

What to see

All five methods swap both people, but the two Motion Transfer versions, TARA and Guide Latents, follow the source best; base H3 loses the timing in places. From here on we show only these two.

Swap · music challenge

Motion Transfer TARA

Result left, the two pictures right; one pass.

What to see

The new couple performs every move of the source, in sync with the song, in the same room as the source.

Swap · music challenge

Motion Transfer Guide Latents

Same pictures, the other Motion Transfer version, one pass.

What to see

The same swap and the same moves in sync with the song; the set is redrawn with a red curtain instead of the source room. Also usable.

Swap · music challenge

TARA on 3 more couples

Same 15 s Migos clip, TARA, one pass; the two pictures in each corner.

What to see

Every couple performs the source’s moves for the whole clip, even a man and a dog standing upright. The result holds beyond one lucky pair.

Swap · music challenge

Guide Latents on the same 3 couples

Same clip and pictures, the other Motion Transfer version, one pass.

What to see

The swaps and the motion are as good as with TARA; the difference is the set, which Guide Latents replaces with a curtain. Both versions are usable.

Face swap · what we see

Motion Transfer is the first method that swaps from start to end

Motion Transfer

Both versions swap the new people in for the whole 15 s and follow the source motion on the dance and Migos, plus a second dance and Love cast and 3 more Migos couples. Limitation: a dog can replace a person only when the motion is person-like. Both versions are usable: TARA keeps the source set, Guide Latents often replaces the background.

base H3 · FastH3 · Pika

Swap well on a simple two-person shot (Migos), but on the dance and Love they lose the source’s order, timing or shots.

Single-person LoRAs

The community character-swap and head-swap LoRAs need one pass per person; chained passes did not work, so we leave them out for now.

Three clips, one sample each: a first look, not a ranking.

Part 2