# US 12,014,753 B1 · Actor-replacement system for videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,014,753 B1
- Document ID: US12014753B1
- Status: Granted
- Kind: B1
- Application number: 18/349,551
- Filed: 2023-07-10
- Published: 2024-06-06
- Granted: 2024-06-18
- Earliest claimed: 2022-12-06
- Assignees: Roku, Inc.
- Applicants: Roku, Inc.
- Inventors: Sunil Ramesh; Michael Cutter; Karina Levitian
- Web page: https://patentgrep.com/patent/US12014753B1

## CPC

- G11B 27/036: Insert-editing
- G06F 18/251: {of input or preprocessed data}
- G06T 7/00: Image analysis
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/174: {Facial expression recognition}
- G06V 40/176: {Dynamic expression}
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 21/003: Changing voice quality, e.g. pitch or formants
- G10L 21/10: Transforming into visible information
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

In one aspect, an example method includes (i) estimating, using a skeletal detection model, a pose of an original actor for each of multiple frames of a video; (ii) obtaining, for each of a plurality of the estimated poses, a respective image of a replacement actor; (iii) obtaining replacement speech in the replacement actor's voice that corresponds to speech of the original actor in the video; (iv) generating, using the estimated poses, the images of the replacement actor, and the replacement speech, synthetic frames corresponding to the multiple frames of the video that depict the replacement actor in place of the original actor, with the synthetic frames including facial expressions for the replacement actor that temporally align with the replacement speech; and (iv) combining the synthetic frames and the replacement speech so as to obtain a synthetic video that replaces the original actor with the replacement actor.

## Claims

### Claim 1

A computing system comprising a processor and a non-transitory computer-readable medium having stored thereon program instructions that upon execution by the processor, cause performance of a set of acts comprising:
estimating, using a skeletal detection model, a pose of an original actor for each of multiple frames of a video;
obtaining, for each of a plurality of the estimated poses of the original actor, a respective image of a replacement actor;
obtaining replacement speech in the replacement actor's voice that corresponds to speech of the original actor in the video;
generating, using the estimated poses, the images of the replacement actor, and the replacement speech, synthetic frames corresponding to the multiple frames of the video that depict the replacement actor in place of the original actor, wherein the synthetic frames depict the replacement actor in respective poses that align with the estimated poses of the original actor in corresponding frames of the video, and wherein the synthetic frames comprise facial expressions for the replacement actor that temporally align with the replacement speech, wherein generating the synthetic frames comprises, for a given frame of the multiple frames, inserting, using an object insertion model, an image of the replacement actor into the given frame at a location indicated by the estimated pose of the original actor so as to obtain a modified frame, and wherein generating the synthetic frames further comprises providing the replacement speech and the modified frame as input to a temporal generative adversarial network having an ensemble of discriminators; and
combining the synthetic frames and the replacement speech so as to obtain a synthetic video that replaces the original actor with the replacement actor.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein obtaining the images of the replacement actor comprises estimating a pose of the replacement actor for each of multiple frames of a sample video of the replacement actor.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein obtaining the images of the replacement actor comprises:
recording a sample video of the replacement actor; and
identifying poses of the replacement actor during the sample video using a motion capture system.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein obtaining the images of the replacement actor comprises rendering an image of the replacement actor using a target pose, an image of the replacement actor, and a pose rendering model.

### Claim 5 (depends on 1)

The computing system of claim 1, wherein obtaining the replacement speech comprises:
obtaining a speech sample for the replacement actor; and
generating, using the speech sample and a speech engine, synthesized speech in the replacement actor's voice that corresponds to speech of the original actor in the video.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein generating the synthetic frames further comprises, for the given frame, extracting the original actor from the given frame using the estimated poses of the original actor.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein the ensemble of discriminators comprises a frame discriminator, a sequence discriminator, and a synchronization discriminator.

### Claim 8 (depends on 1)

The computing system of claim 1, wherein the set of acts further comprises receiving data indicative of a selection of the replacement actor from among a plurality of replacement actors.

### Claim 9 (depends on 1)

The computing system of claim 1, wherein the set of acts further comprises:
obtaining a user attribute of a viewer of the video; and
selecting the replacement actor from among a plurality of replacement actors using the user attribute of the viewer.

### Claim 10 (depends on 9)

The computing system of claim 9, wherein the user attribute of the viewer is a demographic attribute.

### Claim 11 (depends on 9)

The computing system of claim 9, wherein the user attribute of the viewer is a viewing history.

### Claim 12

A method comprising:
estimating, by a computing system using a skeletal detection model, a pose of an original actor for each of multiple frames of a video;
obtaining, by the computing system for each of a plurality of the estimated poses of the original actor, a respective image of a replacement actor;
obtaining, by the computing system, replacement speech in the replacement actor's voice that corresponds to speech of the original actor in the video;
generating, by the computing system using the estimated poses, the images of the replacement actor, and the replacement speech, synthetic frames corresponding to the multiple frames of the video that depict the replacement actor in place of the original actor, wherein the synthetic frames depict the replacement actor in respective poses that align with the estimated poses of the original actor in corresponding frames of the video, and wherein the synthetic frames comprise facial expressions for the replacement actor that temporally align with the replacement speech, wherein generating the synthetic frames comprises, for a given frame of the multiple frames, inserting, using an object insertion model, an image of the replacement actor into the given frame at a location indicated by the estimated pose of the original actor so as to obtain a modified frame, and wherein generating the synthetic frames further comprises providing the replacement speech and the modified frame as input to a temporal generative adversarial network having an ensemble of discriminators; and
combining the synthetic frames and the replacement speech so as to obtain a synthetic video that replaces the original actor with the replacement actor.

### Claim 13 (depends on 12)

The method of claim 12, wherein obtaining the images of the replacement actor comprises estimating a pose of the replacement actor for each of multiple frames of a sample video of the replacement actor.

### Claim 14 (depends on 12)

The method of claim 12, wherein obtaining the images of the replacement actor comprises:
recording a sample video of the replacement actor; and
identifying poses of the replacement actor during the sample video using a motion capture system.

### Claim 15 (depends on 12)

The method of claim 12, wherein obtaining the images of the replacement actor comprises rendering an image of the replacement actor using a target pose, an image of the replacement actor, and a pose rendering model.

### Claim 16 (depends on 12)

The method of claim 12, wherein obtaining the replacement speech comprises:
obtaining a speech sample for the replacement actor; and
generating, using the speech sample and a speech engine, synthesized speech in the replacement actor's voice that corresponds to speech of the original actor in the video.

### Claim 17 (depends on 12)

The method of claim 12, further comprising receiving data indicative of a selection of the replacement actor from among a plurality of replacement actors.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
obtaining a user attribute of a viewer of the video; and
selecting the replacement actor from among a plurality of replacement actors using the user attribute of the viewer.

### Claim 19 (depends on 18)

The method of claim 18, wherein the user attribute of the viewer is a demographic attribute.

### Claim 20

A non-transitory computer-readable medium having stored thereon program instructions that upon execution by a computing system, cause performance of a set of acts comprising:
estimating, using a skeletal detection model, a pose of an original actor for each of multiple frames of a video;
obtaining, for each of a plurality of the estimated poses of the original actor, a respective image of a replacement actor;
obtaining replacement speech in the replacement actor's voice that corresponds to speech of the original actor in the video;
generating, using the estimated poses, the images of the replacement actor, and the replacement speech, synthetic frames corresponding to the multiple frames of the video that depict the replacement actor in place of the original actor, wherein the synthetic frames depict the replacement actor in respective poses that align with the estimated poses of the original actor in corresponding frames of the video, and wherein the synthetic frames comprise facial expressions for the replacement actor that temporally align with the replacement speech, wherein generating the synthetic frames comprises, for a given frame of the multiple frames, inserting, using an object insertion model, an image of the replacement actor into the given frame at a location indicated by the estimated pose of the original actor so as to obtain a modified frame, and wherein generating the synthetic frames further comprises providing the replacement speech and the modified frame as input to a temporal generative adversarial network having an ensemble of discriminators; and
combining the synthetic frames and the replacement speech so as to obtain a synthetic video that replaces the original actor with the replacement actor.