# US 10,121,257 B2 · Computer-implemented method and system for processing video with temporal consistency

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,121,257 B2
- Document ID: US10121257B2
- Status: Granted
- Kind: B2
- Application number: 15/603,604
- Filed: 2017-05-24
- Published: 2017-11-30
- Granted: 2018-11-06
- Earliest claimed: 2016-05-25
- Assignees: Rational Pixels S.L.
- Applicants: Rational Pixels S.L.
- Inventors: Rida Sadek Sadek; Juan Francisco Garamendi; Pablo Arias Martinez
- Web page: https://patentgrep.com/patent/US10121257B2

## CPC

- G06T 7/251: {involving models}
- G06T 7/215: Motion-based segmentation
- G06T 7/269: using gradient-based methods
- G11B 27/028: with computer assistance
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- H04N 19/51: Motion estimation or motion compensation
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

System and computer-implemented method for editing a video sequence with temporal consistency. The method includes the steps of: computing a motion field modeling temporal consistency between successive frames; defining an energy functional modeling the desired properties to be enforced on the video sequence; splitting the video sequence into two sets with even frames and odd frames; computing the motion field between consecutive frames on the splitted sequences; recursively performing steps until the sets to be split contain one frame to edit; minimizing the energy functional for each set containing one frame to edit; merging the edited frames and outputting the edited video sequence.

## Claims

### Claim 1

A computer-implemented method for editing a video sequence with temporal consistency comprising the steps of:
i) computing a motion field modeling temporal consistency between successive frames;
ii) defining an energy functional modeling the desired properties to be enforced on the video sequence;
iii) splitting the video sequence into two sets with even frames and odd frames;
iv) computing the motion field between consecutive frames on the splitted sequences;
v) recursively performing steps iii) and iv) until the sets to be split contain one frame to edit;
vi) minimizing the energy functional for each set containing one frame to edit;
vii) merging the edited frames and outputting the edited video sequence.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein it comprises a previous step of identifying a frame in a video sequence having an edited object to be propagated.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the energy functional comprises a summation of editing energies depending on single-frames and temporal coupling energies depending on pairs of consecutive frames.

### Claim 4 (depends on 1)

The computer implemented method of claim 1, wherein the energy functional is based on at least one of the following model:
global brightness change assumption;
brightness constancy assumption;
gradient constancy assumption.

### Claim 5

A system for editing a video sequence with temporal consistency comprising:
a first processing means or computing a motion field (v) modeling temporal consistency between successive frames (fi, fi+1) and for defining an energy functional (J) modeling the properties to be enforced in the video sequence (f);
a second processing means for splitting the video sequence (f) into two sets with even frames (feven) and odd frames (fodd), wherein the splitting is done recursively until there is only one frame to edit in each set;
a third processing means for computing the motion field (v) between consecutive frames in each set after each split;
a fourth processing means for minimizing the energy functional (J) on each set to obtain edited frames;
a fifth processing means for merging the edited frames and outputting the entire edited video sequence (u).

### Claim 6 (depends on 5)

A system for editing a video sequence with temporal consistency of claim 5, comprising a sixth processing means for identifying a frame in a video sequence having an edited object to be propagated.

### Claim 7 (depends on 1)

A computer program product for editing a video sequence with temporal consistency comprising computer code instructions stored thereon that, when executed by a processor, causes the processor to perform the method of claim 1.