# US 2010/0149419 A1 · Multi-video synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0149419 A1
- Document ID: US20100149419A1
- Status: Published application
- Kind: A1
- Application number: 12/334,231
- Filed: 2008-12-12
- Published: 2010-06-17
- Earliest claimed: 2008-12-12
- Assignees: Microsoft Corporation
- Inventors: Tao Mei; Xian-Sheng Hua; Shipeng Li; Teng Li
- Web page: https://patentgrep.com/patent/US20100149419A1

## CPC

- G11B 27/036: Insert-editing
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

Embodiments that provide multi-video synthesis are disclosed. In accordance with one embodiment, multi-video synthesis includes breaking a main video into a plurality of main frames and break a supplementary video into a plurality of supplementary frames. The multi-video synthesis also includes assigning one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), where each of the plurality of states corresponding to one or more main frames. The multi-video synthesis further includes determining optimal frames in the plurality of main frames for insertion of the plurality of supplementary frames based on the plurality of states and visual properties. The optimal frames include optimal insertion positions. The multi-video synthesis additionally includes inserting the plurality of supplementary frames into the optimal insertion positions to form a synthesized video.

## Claims

### Claim 1

A system, comprising:
a division component to break a main video into a plurality of main frames, and to break a supplementary video into a plurality of supplementary frames;
a modeling component to assign one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), and correspond each of the plurality of states to one or more of the plurality of main frames;
a detection component to determine optimal frames that include optimal insertion positions in the plurality of main frames for insertion of the plurality of supplementary frames based on the plurality of states and visual properties;
an insertion component to insert the plurality of supplementary frames into the optimal insertion positions to form a synthesized video; and
a blend component to blend the inserted supplementary frames and the optimal frames of the synthesized video.

### Claim 2 (depends on 1)

The system of claim 1, further comprising:
an informativeness measurement component to compute expected losses of information from the plurality of the main frames due to insertion of the plurality of the supplementary frames,
wherein the visual properties include the expected losses computed by the informativeness measurement component.

### Claim 3 (depends on 1)

The system of claim 1, further comprising:
a consistency measurement component to compute expected visual consistencies between the plurality of the main frames and the plurality of the supplementary frames,
wherein the visual properties include the expected visual consistencies computed by the consistency measurement component.

### Claim 4 (depends on 1)

The system of claim 1, further comprising:
a visual naturalness component to compute expected visual naturalness between a first boundary areas in the plurality of the main frame and a second boundary areas in the plurality of supplementary frames,
wherein the visual properties include the expected visual naturalness computed by the visual naturalness component.

### Claim 5 (depends on 1)

The system of claim 1, wherein the detection component is to determine the optimal insertion positions included in the optimal frames via a Viterbi beam search.

### Claim 6 (depends on 1)

The system of claim 1, wherein the detection component is to determine the optimal frames via a beam search for an optimal insertion path based on the states and the visual properties, wherein the optimal insertion path includes spatially stable optimal insertion positions that correspond to the optimal frames

### Claim 7 (depends on 1)

The system of claim 1, wherein the blend component is to blend the inserted supplementary frames and the optimal frames using probabilistic matting.

### Claim 8 (depends on 1)

The system of claim 1, wherein the detection component is to determine the optimal frames by formulating insertion of the plurality of the supplementary frames into the optimal frames as a Maximum a Posterior (MAP) problem.

### Claim 9 (depends on 1)

The system of claim 1, wherein the detection component is to determine the optimal frames by treating each of the main frames and each of the supplementary frames as including two spatial coordinates and a time coordinate.

### Claim 10

A computer readable medium storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:
breaking a main video into a plurality of main frames and break a supplementary video into a plurality of supplementary frames;
assigning one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), each of the plurality of states corresponding to one or more main frames;
determining optimal frames that include optimal insertion positions in the plurality of main frames for insertion of the plurality of supplementary frames by performing a beam search for an optimal insertion path based on the plurality of states and visual properties; and
inserting the plurality of supplementary frames into the optimal insertion positions to form a synthesized video.

### Claim 11 (depends on 10)

The computer readable medium of claim 10, wherein the optimal insertion path includes spatially stable optimal insertion positions that correspond to the optimal frames.

### Claim 12 (depends on 10)

The computer readable medium of claim 10, wherein the computer-executable instructions, when executed, further cause the one or more processors to perform acts comprising computing expected losses of information from the plurality of the main frames due to insertion of the plurality of the supplementary frames, wherein the visual properties include the expected losses.

### Claim 13 (depends on 10)

The computer readable medium of claim 10, wherein the computer-executable instructions, when executed. Further cause the one or more processors to perform acts comprising computing expected visual consistencies between the plurality of the main frames and the plurality of the supplementary frames, wherein the visual properties include the expected visual consistencies.

### Claim 14 (depends on 10)

The computer readable medium of claim 10, wherein the computer-executable instructions, when executed, further cause the one or more processors to perform acts comprising computing expected visual naturalness between the first boundary areas in the plurality of the main frame and the second boundary areas in the plurality of supplementary frames, wherein the visual properties include the expected visual naturalness.

### Claim 15 (depends on 10)

The computer readable medium of claim 10, wherein the computer-executable instructions, when executed, further cause the one or more processors to perform acts comprising blending the inserted supplementary frames and the optimal frames of the synthesized video using probabilistic matting.

### Claim 16

A method, comprising:
breaking a main video into a plurality of main frames and break a supplementary video into a plurality of supplementary frames;
assigning one or more supplementary frames into each of a plurality of states of a Hidden Markov Model (HMM), each of the plurality of states corresponding to one or more main frames;
determining optimal frames that include optimal insertion positions in the plurality of main frames for insertion of the plurality of supplementary frames based on the plurality of states and visual properties, the visual properties including informativeness, consistency, and naturalness;
inserting the plurality of supplementary frames into the optimal insertion positions to form a synthesized video; and
blending the inserted supplementary frames and the optimal frames of the synthesized video.

### Claim 17 (depends on 16)

The method of claim 16, further comprising obtaining the informativeness visual property by computing expected losses of information from the plurality of the main frames due to insertion of the plurality of the supplementary frames.

### Claim 18 (depends on 16)

The method of claim 16, further comprising obtaining the consistency visual property by computing expected visual consistencies between the plurality of the main frames and the plurality of the supplementary frames.

### Claim 19 (depends on 16)

The method of claim 16, further comprising obtaining the visual naturalness visual property by computing expected visual naturalness between the first boundary areas in the plurality of the main frame and the second boundary areas in the plurality of supplementary frames.

### Claim 20 (depends on 16)

The method of claim 16, wherein the determining includes determining the optimal frames via a Viterbi beam search for an optimal insertion path based on the states and the visual properties, wherein the optimal insertion path includes spatially stable optimal insertion positions that correspond to the optimal frames.
