# US 2021/0174544 A1 · Video synthesis device, video synthesis method and recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174544 A1
- Document ID: US20210174544A1
- Status: Published application
- Kind: A1
- Application number: 17/263,760
- Filed: 2018-12-25
- Published: 2021-06-10
- Earliest claimed: 2018-11-30
- Assignees: DWANGO Co., Ltd.
- Applicants: DWANGO Co., Ltd.
- Inventors: Nobuo KAWAKAMI; Shinnosuke IWAKI; Takashi KOJIMA; Toshihiro SHIMIZU; Hiroaki SAITO
- Web page: https://patentgrep.com/patent/US20210174544A1

## CPC

- G06T 7/75: {involving models}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/176: {Dynamic expression}
- H04N 5/262: Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects {; Cameras specially adapted for the electronic generation of special effects}
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 23/00: Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N 23/60: Control of cameras or camera modules
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A rear-facing camera (17) captures a live-action video image while a front-facing camera (16) captures an image of a distributor. An avatar controller (13) controls an avatar based on the image of the distributor captured by the front-facing camera (16). A synthesizer (14) arranges the avatar in a predetermined position of a real space coordinate system and synthesizes the avatar with the live-action video image. The face of the distributor captured by the front-facing camera (16) is tracked and reflected on the avatar.

## Claims

### Claim 1

A video synthesis device configured to generate an augmented reality video image in which an avatar is synthesized with a live-action video image, the device comprising:
a first imaging unit configured to capture a live-action video image;
a second imaging unit configured to capture an image of an operator;
a position detector configured to detect a position and an orientation of the first imaging unit or the video synthesis device in a coordinate system corresponding to a real space;
a controller configured to control the avatar based on the image captured by the second imaging unit;
an arrangement unit configured to arrange the avatar in any initial position of the coordinate system in which the avatar is arrangeable, based on an operation by the operator; and
a synthesizer configured to synthesize the avatar with the live-action video image captured by the first imaging unit,
wherein the video synthesis device is implemented as a mobile terminal held by the operator, and the live-action video image in which the avatar is synthesized is configured for distribution to a viewer via a server.

### Claim 2 (depends on 1)

The video synthesis device of claim 1, further comprising:
a spatial measurement unit configured to obtain 3-dimensional spatial information of a location where the live-action video image is captured and further configured to detect an area in which the avatar is arrangeable,
wherein the arrangement unit is configured to present the area in the live-action video image to the operator and to receive the operation by the operator.

### Claim 3 (depends on 2)

The video synthesis device of claim 2, wherein
the arrangement unit is configured to move a position of the avatar within the area according to the operation by the operator to determine the position of the avatar synthesized with the live-action video image.

### Claim 4 (depends on 1)

The video synthesis device of claim 1, wherein
the controller is configured to track a face of the operator captured by the second imaging unit and reflect the face on a face of the avatar.

### Claim 5 (depends on 1)

The video synthesis device of claim 1, wherein
the controller is configured to lock the position and the orientation of the avatar in the coordinate system.

### Claim 6 (depends on 1)

The video synthesis device of claim 1, wherein
capturing directions of the first imaging unit and the second imaging unit are opposite to each other.

### Claim 7

A video synthesis method for generating an augmented reality video image in which an avatar is synthesized with a live-action video image, the method, executed by a mobile terminal held by an operator, comprising:
detecting a position and an orientation of a first imaging unit configured to capture the live-action video image, in a coordinate system corresponding to a real space;
controlling the avatar based on an image captured by a second imaging unit configured to capture the image of the operator;
arranging the avatar in any initial position of the coordinate system in which the avatar is arrangeable, based on an operation by the operator; and
synthesizing the avatar with the live-action video image captured by the first imaging unit,
wherein the live-action video image in which the avatar is synthesized is configured for distribution to a viewer via a server.

### Claim 8

A computer-readable recording medium, storing a video synthesis program for generating an augmented reality video image in which an avatar is synthesized with a live-action video image that causes a processor to execute:
detecting a position and an orientation of a first imaging unit configured to capture the live-action video image in a coordinate system corresponding to a real space;
controlling the avatar based on an image captured by a second imaging unit configured to capture an image of an operator;
arranging the avatar in any initial position of the coordinate system in which the avatar is arrangeable, based on an operation by the operator; and
synthesizing the avatar with the live-action video image captured by the first imaging unit,
wherein the processes are executed by a mobile terminal held by the operator, and the live-action video image in which the avatar is synthesized is configured for distribution to a viewer via a server.

### Claim 9 (depends on 7)

The video synthesis method of claim 7, further comprising:
obtaining 3-dimensional spatial information of a location where the live-action video image is captured;
detecting an area in which the avatar is arrangeable;
presenting the area in the live-action video image to the operator; and
receiving the operation by the operator.

### Claim 10 (depends on 9)

The video synthesis method of claim 9, further comprising:
moving a position of the avatar within the area according to the operation by the operator to determine the position of the avatar synthesized with the live-action video image.

### Claim 11 (depends on 7)

The video synthesis method of claim 7, further comprising:
tracking a face of the operator captured by the second imaging unit; and
reflecting the face on a face of the avatar.

### Claim 12 (depends on 7)

The video synthesis method of claim 7, further comprising:
locking the position and the orientation of the avatar in the coordinate system.

### Claim 13 (depends on 7)

The video synthesis method of claim 7, wherein
capturing directions of the first imaging unit and the second imaging unit are opposite to each other.

### Claim 14 (depends on 8)

The computer-readable recording medium of claim 8, further causes a processor to execute:
obtaining 3-dimensional spatial information of a location where the live-action video image is captured;
detecting an area in which the avatar is arrangeable;
presenting the area in the live-action video image to the operator; and
receiving the operation by the operator.

### Claim 15 (depends on 14)

The computer-readable recording medium of claim 14, further causes a processor to execute:
moving a position of the avatar within the area according to the operation by the operator to determine the position of the avatar synthesized with the live-action video image.

### Claim 16 (depends on 8)

The computer-readable recording medium of claim 8, further causes a processor to execute:
tracking a face of the operator captured by the second imaging unit; and
reflecting the face on a face of the avatar.

### Claim 17 (depends on 8)

The computer-readable recording medium of claim 8, further causes a processor to execute:
locking the position and the orientation of the avatar in the coordinate system.

### Claim 18 (depends on 8)

The computer-readable recording medium of claim 8, wherein
capturing directions of the first imaging unit and the second imaging unit are opposite to each other.

### Claim 19 (depends on 1)

The video synthesis device of claim 1, wherein the first imaging unit is a front-facing camera and the second imaging unit is a rear-facing camera.

### Claim 20 (depends on 1)

The video synthesis device of claim 1, wherein the position detector includes one or more sensors.
