# US 2020/0402310 A1 · Virtual object display control device, virtual object display system, virtual object display control method, and storage medium storing virtual object display control program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402310 A1
- Document ID: US20200402310A1
- Status: Published application
- Kind: A1
- Application number: 16/971,443
- Filed: 2018-02-26
- Published: 2020-12-24
- Earliest claimed: 2018-02-26
- Assignees: Mitsubishi Electric Corporation
- Applicants: Mitsubishi Electric Corporation
- Inventors: Masaya NIDAIRA; Satoshi SAKURAI
- Web page: https://patentgrep.com/patent/US20200402310A1

## CPC

- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 13/20: Three-dimensional [3D] animation
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2219/2016: Rotation, translation, scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A virtual object display control device includes a recognition unit to receive real space information; a viewpoint position judgment unit to judge a position of a viewpoint of an observer; a real object judgment unit to judge a position and a shape of a real object; an image control unit to generate image information for displaying an animation of a virtual object; a display setting unit to judge whether to display the animation of the virtual object or not, based on the position of the viewpoint, the position and the shape of the real object, and the image information on the virtual object and to set image information including the image information on the virtual object or the image information for displaying the animation of the virtual object as display image information based on a result of the judgment; and a drawing unit to output the display image information.

## Claims

### Claim 1

A virtual object display control device comprising:
a processor to execute a program; and
a memory to store the program which, when executed by the processor, performs the processes of:
receiving real space information indicating a real space;
judging a position of a viewpoint of an observer based on the real space information;
judging a position and a shape of a real object based on the real space information;
receiving image information on a virtual object and processing the image information on the virtual object, thereby generating image information for displaying an animation of the virtual object;
judging whether to display the animation of the virtual object or not, based on the position of the viewpoint, the position and the shape of the real object, and the image information on the virtual object, and setting image information including any one of the image information on the virtual object and the image information for displaying the animation of the virtual object, as display image information, based on a result of the judgment; and
outputting the display image information,
wherein the image information for displaying the animation of the virtual object is set as the display image information when a whole or part of the virtual object is hidden by the real object as viewed from the position of the viewpoint.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The virtual object display control device according to claim 1, wherein the image information for displaying the animation of the virtual object is image information for scaling display as display performed so that size of the virtual object changes or image information for moving display as display performed so that the virtual object moves.

### Claim 4 (depends on 1)

The virtual object display control device according to claim 1, wherein the image information for displaying the animation of the virtual object is image information for scaling display as display performed so that size of the virtual object changes and image information for moving display as display performed so that the virtual object moves, and any one of the image information for the scaling display and the image information for the moving display is selected based on a predetermined condition.

### Claim 5 (depends on 1)

The virtual object display control device according to claim 1, wherein the real space information includes image information on the real space and depth information on the real object.

### Claim 6 (depends on 1)

The virtual object display control device according to claim 1, wherein synthetic image information obtained by combining the image information on the virtual object or the image information for displaying the animation of the virtual object with image information on the real space is outputted as the display image information.

### Claim 7 (depends on 1)

A virtual object display system comprising:
a camera to acquire real space information indicating a real space;
the virtual object display control device according to claim 1; and
a display device to display an image based on the display image information.

### Claim 8 (depends on 7)

The virtual object display system according to claim 7, wherein the camera acquires image information on the real space and depth information on the real object.

### Claim 9 (depends on 7)

The virtual object display system according to claim 7, wherein synthetic image information obtained by combining image information outputted from the image control unit with image information on the real space is set as the display image information.

### Claim 10 (depends on 7)

The virtual object display system according to claim 7, wherein the display device includes:
another camera to acquire image capture information as viewed from the position of the viewpoint; and
a display screen;
wherein the display device makes the display screen display an image in which image information for displaying an animation of the virtual object is superimposed on the image capture information.

### Claim 11 (depends on 7)

The virtual object display system according to claim 7, wherein the display device includes a projector to project the display image information onto the real space.

### Claim 12

A virtual object display control method comprising:
receiving real space information indicating a real space;
judging a position of a viewpoint of an observer based on the real space information;
judging a position and a shape of a real object based on the real space information;
receiving image information on a virtual object and processing the image information on the virtual object, thereby generating image information for displaying an animation of the virtual object;
judging whether to display the animation of the virtual object or not, based on the position of the viewpoint, the position and the shape of the real object, and the image information on the virtual object, and setting image information including any one of the image information on the virtual object and the image information for displaying the animation of the virtual object, as display image information based on a result of the judgment; and
outputting the display image information,
wherein, at the setting of the image information, the image information for displaying the animation of the virtual object is set as the display image information when a whole or part of the virtual object is hidden by the real object as viewed from the position of the viewpoint.

### Claim 13

A non-transitory computer-readable storage medium storing a virtual object display control program that causes a computer to execute processing comprising:
receiving real space information indicating a real space;
judging a position of a viewpoint of an observer based on the real space information;
judging a position and a shape of a real object based on the real space information;
receiving image information on a virtual object and processing the image information on the virtual object, thereby generating image information for displaying an animation of the virtual object;
judging whether to display the animation of the virtual object or not, based on the position of the viewpoint, the position and the shape of the real object, and the image information on the virtual object, and setting image information including any one of the image information on the virtual object and the image information for displaying the animation of the virtual object, as display image information based on a result of the judgment; and
outputting the display image information,
wherein, at the setting of the image information, the image information for displaying the animation of the virtual object is set as the display image information when a whole or part of the virtual object is hidden by the real object as viewed from the position of the viewpoint.
