# US 2020/0401806 A1 · Augmented reality display system, program, and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401806 A1
- Document ID: US20200401806A1
- Status: Published application
- Kind: A1
- Application number: 17/007,699
- Filed: 2020-08-31
- Published: 2020-12-24
- Earliest claimed: 2017-05-26
- Assignees: DWANGO Co., Ltd.
- Applicants: DWANGO Co., Ltd.
- Inventors: Kouichi NAKAMURA; Kazuya ASANO
- Web page: https://patentgrep.com/patent/US20200401806A1

## CPC

- G06V 20/20: in augmented reality scenes
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/216: using geographical information, e.g. location of the game device or player using GPS
- A63F 13/52: involving aspects of the displayed game scene
- A63F 13/5378: for displaying an additional top view, e.g. radar screens or maps
- A63F 13/655: by importing photos, e.g. of the player
- G06F 3/147: using display panels
- G06F 16/55: Clustering; Classification
- G06F 16/587: using geographical or spatial information, e.g. location
- G06T 7/74: {involving reference images or patches}
- G06T 15/503: {Blending, e.g. for anti-aliasing}
- G06T 17/05: Geographic models
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09B 29/10: Map spot or coordinate position indicators; Map reading aids (optical projection apparatus per se G03B)
- G09B 29/106: {using electronic means}
- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30244: Camera pose
- G09G 2340/10: Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

An augmented reality displaying system for displaying a virtual object through compositing on an image taken of the real world, comprising: a camera for capturing an image of the real world; a location information acquiring portion for acquiring, as location information, the coordinates and orientation at the instant of imaging by the camera; an image analyzing portion for analyzing, as depth information, the relative distances to imaging subjects for individual pixels that structure the real world image that has been captured; a virtual display data generating portion for generating virtual display data, on real map information that includes geographical information in the real world, based on the location information acquired by the location information acquiring portion; and a compositing processing portion for displaying the virtual display data, generated by the virtual display data generating portion, superimposed on an image captured by the camera in accordance with the depth information.

## Claims

### Claim 1

A computer program product configured to cause a computer to function as:
an imaging portion configured to capture an image of the real world, wherein the image of the real world comprises image information from at least two points that are physically separate from each other;
a location information acquiring portion configured to acquire location information comprising coordinates and orientation at the moment at which the image is captured;
an image analyzing portion configured to analyze, as depth information, relative distances to imaging subjects of individual pixels that constitute the image of the real world that has been captured;
a virtual display data generating portion configured to generate virtual display data that expresses the state of placement of virtual objects on real map information that includes geographical information in the real world, based on location information acquired by the location information acquiring portion;
a compositing processing portion configured to display the virtual display data superimposed on the image that has been captured by the imaging portion, including color and position of each individual pixel that constitute the image of the real world; and
a learning processing portion configured to generate correlation information between colors and locations within the image, location information at the moment of image capture, the pixels that constitute the image, relating to a relative distance at the moment of image capture, and other relative distances calculated by the image analyzing portion,
wherein the image analyzing portion is further configured to analyze depth information of a single image of the real world using correlation results from the learning processing portion,
wherein the learning processing portion is configured to store a distance to an object obtained by analyzing a parallax of an image captured at the two physically separate points as the correlation information in association with a hierarchical combination pattern of characteristics for each pixel and position information at the real world location of image capture, and the image analyzing portion is configured to store the correlation information, and
wherein the image of the real world captured by the imaging portion references the correlation information and the depth information of the image of the real world to determine the degree of match to the hierarchical combination patterns.

### Claim 2 (depends on 1)

The computer program product of claim 1,
wherein the learning processing portion is configured to correct the correlation information based on information about a height of a building, terrain, or topography contained within the image of the real world.

### Claim 3 (depends on 1)

The computer program product of claim 1, further causing a computer to function as:
a layer generating portion configured to distribute each of the pixels constituting the image to a plurality of layers that form a hierarchy in the direction of depth of the image of the real world, and is further configured to provide the plurality of layers to the compositing processing portion,
wherein compositing processing portion is further configured to superimpose the virtual display data on the image of the real world depending on the respective depth information for individual layers of the plurality of layers.

### Claim 4

An augmented reality displaying method for compositing and displaying virtual objects on an image of the real world, including:
capturing an image of the real world, wherein the image of the real world comprises image information from at least two points that are physically separate from each other;
acquiring location information comprising coordinates and orientation at the moment at which the image is captured;
analyzing, as depth information, relative distances to imaging subjects of individual pixels that constitute the image of the real world;
generating virtual display data that expresses the state of placement of virtual objects on real map information that includes geographical information in the real world, based on acquired location information;
displaying the virtual display data superimposed on the image that has been captured, including color and position of each individual pixel that constitute the image of the real world;
generating correlation information between colors and locations within the image, location information at the time of image capture, the pixels that constitute the image, relating a relative distance at the moment of image capture, and other relative distances to imaging subjects of the individual pixels;
storing a distance to an object obtained by analyzing a parallax of an image captured at the two physically separate points as the correlation information in association with a hierarchical combination pattern of characteristics for each pixel and position information at the real world location of image capture; and
determining the degree of match to the hierarchical combination patterns by referencing the correlation information and the depth information of the image of the real world.

### Claim 5 (depends on 4)

The augmented reality display method of claim 4, further comprising:
correcting the correlation information based on information about a height of a building, terrain, or topography contained within the image of the real world.

### Claim 6 (depends on 4)

The augmented reality display method of claim 4, further comprising”
distributing each of the pixels constituting the image to a plurality of layers that form a hierarchy in the direction of depth of the image of the real world; and
superimposing the virtual display data on the image of the real world depending on the respective depth information for individual layers of the plurality of layers.