# US 2014/0192164 A1 · System and method for determining depth information in augmented reality scene

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0192164 A1
- Document ID: US20140192164A1
- Status: Published application
- Kind: A1
- Application number: 13/735,838
- Filed: 2013-01-07
- Published: 2014-07-10
- Earliest claimed: 2013-01-07
- Assignees: Industrial Technology Research Institute
- Applicants: Industrial Technology Research Institute
- Inventors: Hian-Kun TENN; Yao-Yang TSAI; Ko-Shyang WANG; Po-Lung CHEN
- Web page: https://patentgrep.com/patent/US20140192164A1

## CPC

- H04N 13/246: Calibration of cameras
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system and method for determining individualized depth information in an augmented reality scene are described. The method includes receiving a plurality of images of a physical area from a plurality of cameras, extracting a plurality of depth maps from the plurality of images, generating an integrated depth map from the plurality of depth maps, and determining individualized depth information corresponding to a point of view of the user based on the integrated depth map and a plurality of position parameters.

## Claims

### Claim 1

A method for determining individualized depth information in an augmented reality scene, comprising:
receiving a plurality of images of a physical area from a plurality of cameras;
extracting a plurality of depth maps from the plurality of images;
generating an integrated depth map from the plurality of depth maps; and
determining individualized depth information corresponding to a point of view of a user based on the integrated depth map and a plurality of position parameters.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving the position parameters from a user device, the position parameters indicative of the point of view of the user associated with the user device within the physical area.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating an image of an augmented reality scene based on the individualized depth information, the augmented reality scene including a combination of the physical area and a computer-generated virtual object, the image representing a view of the augmented reality scene consistent with the point of view of the user.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
detecting a change in the point of view of the user; and
updating the image of the augmented reality, in real time, in response to the change in the point of view.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
receiving an additional image of the physical area;
generating the image of the augmented reality based additionally on the additional image of the physical area.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
receiving the additional image of the physical area from the user device.

### Claim 7 (depends on 5)

The method of claim 5, wherein the additional image of the physical area includes at least one image of a physical object disposed within the physical area, and the individualized depth information indicates a relative position of the physical object within the physical area.

### Claim 8 (depends on 7)

The method of claim 7, wherein the generating of the image of the augmented reality scene comprises:
generating a virtual object;
determining an occlusion relationship between the virtual object and the physical object based on the individualized depth information; and
forming the image of the augmented reality scene by combining the image of the virtual object with the additional image of the physical area according to the occlusion relationship.

### Claim 9 (depends on 1)

The method of claim 1, wherein each depth map is defined in a coordinate system associated with one of the cameras, the generating of the integrated depth map further comprising:
selecting the coordinate system associated with one of the cameras as a common coordinate system;
transforming the depth maps defined in other coordinate systems associated with other ones of the cameras to the common coordinate system; and
combining the transformed depth maps and the depth map defined in the common coordinate system.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
transforming the position parameters of the user device to the common coordinate system.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
receiving, from the cameras, a plurality of images of a calibration object including a plurality of feature points;
identifying the feature points in the images of the calibration object;
determining at least one transformation matrix indicative of a coordinate transformation from the other coordinate systems to the common coordinate system; and
transforming the depth maps defined in the other coordinate systems based on the transformation matrix.

### Claim 12 (depends on 1)

The method of claim 1, wherein the images of the physical area from the cameras correspond to different points of view.

### Claim 13 (depends on 2)

The method of claim 2, further comprising transmitting the individualized depth information to the user device.

### Claim 14

A non-transitory computer-readable medium comprising instructions, which, when executed by a processor, causes the processor to perform a method for determining individualized depth information in an augmented reality scene, the method comprising:
receiving a plurality of images of a physical area from a plurality of cameras;
extracting a plurality of depth maps from the plurality of images;
generating an integrated depth map from the plurality of depth maps; and
determining individualized depth information corresponding to a point of view of a user based on the integrated depth map and a plurality of position parameters.

### Claim 15 (depends on 14)

The computer-readable medium of claim 14, the method further comprising:
receiving the position parameters from a user device, the position parameters indicative of a point of view of a user associated with the user device within the physical area.

### Claim 16 (depends on 14)

The computer-readable medium of claim 14, the method further comprising:
generating an image of an augmented reality scene based on the individualized depth information, the augmented reality scene including a combination of the physical area and a computer-generated virtual object, the image representing a view of the augmented reality scene consistent with the point of the view of the user.

### Claim 17 (depends on 16)

The computer-readable medium of claim 16, the method further comprising:
detecting a change in the point of view of the user; and
updating the image of the augmented reality, in real time, in response to the change in the point of view.

### Claim 18 (depends on 16)

The computer-readable medium of claim 16, the method further comprising:
receiving an additional image of the physical area;
generating the image of the augmented reality scene based additionally on the additional image of the physical area.

### Claim 19 (depends on 18)

The computer-readable medium of claim 18, the method further comprising:
receiving the additional image of the physical area from the user device.

### Claim 20 (depends on 18)

The computer-readable medium of claim 18, wherein the additional image of the physical area includes at least one image of a physical object disposed within the physical area, and the individualized depth information indicates a relative position of the physical object within the physical area.

### Claim 21 (depends on 20)

The computer-readable medium of claim 20, wherein the generating of the image of the augmented reality scene comprises:
generating a virtual object;
determining an occlusion relationship between the virtual object and the physical object based on the individualized depth information; and
forming the image of the augmented reality scene by combining the image of the virtual object with the additional image of the physical area according to the occlusion relationship.

### Claim 22

A system for determining individualized depth information in an augmented reality scene, comprising:
a memory for storing instructions; and
a processor for executing the instructions to:
receive a plurality of images of a physical area from a plurality of cameras;
extract a plurality of depth maps from the plurality of images;
generate an integrated depth map from the plurality of depth maps;
receive position parameters from a user device, the position parameters indicative of a point of view of a user associated with the user device within the physical area; and
determine individualized depth information corresponding to the point of view of the user based on the integrated depth map and the position parameters.