# US 2014/0152697 A1 · Method and apparatus for providing augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0152697 A1
- Document ID: US20140152697A1
- Status: Published application
- Kind: A1
- Application number: 14/068,547
- Filed: 2013-10-31
- Published: 2014-06-05
- Earliest claimed: 2012-12-05
- Assignees: Hyundai Motor Company
- Applicants: Hyundai Motor Company
- Inventors: Dongsoo Shin
- Web page: https://patentgrep.com/patent/US20140152697A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/01: Head-up displays
- G06T 7/44: using image operators, e.g. filters, edge density metrics or local histograms
- G06T 7/73: using feature-based methods
- G02B 2027/014: {comprising information/image processing systems}
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/30248: Vehicle exterior or interior
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A method and apparatus for providing augmented reality are provided and include a controller that is configured to match a head up display area of a windshield and an input image and determine an area in which an information amount is a minimum in the matched image as a position of a display window to display virtual information. In addition, the controller is configured to output virtual information at the determined position.

## Claims

### Claim 1

A method of providing augmented reality of an augmented reality providing apparatus mounted within a vehicle, the method comprising:
matching, by a controller, a head up display area of a windshield and an input image;
determining, by the controller, an area in which an information amount is a minimum in the matched image as a position of a display window to display virtual information; and
outputting, by the controller, virtual information at the determined position.

### Claim 2 (depends on 1)

The method of claim 1, wherein the determining of an area includes:
determining, by the controller, whether a marker related to the virtual information is included in the matched image;
when the marker is included in the matched image, dividing, by the controller, an area in which virtual information may be displayed in the matched image into a plurality of sub-blocks;
calculating, by the controller, the information amount based on the plurality of sub-blocks;
searching, by the controller, for a position at which the information amount is the minimum based on the plurality of sub-blocks; and
determining, by the controller, the position at which the information amount is the minimum as a position of the display window.

### Claim 3 (depends on 2)

The method of claim 2, wherein the calculating of an information amount includes:
acquiring, by the controller, included edges based on the plurality of sub-blocks;
counting, by the controller, the number of the edges;
calculating, by the controller, an average distance based on the plurality of sub-blocks;
normalizing, by the controller, the number of the edges; and
calculating, by the controller, the information amount based on the sub-block using the calculated average distance and the number of the normalized edges.

### Claim 4 (depends on 3)

The method of claim 3, wherein the searching of for a position includes:
generating, by the controller, an integral image based on the information amount; and
searching, by the controller, for a position at which the information amount is a first least amount using a window size as a search unit in the integral image.

### Claim 5 (depends on 4)

The method of claim 4, wherein the outputting of virtual information includes:
when an already output virtual information display window exists at the position at which the information amount is the first least amount, outputting, by the controller, virtual information at a position at which an information amount is a second least amount;
when the already output virtual information display window exists at the position at which the information amount is the second least amount, determining, by the controller, whether to output the virtual information display window according to a priority of previously defined display information;
determining, by the controller, whether output space exists at the determined position;
when output space exists, outputting, by the controller, the virtual information display window according to the priority at the output space; and
when output space does not exist, searching for, by the controller, a periphery and outputting the virtual information display window according to the priority on the already output virtual information display window in which the priority is below a predetermined threshold in an area in which the information amount is the minimum.

### Claim 6 (depends on 5)

The method of claim 5, wherein the outputting the virtual information display window according to the priority on the already output virtual information display window includes:
determining, by the controller, whether an ‘OFF’ flag is listed in the overwrite field in the area in which the information amount is the minimum; and
when the ‘OFF’ flag is not listed in the overwrite field, outputting, by the controller, a virtual information display window according to the priority on the already output virtual information display window in which the priority is below the predetermined threshold in the area in which the information amount is the minimum;

### Claim 7

An augmented reality providing apparatus mounted within a vehicle, comprising:
a controller configured to:
capture an input image positioned at a front side of a driver using an imaging device;
match the input image and a head up display area of a windshield;
determine whether a marker is detected in a matched image;
determine an area in which an information amount is a minimum in the matched image as a position of a display window to display virtual information that corresponds to the marker, when the marker is detected; and
output an augmented reality screen in which virtual information is displayed at the determined position.

### Claim 8 (depends on 7)

The augmented reality providing apparatus of claim 7, wherein the controller is further configured to:
divide an area in which virtual information is displayed in the matched image into a plurality of sub-blocks;
calculate the information amount based on the plurality of sub-blocks; and
determine a position at which the information amount is the minimum in the plurality of sub-blocks as the position to display the virtual information.

### Claim 9 (depends on 8)

The augmented reality providing apparatus of claim 8, wherein the controller is further configured to:
normalize the number of included edges based on the plurality of sub-block;
calculate the information amount based on the sub-block together with the sub-block average distance; and
search for the position at which the information amount is a first least amount using a size of a window as a search unit in a generated integral image based on the information amount.

### Claim 10 (depends on 9)

The augmented reality providing apparatus of claim 9, wherein the controller is further configured to:
output virtual information at the position at which the information amount is a second least amount, when an already output virtual information display window exists at the position at which the information amount is the first least amount;
determine whether to output the virtual information display window according to a priority of previously defined display information, when the already output virtual information display window exists at the position at which the information amount is the second least amount; and
search for a periphery, when output space does not exist, and output the virtual information display window according to the priority on the already output virtual information display window having a priority below a predetermined threshold in the area in which the information amount is the minimum.

### Claim 11

A non-transitory computer readable medium containing program instructions executed by a controller, the computer readable medium comprising:
program instructions that control an imaging device to capture an input image positioned at a front side of a driver;
program instructions that match the input image and a head up display area of a windshield;
program instructions that determine whether a marker is detected in a matched image;
program instructions that determine an area in which an information amount is a minimum in the matched image as a position of a display window to display virtual information that corresponds to the marker, when the marker is detected; and
program instructions that output an augmented reality screen in which virtual information is displayed at the determined position.

### Claim 12 (depends on 11)

The non-transitory computer readable medium of claim 11, further comprising:
program instructions that divide an area in which virtual information is displayed in the matched image into a plurality of sub-blocks;
program instructions that calculate the information amount based on the plurality of sub-blocks; and
program instructions that determine a position at which the information amount is the minimum in the plurality of sub-blocks as the position to display the virtual information.

### Claim 13 (depends on 12)

The non-transitory computer readable medium of claim 12, further comprising:
program instructions that normalize the number of included edges based on the plurality of sub-block;
program instructions that calculate the information amount based on the sub-block together with the sub-block average distance; and
program instructions that search for the position at which the information amount is a first least amount using a size of a window as a search unit in a generated integral image based on the information amount.

### Claim 14 (depends on 13)

The non-transitory computer readable medium of claim 13, further comprising:
program instructions that output virtual information at the position at which the information amount is a second least amount, when an already output virtual information display window exists at the position at which the information amount is the first least amount;
program instructions that determine whether to output the virtual information display window according to a priority of previously defined display information, when the already output virtual information display window exists at the position at which the information amount is the second least amount; and
program instructions that search for a periphery, when output space does not exist, and output the virtual information display window according to the priority on the already output virtual information display window having a priority below a predetermined threshold in the area in which the information amount is the minimum.