# US 12,008,683 B2 · Vehicle augmented reality navigational image output device and control methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,683 B2
- Document ID: US12008683B2
- Status: Granted
- Kind: B2
- Application number: 17/771,657
- Filed: 2020-01-06
- Published: 2022-12-01
- Granted: 2024-06-11
- Earliest claimed: 2020-01-06
- Assignees: LG Electronics Inc.
- Applicants: LG Electronics Inc.
- Inventors: Sunghwan Choi; Dukyung Jung
- Web page: https://patentgrep.com/patent/US12008683B2

## CPC

- G06V 20/20: in augmented reality scenes
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/22: Display screens
- B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10)
- B60K 35/81: for controlling displays
- B60K 35/85: Arrangements for transferring vehicle- or driver-related data
- G01C 21/3647: {Guidance involving output of stored or live camera images or video streams}
- G01C 21/365: {Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself}
- G01C 21/3658: {Lane guidance}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/65: on geographic maps
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/945: {User interactive design; Environments; Toolboxes}
- G06V 20/582: {of traffic signs}
- B60K 35/23: Head-up displays [HUD] (optical aspects of head-up displays G02B27/01)
- B60K 35/25: using haptic output
- B60K 35/26: using acoustic output
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 2360/166: Navigation
- B60K 2360/177: Augmented reality
- B60K 2360/31: Virtual images
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2210/32: Image data format
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates to an image output device disposed at a vehicle to enable augmented reality and a method for controlling the same. The image output device includes an image output unit configured to output visual information for realizing the augmented reality, a communication unit configured to receive a forward image that captures an image in a forward direction of the vehicle, and a processor configured to search for at least one planned driving lane on which the vehicle is planned to travel from the forward image and control the image output unit such that a main carpet image indicating the planned driving lane is output in lane units.

## Claims

### Claim 1

An image output device disposed at a vehicle to enable augmented reality, the image output device comprising:
an image output unit configured to output visual information for realizing the augmented reality;
a communication unit configured to receive a forward image that captures an image in a forward direction of the vehicle; and
a processor configured to search for at least one planned driving lane on which the vehicle is planned to travel from the forward image and control the image output such that a main carpet image indicating the planned driving lane is output in lane units,
wherein the processor is further configured to:
control the image output unit to output the main carpet image overlapped with the planned driving lane,
extract a sign from the forward image and search for the planned driving lane based on at least one of a character or a symbol included in the sign,
acquire road information regarding planned travel of the vehicle,
search, based on the sign, for at least one of the character or the symbol corresponding to the road information,
specify the planned driving lane based on at least one of the character or the symbol included in the sign corresponding to the road information, and
change the main carpet image based on the specified planned driving lane.

### Claim 2 (depends on 1)

The image output device of claim 1, wherein the processor is configured to specify the planned driving lane based on route navigation information guiding a destination set in the vehicle.

### Claim 3 (depends on 2)

The image output device of claim 2, wherein the route navigation information includes a virtual image of the forward direction of the vehicle, and
wherein the processor is configured to extract a graphic object corresponding to the planned driving lane from the virtual image and generate the main carpet image based on the graphic object.

### Claim 4 (depends on 3)

The image output device of claim 3, wherein the processor is configured to:
determine a matching condition for matching the forward image and the virtual image; and
generate the main carpet image by changing the graphic object based on the matching condition.

### Claim 5 (depends on 4)

The image output device of claim 4, wherein the matching condition is a coordinate system transformation matrix for converting the virtual image into a coordinate system of the forward image.

### Claim 6 (depends on 3)

The image output device of claim 3, wherein the processor is configured to:
determine a degree of similarity between the virtual image and the forward image; and
control the image output unit such that the main carpet image is output from a point in time based on the degree of similarity being greater than a reference value.

### Claim 7 (depends on 2)

The image output device of claim 2, wherein the route navigation information is received from a navigation device through the communication unit, and
wherein the processor is configured to merge the main carpet image into the forward image and output the merged image through the image output unit.

### Claim 8 (depends on 1)

The image output device of claim 1, wherein the processor is configured to:
search for an object corresponding to a preset conditions; and
control, based on a preset user input being received, the image output unit such that a sub carpet image indicating a route to a destination corresponding to the object is output together with the main carpet image.

### Claim 9 (depends on 8)

The image output device of claim 8, wherein the processor is configured to, abased on a determination that plurality of objects corresponding to the preset condition are present, control the image output unit such that a plurality of sub carpet images are output together with the main carpet image.

### Claim 10 (depends on 9)

The image output device of claim 9, wherein the processor is configured to highlight one of the sub carpet images, and
wherein the one sub carpet image highlighted varies according to a user input.

### Claim 11 (depends on 10)

The image output device of claim 10, wherein the processor is configured to, based on a user selection of the one sub carpet image, control the communication unit such that a driving direction of the vehicle is changed to a route set in the one sub carpet image.

### Claim 12 (depends on 11)

The image output device of claim 11, wherein the processor is configured to, based on a user selection of the one sub carpet image, control the image output unit such that the main carpet image disappears.

### Claim 13

A method for controlling an image output device, the method comprising:
receiving a forward image that captures an image in a forward direction of a vehicle;
specifying at least one planned driving lane on which the vehicle is planned to travel based on route navigation information guiding a destination set in the vehicle;
searching for the planned driving lane from the forward image; and
outputting a main carpet image indicating the planed driving lane in lane units through an image output unit, wherein the outputting the main carpet image comprises outputting the main carpet image overlapped with the planned driving lane;
extracting a sign from the forward image and searching for the planned driving lane by using at least one of a character or a symbol included in the sign;
acquiring road information regarding planned travel of the vehicle;
searching, based on the sign, for at least one of the character or the symbol corresponding to the road information;
specifying the planned driving lane based on at least one of the character or the symbol included in the sign corresponding to the road information; and
changing the main carpet image based on the specified planned driving lane.

### Claim 14 (depends on 13)

The method of claim 13, wherein the route navigation information includes a virtual image of the forward direction of the vehicle,
wherein the outputting of the main carpet image through the image output unit includes:
extracting a graphic object corresponding to the planned driving lane from the virtual image; and
generating the main carpet image based on the graphic object.

### Claim 15 (depends on 14)

The method of claim 14, wherein the generating of the main carpet image using the graphic object includes:
determining a matching condition for matching the forward image and the virtual image; and
generating, based on the matching condition, the main carpet image by changing the graphic object.

### Claim 16 (depends on 13)

The method of claim 13, further comprising:
searching for an object corresponding to a preset condition; and
outputting, based on a preset user input, a sub carpet image indicating a route to a destination corresponding to the object together with the main carpet image.

### Claim 17 (depends on 16)

The method of claim 16, wherein a plurality of sub carpet images are output together with the main carpet image based on a determination that a plurality of objects corresponding to the preset condition are present.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
changing, based on a user selection of one of the plurality of sub carpet images, a driving direction of the vehicle to a route set in the one sub carpet image.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
controlling the image output unit such that the sub carpet image disappears based on a determination that the object is located out of a predetermined range with respect to the vehicle while the sub carpet image is being output.