# US 2020/0394906 A1 · Apparatus for assisting driving of a host vehicle based on augmented reality and method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394906 A1
- Document ID: US20200394906A1
- Status: Published application
- Kind: A1
- Application number: 16/899,496
- Filed: 2020-06-11
- Published: 2020-12-17
- Earliest claimed: 2019-06-13
- Assignees: Mando Corporation
- Applicants: Mando Corporation
- Inventors: Byung-Joo KIM
- Web page: https://patentgrep.com/patent/US20200394906A1

## CPC

- H04W 4/40: for vehicles, e.g. vehicle-to-pedestrians [V2P]
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 11/65: on geographic maps
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]}
- G08G 1/0141: {for traffic information dissemination}
- G08G 1/096775: {where the origin of the information is a central station}
- G06V 10/16: {using multiple overlapping images; Image stitching}
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

An apparatus for assisting driving of a host vehicle based on augmented reality and a method thereof are provided. The apparatus for assisting driving of a host vehicle based on augmented reality includes an image sensor configured to capture an image of surroundings of the host vehicle, and a controller communicatively connected to the image sensor. The controller is configured to transmit the captured image to a cloud server through a wireless communicator, determine whether information about the captured image is lost due to the other vehicle neighboring the host vehicle, receive a panoramic image of a corresponding location from the cloud server through the wireless communicator if the information of the captured image is lost, process the received panoramic image in combination with the image, the information of which is lost, so as to generate an augmented-reality image, and perform autonomous driving according to the augmented-reality image or display the augmented-reality image on a display.

## Claims

### Claim 1

An apparatus for assisting driving of a host vehicle based on augmented reality, the apparatus comprising:
an image sensor configured to capture an image of surroundings of the host vehicle; and
a controller communicatively connected to the image sensor and configured to:
transmit the captured image to a cloud server through a wireless communicator;
determine whether information about the captured image is lost due to another vehicle neighboring the host vehicle;
receive a panoramic image of a corresponding location from the cloud server through the wireless communicator, if the information of the captured image is lost;
process the received panoramic image in combination with the image, the information of which is lost, so as to generate an augmented-reality image; and
perform autonomous driving according to the augmented-reality image or display the augmented-reality image on a display.

### Claim 2 (depends on 1)

The apparatus of claim 1, further comprising a position sensor configured to detect a position of an object near the host vehicle, and
wherein the controller is further configured to:
transmit detected position data to the cloud server through the wireless communicator;
determine whether information of the detected position data is lost due to the other vehicle; and
request the cloud server to provide the panoramic image of the location through the wireless communicator, if the information of the detected position data is lost.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the position sensor comprises at least one of a camera sensor, a LiDAR sensor, an ultrasonic sensor, or a radar sensor.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the image sensor comprises at least one of a camera sensor or a LiDAR sensor.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the controller is further configured to:
determine whether a current condition is night or bad weather based on the captured image; and
request the cloud server to provide the panoramic image of the location through the wireless communicator, if the current condition is night or bad weather.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the controller is further configured to generate the augmented-reality image such that the image, the information of which is lost, is combined in a perspective manner based on the panoramic image.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the panoramic image is based on images currently transmitted from the host vehicle and the other vehicle.

### Claim 8 (depends on 5)

The apparatus of claim 5, wherein the panoramic image is based on images transmitted from vehicles passing the location during daytime or on an ordinary day.

### Claim 9

A method of assisting driving of a host vehicle based on augmented reality, the method comprising:
capturing an image of surroundings of a host vehicle by an image sensor;
transmitting the captured image to a cloud server through a wireless communicator by using a controller communicatively connected to the image sensor;
determining, by the controller, whether information about the captured image is lost due to another vehicle neighboring the host vehicle;
receiving, by the controller, a panoramic image of a corresponding location from the cloud server through the wireless communicator, if the information about the captured image is lost;
processing, by the controller, the received panoramic image in combination with the image, the information of which is lost, so as to generate an augmented-reality image; and
performing, by the controller, autonomous driving according to the augmented-reality image or display the augmented-reality image on a display.

### Claim 10 (depends on 9)

The method of claim 9, further comprising detecting a position of an object near the host vehicle by a position sensor, and
wherein the transmitting of the captured image comprises transmitting detected position data to the cloud server,
the determining of whether the information about the captured image is lost comprises determining whether information of the detected position data is lost due to the other vehicle, and
if the information of the detected position data is lost, the receiving of the panoramic image comprises requesting the cloud server to provide the panoramic image of the location.

### Claim 11 (depends on 10)

The method of claim 10, wherein the position sensor comprises at least one of a camera sensor, a LiDAR sensor, an ultrasonic sensor, or a radar sensor.

### Claim 12 (depends on 9)

The method of claim 9, wherein the image sensor comprises at least one of a camera sensor or a LiDAR sensor.

### Claim 13 (depends on 9)

The method of claim 9, further comprising:
determining, by the controller, whether a current condition is night or bad weather based on the captured image; and
requesting, by the controller, the cloud server to provide the panoramic image of the location through the wireless communicator, if the current condition is night or bad weather.

### Claim 14 (depends on 9)

The method of claim 9, wherein the processing of the received panoramic image comprises generating the augmented-reality image such that the image, the information of which is lost, is combined in a perspective manner based on the panoramic image.

### Claim 15 (depends on 9)

The method of claim 9, wherein the panoramic image is based on images currently transmitted from the host vehicle and the other vehicle.

### Claim 16 (depends on 13)

The method of claim 13, wherein the panoramic image is based on images transmitted from vehicles passing the location during daytime or on an ordinary day.

### Claim 17

A non-transitory computer-readable medium storing computer-executable instructions, when executed by a processor, to perform operations of:
capturing an image of surroundings of a host vehicle by an image sensor;
transmitting the captured image to a cloud server through a wireless communicator;
determining whether information about the captured image is lost due to another vehicle neighboring the host vehicle;
receiving a panoramic image of a corresponding location from the cloud server through the wireless communicator, if the information about the captured image is lost;
processing the received panoramic image in combination with the image, the information of which is lost, so as to generate an augmented-reality image; and
performing autonomous driving according to the augmented-reality image or displaying the augmented-reality image on a display.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the operations further comprise detecting a position of an object near the host vehicle by a position sensor;
the operation of transmitting the captured image comprises transmitting detected position data to the cloud server,
the operation of determining of whether the information about the captured image is lost comprises determining whether information of the detected position data is lost due to the other vehicle, and
if the information of the detected position data is lost, the operation of receiving of the panoramic image comprises requesting the cloud server to provide the panoramic image of the location.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the operations further comprise:
determining whether a current condition is night or bad weather based on the captured image; and
requesting the cloud server to provide the panoramic image of the location through the wireless communicator, if the current condition is night or bad weather.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the operation of processing the received panoramic image comprises generating the augmented-reality image such that the image, the information of which is lost, is combined in a perspective manner based on the panoramic image.
