# US 11,440,473 B2 · Driving assistance apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,440,473 B2
- Document ID: US11440473B2
- Status: Granted
- Kind: B2
- Application number: 16/663,638
- Filed: 2019-10-25
- Published: 2020-04-30
- Granted: 2022-09-13
- Earliest claimed: 2018-10-29
- Assignees: Aisin Corporation
- Applicants: Aisin Corporation
- Inventors: Shinichiro Murakami
- Web page: https://patentgrep.com/patent/US11440473B2

## CPC

- B60Q 9/00: Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling}
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 35/285: {for improving awareness by directing driver's gaze direction or eye points}
- B60K 35/65: Instruments specially adapted for specific vehicle types or users, e.g. for left- or right-hand drive
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06V 10/464: {using a plurality of salient features, e.g. bag-of-words [BoW] representations}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 40/193: {Preprocessing; Feature extraction}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/22: Display screens
- B60K 35/26: using acoustic output
- B60K 35/60: Instruments characterised by their location or relative disposition in or on vehicles (arrangements of lighting devices on dashboards B60Q3/10)
- B60K 2360/149: Instrument input by detecting viewing direction not otherwise provided for
- B60K 2360/178: Warnings
- B60K 2360/21: using cameras
- B60K 2360/741: Instruments adapted for user detection
- B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
- B60R 2011/0003: {inside the vehicle}
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR

## Abstract

A driving assistance apparatus includes a gaze detection portion detecting a gaze distribution of a driver for a vehicle, an image acquisition portion acquiring a captured image from an imaging device that captures an image in surroundings of the vehicle, a driver information acquisition portion acquiring driver information that allows identification of the driver for the vehicle, a generation portion generating a personalized saliency map based on the captured image and the driver information, the personalized saliency map that serves as a saliency map for the captured image and that differs depending on the driver, and a determination portion determining whether or not the driver looks at a visual confirmation target in the surroundings of the vehicle by comparing the gaze distribution detected by the gaze detection portion and the personalized saliency map generated by the generation portion.

## Claims

### Claim 1

A driving assistance apparatus comprising:
at least one processor configured to implement:
a gaze detection portion detecting a gaze distribution of a driver for a vehicle;
an image acquisition portion acquiring a captured image from an imaging device that captures an image in surroundings of the vehicle;
a driver information acquisition portion acquiring driver information that allows identification of the a driver from a plurality of potential drivers for the vehicle;
a generation portion generating a personalized saliency map based on the captured image and the driver information for each of the plurality of drivers, the personalized saliency map serves as a saliency map for the captured image and that differs for each driver of the plurality of drivers; and
a determination portion determining whether or not the driver looks at a visual confirmation target in the surroundings of the vehicle by comparing the gaze distribution detected by the gaze detection portion and the personalized saliency map generated by the generation portion for the driver.

### Claim 2 (depends on 1)

The driving assistance apparatus according to claim 1, wherein the generation portion learns a generative model by a generative deep neural network, the generative model being configured to generate the personalized saliency map based on the captured image and the driver information,
the generation portion generates the personalized saliency map by the generative model.

### Claim 3 (depends on 1)

The driving assistance apparatus according to claim 1, further comprising an alert portion alerting the driver to danger in a case where it is determined that the driver fails to look at the visual confirmation target.

### Claim 4 (depends on 3)

The driving assistance apparatus according to claim 3, wherein the driver information acquisition portion acquires the driver information including driving skill of the driver,
the generation portion generates the personalized saliency map that differs depending on the driving skill of the driver based on the captured image and the driver information, and
the determination portion determines the driving skill of the driver based on the gaze distribution detected by the gaze detection portion and the personalized saliency map generated by the generation portion for the driver.

### Claim 5 (depends on 4)

The driving assistance apparatus according to claim 4, wherein the alert portion changes a frequency of alerts based on a determination result of the driving skill of the driver determined by the determination portion.

### Claim 6 (depends on 4)

The driving assistance apparatus according to claim 4, wherein the alert portion differentiates the frequency of alerts between a case where the personalized saliency map for a first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other and a case where the personalized saliency map for a second driving skill higher than the first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other.

### Claim 7 (depends on 5)

The driving assistance apparatus according to claim 5, wherein the alert portion differentiates the frequency of alerts between a case where the personalized saliency map for a first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other and a case where the personalized saliency map for a second driving skill higher than the first driving skill and the gaze distribution detected by the gaze detection portion deviate from each other.

### Claim 8 (depends on 3)

The driving assistance apparatus according to claim 3, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

### Claim 9 (depends on 4)

The driving assistance apparatus according to claim 4, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

### Claim 10 (depends on 5)

The driving assistance apparatus according to claim 5, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

### Claim 11 (depends on 6)

The driving assistance apparatus according to claim 6, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.

### Claim 12 (depends on 7)

The driving assistance apparatus according to claim 7, wherein the generation portion generates a bottom-up saliency map based on the captured image and generates a top-down saliency map by removing the bottom-up saliency map from the personalized saliency map, the generation portion differentiating the frequency of alerts between a case where the gaze distribution detected by the gaze detection portion and the bottom-up saliency map deviate from each other and a case where the gaze distribution detected by the gaze detection portion and the top-down saliency map deviate from each other.