# US 2022/0277165 A1 · System and method for improving driver situation awareness prediction using human visual sensory and memory mechanism

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277165 A1
- Document ID: US20220277165A1
- Status: Published application
- Kind: A1
- Application number: 17/324,453
- Filed: 2021-05-19
- Published: 2022-09-01
- Earliest claimed: 2021-02-26
- Assignees: Honda Motor Co., Ltd.
- Applicants: Honda Motor Co., Ltd.
- Inventors: Haibei ZHU; Teruhisa MISU; Sujitha Catherine MARTIN; Xingwei WU; Kumar AKASH
- Web page: https://patentgrep.com/patent/US20220277165A1

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- B60R 16/0231: {Circuits relating to the driving or the functioning of the vehicle (monitoring tyres B60C23/00; indicating overspeed B60K31/00; for dash boards B60K37/00, B60Q3/10; for indicating emergencies B60Q1/52; brake control systems B60T; registering or indicating the working of vehicles G07C5/00; measuring distance G01C, e.g. combinations of speed and distance G01C23/00; engine indicators G01L; measuring speed or acceleration G01P)}
- G06F 18/24: Classification techniques
- G06F 18/251: {of input or preprocessed data}
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 40/193: {Preprocessing; Feature extraction}
- G08G 1/096725: {where the received information generates an automatic action on the vehicle control}
- B60Q 9/00: Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling}
- B60Q 9/008: {for anti-collision purposes}
- B60W 40/09: Driving style or behaviour
- B60W 2540/225: Direction of gaze
- G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G08G 1/0962: having an indicator mounted inside the vehicle, e.g. giving voice messages
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

A system and method for improving driver situation awareness prediction using human visual sensory and memory mechanism that includes receiving data associated with a driving scene of a vehicle and an eye gaze of a driver of the vehicle. The system and method also include analyzing the data and extracting features associated with objects located within the driving scene and determining a situational awareness score that is associated a situational awareness of the driver with respect to each of the objects located within the driving scene. The system and method further include communicating control signals to electronically control at least one system of the vehicle based on the situational awareness score that is associated with each of the objects located within the driving scene.

## Claims

### Claim 1

A computer-implemented method for improving driver situation awareness prediction using human visual sensory and memory mechanism comprising:
receiving data associated with a driving scene of a vehicle and an eye gaze of a driver of the vehicle;
analyzing the data and extracting features associated with objects located within the driving scene;
determining a situational awareness score that is associated a situational awareness of the driver with respect to each of the objects located within the driving scene; and
communicating control signals to electronically control at least one system of the vehicle based on the situational awareness score that is associated with each of the objects located within the driving scene.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein receiving the data associated with the driving scene of the vehicle includes receiving image data associated with images of the objects located within the driving scene of the vehicle and LiDAR data associated with LiDAR detection of the objects located within the driving scene of the vehicle.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further including aggregating the image data and the LiDAR data into aggregated driving scene data that is associated with the driving scene, wherein the aggregated driving scene data is analyzed with respect to a classifier to classify the objects into dynamic objects and static objects.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract gaze point-based features, wherein the gaze point-based features utilize a positional relationship between target objects and participants' gaze center coordinates.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract human visual sensory dependent features associated with non-foveal vision for tracking detected objects located within the driving scene.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract object spatial-based features that consider properties of each object that is located within the driving scene, wherein the object spatial-based features include a difference in each object's position information in the driving scene which impacts the awareness of the driver with respect to each object within an analysis time window.

### Claim 7 (depends on 3)

The computer-implemented method of claim 3, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract object property-based features that are associated with each of the objects that are located within the driving scene of the vehicle, wherein the object property-based features consider each object's visual salience from the driver's point of view.

### Claim 8 (depends on 3)

The computer-implemented method of claim 3, wherein determining the situational awareness score includes analyzing a short-term memory of the driver as indicator of the situational awareness with respect to each object, wherein the short-term memory is associated with a memory of salient objects versus non-salient objects within the driving scene and a memory score is calculated for each object.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein determining the situational awareness score includes weighing and aggregating gaze point-based features, human visual sensory dependent features, object spatial-based features, object property-based features, and human short-term memory-based features to calculate the situational awareness score for each of the objects, wherein each of the objects is ranked based on the situational awareness score for each of the objects.

### Claim 10

A system for improving driver situation awareness prediction using human visual sensory and memory mechanism comprising:
a memory storing instructions when executed by a processor cause the processor to:
receive data associated with a driving scene of a vehicle and an eye gaze of a driver of the vehicle;
analyze the data and extracting features associated with objects located within the driving scene;
determine a situational awareness score that is associated a situational awareness of the driver with respect to each of the objects located within the driving scene; and
communicate control signals to electronically control at least one system of the vehicle based on the situational awareness score that is associated with each of the objects located within the driving scene.

### Claim 11 (depends on 10)

The system of claim 10, wherein receiving the data associated with the driving scene of the vehicle includes receiving image data associated with images of the objects located within the driving scene of the vehicle and LiDAR data associated with LiDAR detection of the objects located within the driving scene of the vehicle.

### Claim 12 (depends on 11)

The system of claim 11, further including aggregating the image data and the LiDAR data into aggregated driving scene data that is associated with the driving scene, wherein the aggregated driving scene data is analyzed with respect to a classifier to classify the objects into dynamic objects and static objects.

### Claim 13 (depends on 12)

The system of claim 12, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract gaze point-based features, wherein the gaze point-based features utilize a positional relationship between target objects and participants' gaze center coordinates.

### Claim 14 (depends on 12)

The system of claim 12, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract human visual sensory dependent features associated with non-foveal vision for tracking detected objects located within the driving scene.

### Claim 15 (depends on 12)

The system of claim 12, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract object spatial-based features that consider properties of each object that is located within the driving scene, wherein the object spatial-based features include a difference in each object's position information in the driving scene which impacts the awareness of the driver with respect to each object within an analysis time window.

### Claim 16 (depends on 12)

The system of claim 12, wherein analyzing the data and extracting features associated with objects includes analyzing determined gaze point values that are associated with the eye gaze of the driver towards the driving scene to extract object property-based features that are associated with each of the objects that are located within the driving scene of the vehicle, wherein the object property-based features consider each object's visual salience from the driver's point of view.

### Claim 17 (depends on 12)

The system of claim 12, wherein determining the situational awareness score includes determining analyze a short-term memory of the driver as indicator of the situational awareness with respect to each object, wherein the short-term memory is associated with a memory of salient objects versus non-salient objects within the driving scene and a memory score is calculated for each object.

### Claim 18 (depends on 10)

The system of claim 10, wherein determining the situational awareness score includes weighing and aggregating gaze point-based features, human visual sensory dependent features, object spatial-based features, object property-based features, and human short-term memory-based features to calculate the situational awareness score for each of the objects, wherein each of the objects is ranked based on the situational awareness score for each of the objects.

### Claim 19

A non-transitory computer readable storage medium storing instructions that when executed by a computer, which includes a processor perform a method, the method comprising:
receiving data associated with a driving scene of a vehicle and an eye gaze of a driver of the vehicle;
analyzing the data and extracting features associated with objects located within the driving scene;
determining a situational awareness score that is associated a situational awareness of the driver with respect to each of the objects located within the driving scene; and
communicating control signals to electronically control at least one system of the vehicle based on the situational awareness score that is associated with each of the objects located within the driving scene.

### Claim 20 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein determining the situational awareness score includes weighing and aggregating gaze point-based features, human visual sensory dependent features, object spatial-based features, object property-based features, and human short-term memory-based features to calculate the situational awareness score for each of the objects, wherein each of the objects is ranked based on the situational awareness score for each of the objects.
