# US 2024/0199082 A1 · Attention-based agent interaction system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0199082 A1
- Document ID: US20240199082A1
- Status: Published application
- Kind: A1
- Application number: 18/082,495
- Filed: 2022-12-15
- Published: 2024-06-20
- Earliest claimed: 2022-12-15
- Assignees: Toyota Research Institute, Inc.; Toyota Jidosha Kabushiki Kaisha
- Applicants: Toyota Research Institute, Inc.
- Inventors: Thomas BALCH; Paul DREWS; Xiongyi CUI; Stephen G. MCGILL; Guy ROSMAN
- Web page: https://patentgrep.com/patent/US20240199082A1

## CPC

- B60W 60/0027: {using trajectory prediction for other traffic participants}
- B60W 60/0055: {only part of driving tasks shifted to occupants}
- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- B60W 30/095: Predicting travel path or likelihood of collision
- B60W 50/00: Details of control systems for road vehicle drive control not related to the control of a particular sub-unit {, e.g. process diagnostic or vehicle driver interfaces}
- B60W 50/08: Interaction between the driver and the control system
- B60W 2040/0818: {Inactivity or incapacity of driver}
- B60W 2040/0872: {Driver physiology}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2540/225: Direction of gaze
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

A method for resource allocation of an advanced driver assistance system (ADAS) is described. The method includes dynamically tracking gaze directions of a vehicle operator regarding a scene surrounding an ego vehicle. The method also includes identifying operator monitored regions and operator unmonitored regions in the scene based on dynamically tracking the gaze directions of the vehicle operator. The method further includes allocating an increased portion of ADAS perception resources to the operator unmonitored regions of the scene. The method also includes tracking external road agents detected in the operator unmonitored regions of the scene using the increased portion of ADAS perception resources.

## Claims

### Claim 1

A method for resource allocation of an advanced driver assistance system (ADAS), the method comprising:
dynamically tracking gaze directions of a vehicle operator regarding a scene surrounding an ego vehicle;
identifying operator monitored regions and operator unmonitored regions in the scene based on dynamically tracking the gaze directions of the vehicle operator;
allocating an increased portion of ADAS perception resources to the operator unmonitored regions of the scene; and
tracking external road agents detected in the operator unmonitored regions of the scene using the increased portion of ADAS perception resources.

### Claim 2 (depends on 1)

The method of claim 1, in which dynamically tracking comprises visualizing a driver's gaze-direction behavior using a driver's attention heatmap, indicating where and how often the vehicle operator is focusing their gaze.

### Claim 3 (depends on 1)

The method of claim 1, in which dynamically tracking comprises dynamically determining the gaze direction of the vehicle operator based on sensor data captured by a driver facing camera to monitor the vehicle operator.

### Claim 4 (depends on 1)

The method of claim 1, further comprising allocating a reduced portion of the ADAS perception resources to the operator monitored regions of the scene.

### Claim 5 (depends on 1)

The method of claim 1, in which allocating the increased portion of ADAS perception resources comprises assigning increased external-road-agent predictor resources to track external road agents in the unmonitored regions of the scene.

### Claim 6 (depends on 5)

The method of claim 5, in which the increased external-road-agent predictor resources comprises an increased model complexity and/or a number of samples.

### Claim 7 (depends on 1)

The method of claim 1, in which tracking external road agents comprises tracking autonomous dynamic objects (ADOs) identified in the operator unmonitored regions of the scene using the increased portion of ADAS perception resources.

### Claim 8 (depends on 1)

The method of claim 1, further comprising controlling the ego vehicle to avoid a collision with an external road agent detected in the operator unmonitored regions of the scene.

### Claim 9

A non-transitory computer-readable medium having program code recorded thereon for resource allocation of an advanced driver assistance system (ADAS), the program code being executed by a processor and comprising:
program code to dynamically track gaze directions of a vehicle operator regarding a scene surrounding an ego vehicle;
program code to identify operator monitored regions and operator unmonitored regions in the scene based on dynamically tracking the gaze directions of the vehicle operator;
program code to allocate an increased portion of ADAS perception resources to the operator unmonitored regions of the scene; and
program code to track external road agents detected in the operator unmonitored regions of the scene using the increased portion of ADAS perception resources.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the program code to dynamically track comprises program code to visualize a driver's gaze-direction behavior using a driver's attention heatmap, indicating where and how often the vehicle operator is focusing their gaze.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the program code to dynamically track comprises program code to dynamically determine the gaze direction of the vehicle operator based on sensor data captured by a driver facing camera to monitor the vehicle operator.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, further comprising program code to allocate a reduced portion of the ADAS perception resources to the operator monitored regions of the scene.

### Claim 13 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the program code to allocate the increased portion of ADAS perception resources comprises program code to assign increased external-road-agent predictor resources to track external road agents in the unmonitored regions of the scene.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, in which the increased external-road-agent predictor resources comprise an increased model complexity and/or a number of samples.

### Claim 15 (depends on 9)

The non-transitory computer-readable medium of claim 9, in which the program code to track external road agents comprises program code to track autonomous dynamic objects (ADOs) identified in the operator unmonitored regions of the scene using the increased portion of ADAS perception resources.

### Claim 16 (depends on 9)

The non-transitory computer-readable medium of claim 9, further comprising program code to control the ego vehicle to avoid a collision with an external road agent detected in the operator unmonitored regions of the scene.

### Claim 17

A system for resource allocation of an advanced driver assistance system (ADAS), the system comprising:
a gaze tracking module to dynamically track gaze directions of a vehicle operator regarding a scene surrounding an ego vehicle;
an unmonitored region(s) detection module to identify operator monitored regions and operator unmonitored regions in the scene based on dynamically tracking the gaze directions of the vehicle operator;
an ADAS resource allocation module to allocate an increased portion of ADAS perception resources to the operator unmonitored regions of the scene; and
an external environment perception module to track external road agents detected in the operator unmonitored regions of the scene using the increased portion of ADAS perception resources.

### Claim 18 (depends on 17)

The system of claim 17, in which the gaze tracking module is further to visualize a driver's gaze-direction behavior using a driver's attention heatmap, indicating where and how often the vehicle operator is focusing their gaze.

### Claim 19 (depends on 17)

The system of claim 17, in which the gaze tracking module is further to dynamically determine the gaze direction of the vehicle operator based on sensor data captured by a driver facing camera to monitor the vehicle operator.

### Claim 20 (depends on 17)

The system of claim 17, further comprising a controller to control the ego vehicle to avoid a collision with an external road agent detected in the operator unmonitored regions of the scene.
