# US 11,908,208 B2 · Interface sharpness distraction mitigation method and system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,208 B2
- Document ID: US11908208B2
- Status: Granted
- Kind: B2
- Application number: 17/074,727
- Filed: 2020-10-20
- Published: 2022-04-21
- Granted: 2024-02-20
- Earliest claimed: 2020-10-20
- Assignees: Toyota Motor Engineering & Manufacturing North America, Inc.
- Applicants: Toyota Motor Engineering & Manufacturing North America, Inc.
- Inventors: Benjamin P. Austin; George M. Evans; Joshua E. Domeyer; John K Lenneman
- Web page: https://patentgrep.com/patent/US11908208B2

## CPC

- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- B60W 40/09: Driving style or behaviour
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G06V 40/193: {Preprocessing; Feature extraction}
- B60W 2540/225: Direction of gaze
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

A driver monitor system and method of optimizing use of cognitive resources of a user in a vehicle is disclosed. The system includes in-cabin video cameras, a display device, and processing circuitry. The method performed by the processing circuitry includes monitoring eye gaze direction using images from the in-cabin video cameras, gradually softening image regions and lines displayed in the display device that are outside a first vision area of the user of the vehicle, and using the eye gaze direction to predict whether the user is transitioning into a new focus vision area that had been outside the first vision area. When the user's eyes begin to move, begin sharpening softened image regions and lines such that by the time eyes have shifted into the new focus vision area, the image regions and lines in the new focus vision area reach full sharpness.

## Claims

### Claim 1

A driver monitor system for optimizing use of cognitive resources of a user in a vehicle, the system comprising:
at least one in-cabin video camera;
a dashboard-mounted display device; and
processing circuitry configured to:
monitor eye gaze direction using images from the at least one in-cabin video camera,
gradually soften image regions and lines displayed in the display device that are outside a first peripheral view of the user of the vehicle,
use the eye gaze direction to predict whether the user is transitioning into a new focus vision area that had been outside the first peripheral view,
when the user's eyes begin to move, begin sharpening softened image regions and lines such that by the time eyes have shifted into the new focus vision area, the image regions and lines in the new focus vision area reach full sharpness, and
display increasing levels of softening of image regions and lines displayed in the display device as the distance increases away from the first peripheral view, wherein the gradually softening is by gradually adjusting color intensity of inner pixels based on an average color intensity of a relatively large pixel radius, next further pixels away from the inner pixels based on a slightly smaller pixel radius, down to final edge pixels based on a smallest pixel radius,
wherein the peripheral view is a peripheral view of each eye of 15 degrees, and
wherein the processing circuitry is further configured to display increasing levels of softening of image regions and lines displayed in the display device as the distance increases away from the peripheral view of 15 degrees.

### Claim 2 (depends on 1)

The driver monitor system of claim 1, further including a machine learning device and an audio-video device,
wherein the audio-video device outputs a verification request and receives a response to the verification request, and
wherein the eye gaze direction is fed back to the machine learning device which learns to predict whether the user is transitioning into the new focus vision area that had been outside the first peripheral view.

### Claim 3 (depends on 2)

The driver monitor system of claim 2, wherein parameters of the machine learning device that are learned are stored in a memory as a profile associated with the user.

### Claim 4 (depends on 3)

The driver monitor system of claim 3, wherein independent profiles are stored in the memory in association with respective different users.

### Claim 5 (depends on 1)

The driver monitor system of claim 1, further comprising at least one exterior sensor for monitoring driving conditions,
wherein the processing circuitry is further configured to
perform image region and line softening and sharpening of a softened image when information based on the at least one external sensor indicates dangerous driving conditions.

### Claim 6 (depends on 2)

The driver monitor system of claim 2, wherein the processing circuitry further monitors head movement,
wherein the head movement is fed back to the machine learning device which learns to predict whether the user is transitioning into the new focus vision area that had been outside the first peripheral view.

### Claim 7 (depends on 2)

The driver monitor system of claim 2, wherein the machine learning device learns by performing a reinforcement learning algorithm.

### Claim 8 (depends on 2)

The driver monitor system of claim 2, further comprising at least one exterior sensor,
wherein the processing circuitry uses data from the at least one external sensor to monitor the position of the vehicle relative to the road, and
wherein the position of the vehicle relative to the road is fed back to the machine learning device which learns to predict whether the driver is transitioning into the new focus vision area that had been outside the peripheral view when the position of the vehicle relative to the road is veering out of lane.

### Claim 9 (depends on 1)

The driver monitor system of claim 1, wherein the processing circuitry is further configured to
without delay perform the sharpening of a softened image region or line in an image in anticipation of the user shifting to a new focus vision area in the image, and perform the gradual softening of image regions and lines displayed in the display device over a delayed time period as the user shifts focus away from the focus vision area of the image.

### Claim 10 (depends on 1)

The driver monitor system of claim 1, wherein the processing circuitry is further configured to
perform an abrupt change of an image region or line in an image from softened to sharp in order to draw attention.

### Claim 11 (depends on 1)

The driver monitor system of claim 1, wherein the processing circuitry is further configured to
detect a passenger's eye focus to be to a point in the softened image regions and lines and sharpen the image regions and lines of an image within a range of focus vision area of the passenger, and maintain softened image regions and lines for remaining sections of an image so as not to draw attention to the user.

### Claim 12 (depends on 1)

The driver monitor system of claim 1, wherein the processing circuitry is further configured to
determine whether sharpening is allowed and prevent sharpening of softened image regions and lines when sharpening is not allowed.

### Claim 13 (depends on 5)

The driver monitor system of claim 5,
wherein the processing circuitry is configured to use data from the at least one external sensor to monitor driving conditions, and
wherein the processing circuitry displays a sharp warning and/or notification in place of the softened image regions and lines.

### Claim 14 (depends on 1)

The driver monitor system of claim 1,
wherein the processing circuitry is further configured to maintain certain items in the peripheral view in a sharp state, while other items in the peripheral view are allowed to change from a sharpened state to a softened state.

### Claim 15 (depends on 1)

The driver monitor system of claim 1,
wherein the image regions and lines that have been sharpened by the processing circuitry when the user's eyes shift to a new focus point revert back to softened image regions and lines after a predetermined period of time.

### Claim 16

A method of optimizing use of cognitive resources of a user in a vehicle with a driver monitor system, the system including at least one in-cabin video camera, a dashboard-mounted display device, and processing circuitry, the method comprising:
monitoring, by the processing circuitry, eye gaze direction using images from the at least one in-cabin video camera;
gradually softening image regions and lines displayed in the display device that are outside a first peripheral view of the user of the vehicle;
using the eye gaze direction to predict whether the user is transitioning into a new focus vision area that had been outside the first peripheral view;
when the user's eyes begin to move, begin sharpening softened image regions and lines such that by the time eyes have shifted into the new focus vision area, the image regions and lines in the new focus vision area reach full sharpness, and
displaying increasing levels of softening of image regions and lines displayed in the display device as the distance increases away from the peripheral view, wherein the gradually softening is by gradually adjusting color intensity of inner pixels based on an average color intensity of a relatively large pixel radius, next further pixels away from the inner pixels based on a slightly smaller pixel radius, down to final edge pixels based on a smallest pixel radius.

### Claim 17 (depends on 16)

The method of claim 16, wherein the system further includes a machine learning device and an audio-video device, the method further comprising:
outputting, by the audio-video device, a verification request and receiving a response to the verification request; and
feeding back the eye gaze direction to the machine learning device which learns to predict whether the user is transitioning into the new focus vision area that had been outside the first peripheral view.

### Claim 18 (depends on 17)

The method of claim 17, wherein the system further includes at least one exterior sensor for monitoring driving conditions,
the method further comprising:
performing, by the processing circuitry, image region and line softening and sharpening of a softened image when information based on the at least one external sensor indicates dangerous driving conditions.

### Claim 19 (depends on 16)

The method of claim 16, further comprising:
without delay performing the sharpening of a softened image region or line in an image in anticipation of the user shifting to a new focus vision area of the image, and performing the gradual softening of image regions and lines displayed in the display device over a delayed time period as the user shifts focus away from the focus vision area in the image.