# US 2017/0286782 A1 · Generating Real-Time Driver Familiarity Index for Fine-Grained Dynamic Road Scenes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286782 A1
- Document ID: US20170286782A1
- Status: Published application
- Kind: A1
- Application number: 15/087,726
- Filed: 2016-03-31
- Published: 2017-10-05
- Earliest claimed: 2016-03-31
- Assignees: Toyota Jidosha Kabushiki Kaisha
- Applicants: Toyota Jidosha Kabushiki Kaisha
- Inventors: Preeti J. Pillai; Veeraganesh Yalla; Rahul Ravi Parundekar; Kentaro Oguchi
- Web page: https://patentgrep.com/patent/US20170286782A1

## CPC

- G01C 21/3691: {Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions (arrangements for giving variable traffic instructions G08G1/09)}
- B60W 40/08: related to drivers or passengers
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- G06F 18/23: Clustering techniques
- G06F 18/2431: Multiple classes
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In an example embodiment, a computer-implemented method is disclosed that generates a spectral signature describing one or more dynamic objects and a scene layout of a current road scene; identifies, from among one or more scene clusters included in a familiarity graph associated with a user, a road scene cluster corresponding to the current road scene; determine a position of the spectral signature relative to other spectral signatures comprising the identified road scene cluster; and generates a familiarity index estimating familiarity of the user with the current road scene based on the position of the spectral signature. The method can further include determining an assistance level based on the familiarity index of the user; and providing one or more of an auditory instruction, a visual instruction, and a tactile instruction to the user via one or more output devices of a vehicle at the determined assistance level.

## Claims

### Claim 1

A computer-implemented method comprising:
generating a spectral signature describing one or more dynamic objects and a scene layout of a current road scene;
identifying, from among one or more scene clusters included in a familiarity graph associated with a user, a road scene cluster corresponding to the current road scene;
determining a position of the spectral signature relative to other spectral signatures comprising the identified road scene cluster; and
generating a familiarity index estimating familiarity of the user with the current road scene based on the position of the spectral signature.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein generating the spectral signature describing the one or more dynamic objects and the scene layout of the current road scene includes:
capturing a road scene image of a roadway at a particular point in time using one or more sensors of a vehicle directed to the roadway; and
generating the spectral signature including a signature vector using the road scene image.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein identifying the road scene cluster corresponding to the current road scene includes:
categorizing the current road scene into a macro-scene category using the spectral signature; and
retrieving the road scene cluster including a plurality of historical spectral signatures associated with the macro-scene category, the plurality of historical spectral signatures including a plurality of historical signature vectors, respectively.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the position of the spectral signature is a position of the signature vector, and wherein determining the position of the spectral signature relative to the other spectral signatures includes:
determining the position of the signature vector relative to respective positions of the plurality of historical signature vectors of the road scene cluster.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the macro-scene category includes one of an urban category, a rural category, a residential category, a construction zone category, a highway category, and an accident category.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
updating the road scene cluster to include the spectral signature.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein generating the familiarity index estimating the familiarity of the user with the current road scene includes:
determining a set of nested familiarity zones for the road scene cluster based on a density of spectral signatures within the road scene cluster; and
determining the familiarity index of the user to the current road scene by determining the position of the spectral signature within the set of nested familiarity zones.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining an assistance level based on the familiarity index of the user; and
providing one or more of an auditory instruction, a visual instruction, and a tactile instruction to the user via one or more output devices of a vehicle at the determined assistance level.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the assistance level includes one or more of 1) a frequency for providing the one or more of the auditory instruction, the visual instruction, and the tactile instruction to the user, and 2) a level of detail of the one or more of the auditory instruction, the visual instruction, and the tactile instruction.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the current road scene is depicted in a road scene image being captured at a particular point in time, the method further comprises:
determining physiological data associated with the user at the particular point in time;
generating a physiological weight value based on the physiological data; and
augmenting the spectral signature using the physiological weight value.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the physiological data associated with the user includes one or more of head movement, eye movement, electrocardiography data, respiration data, skin conductance, and muscle tension of the user at the particular point in time.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining historical navigation data associated with the user;
generating a historical navigation weight value based on the historical navigation data; and
augmenting the spectral signature using the historical navigation weight value.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the historical navigation data associated with the user includes one or more of a road name, a road type, road speed information, and time information associated with one or more road segments previously travelled by the user.

### Claim 14

A system comprising:
one or more processors;
one or more memories storing instructions that, when executed by the one or more processors, cause the system to:
generate a spectral signature describing one or more dynamic objects and a scene layout of a current road scene;
identify, from among one or more scene clusters included in a familiarity graph associated with a user, a road scene cluster corresponding to the current road scene;
determine a position of the spectral signature relative to other spectral signatures comprising the identified road scene cluster; and
generate a familiarity index estimating familiarity of the user with the current road scene based on the position of the spectral signature.

### Claim 15 (depends on 14)

The system of claim 14, wherein to generate the spectral signature describing the one or more dynamic objects and the scene layout of the current road scene includes:
capturing a road scene image of a roadway at a particular point in time using one or more sensors of a vehicle directed to the roadway; and
generating the spectral signature including a signature vector using the road scene image.

### Claim 16 (depends on 15)

The system of claim 15, wherein to identify the road scene cluster corresponding to the current road scene includes:
categorizing the current road scene into a macro-scene category using the spectral signature; and
retrieving the road scene cluster including a plurality of historical spectral signatures associated with the macro-scene category, the plurality of historical spectral signatures including a plurality of historical signature vectors, respectively.

### Claim 17 (depends on 16)

The system of claim 16, wherein the position of the spectral signature is a position of the signature vector, and wherein to determine the position of the spectral signature relative to the other spectral signatures includes:
determining the position of the signature vector relative to respective positions of the plurality of historical signature vectors of the road scene cluster.

### Claim 18 (depends on 16)

The system of claim 16, wherein the macro-scene category includes one of an urban category, a rural category, a residential category, a construction zone category, a highway category, and an accident category.

### Claim 19 (depends on 14)

The system of claim 14, wherein the instructions, when executed by the one or more processors, further cause the system to:
update the road scene cluster to include the spectral signature.

### Claim 20 (depends on 14)

The system of claim 14, wherein to generate the familiarity index estimating the familiarity of the user with the current road scene includes:
determining a set of nested familiarity zones for the road scene cluster based on a density of spectral signatures within the road scene cluster; and
determining the familiarity index of the user to the current road scene by determining the position of the spectral signature within the set of nested familiarity zones.

### Claim 21 (depends on 14)

The system of claim 14, wherein the instructions, when executed by the one or more processors, further cause the system to:
determine an assistance level based on the familiarity index of the user; and
provide one or more of an auditory instruction, a visual instruction, and a tactile instruction to the user via one or more output devices of a vehicle at the determined assistance level.

### Claim 22 (depends on 21)

The system of claim 21, wherein the assistance level includes one or more of 1) a frequency for providing the one or more of the auditory instruction, the visual instruction, and the tactile instruction to the user, and 2) a level of detail of the one or more of the auditory instruction, the visual instruction, and the tactile instruction.

### Claim 23 (depends on 14)

The system of claim 14, wherein the current road scene is depicted in a road scene image being captured at a particular point in time, and wherein the instructions, when executed by the one or more processors, further cause the system to:
determine physiological data associated with the user at the particular point in time;
generate a physiological weight value based on the physiological data; and
augment the spectral signature using the physiological weight value.

### Claim 24 (depends on 23)

The system of claim 23, wherein the physiological data associated with the user includes one or more of head movement, eye movement, electrocardiography data, respiration data, skin conductance, and muscle tension of the user at the particular point in time.

### Claim 25 (depends on 14)

The system of claim 14, wherein the instructions, when executed by the one or more processors, further cause the system to:
determine historical navigation data associated with the user;
generate a historical navigation weight value based on the historical navigation data; and
augment the spectral signature using the historical navigation weight value.

### Claim 26 (depends on 25)

The system of claim 25, wherein the historical navigation data associated with the user includes one or more of a road name, a road type, road speed information, and time information associated with one or more road segments previously travelled by the user.
