# US 10,869,627 B2 · System and method for fusing information related to a driver of a vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,869,627 B2
- Document ID: US10869627B2
- Status: Granted
- Kind: B2
- Application number: 15/642,323
- Filed: 2017-07-05
- Published: 2019-01-10
- Granted: 2020-12-22
- Earliest claimed: 2017-07-05
- Assignees: Osr Enterprises AG
- Applicants: OSR Enterprises AG
- Inventors: Yosef Ben-Ezra; Samuel Hazak; Yaniv Ben-Haim; Shai Nissim; Yoni Schiff
- Web page: https://patentgrep.com/patent/US10869627B2

## CPC

- A61B 5/18: for vehicle drivers {or machine operators}
- A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet}
- A61B 5/0075: {by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy (A61B5/0071 takes precedence)}
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/14551: {for measuring blood gases}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/7267: {involving training the classification device}
- A61B 5/7405: {using sound}
- A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image}
- A61B 5/746: {Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms}
- B60W 50/0098: {Details of control systems ensuring comfort, safety or stability not otherwise provided for}
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G06F 18/251: {of input or preprocessed data}
- G06Q 40/08: Insurance
- G06V 10/143: Sensing or illuminating at different wavelengths
- G06V 10/803: {of input or preprocessed data}
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G08B 21/06: indicating a condition of sleep, e.g. anti-dozing alarms
- G08B 21/182: {Level alarms, e.g. alarms responsive to variables exceeding a threshold}
- A61B 2503/22: Motor vehicles operators, e.g. drivers, pilots, captains
- A61B 2560/0493: controlled by voice
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2540/043: Identity of occupants
- B60W 2540/21: Voice
- B60W 2540/22: Psychological state; Stress level or workload
- B60W 2555/20: Ambient conditions, e.g. wind or rain
- B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data
- G06V 40/166: {using acquisition arrangements}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

A method, apparatus and computer program product of determining a state of a vehicle driver, the method comprising: receiving an image of the driver captured by a hyper spectral camera capable of imaging body features invisible to a human; receiving telemetry information from a car telemetry system; analyzing the image to receive at least one indicator to a clinical parameter of the driver; and fusing the at least one indicator with the telemetry information to obtain an assessment to a stress level of the driver.

## Claims

### Claim 1

A method of determining a state of a vehicle driver, to be performed by a device comprising a processor and a memory device, the method comprising:
receiving by a vehicle telemetry interface, vehicle-related telemetry information from a car telemetry system, wherein the vehicle-related telemetry information comprises at least one of vehicle speed, vehicle acceleration, lane passing and honking;
assessing by a telemetric analysis engine, a stress level of the driver from the information received from the car telemetry system;
receiving by an information fusing apparatus, an image of the driver captured by a hyper spectral camera capable of imaging body features invisible to a human;
analyzing the image by an image analysis engine, to receive at least one indicator to a clinical parameter of the driver;
learning by a learning engine, stress-related parameters or thresholds from a behavior of the driver over time;
fusing by a stress level analysis engine, the stress level assessed from the vehicle-related telemetry information received from the car telemetry system, with the at least one indicator received from analyzing the image, and using the stress-related parameters or thresholds as learned to obtain a combined assessment to a stress level of the driver;
based on the combined assessment, determining an action by an action selection engine; and
taking the action by an action activation engine, in response to the stress level exceeding a predetermined threshold.

### Claim 2 (depends on 1)

The method of claim 1, wherein the action is collecting information to be provided to an insurer or making an offer to the driver.

### Claim 3 (depends on 1)

The method of claim 1, further comprising collecting information related to a multiplicity of drivers in a road by an additional source interface, and wherein the action is issuing an alert related to a road, subject to a multiplicity of drivers having stress levels exceeding a threshold.

### Claim 4 (depends on 1)

The method of claim 1, wherein learning the stress-related parameters or thresholds comprises:
transmitting by the information fusing apparatus, the measurements and stress assessments to a server; and
normalizing by the server, the measurements and stress assessments with other drivers or normalizing measurements of the driver over time.

### Claim 5 (depends on 1)

The method of claim 1, wherein the action is affecting a behavior of a car system.

### Claim 6 (depends on 5)

The method of claim 5, wherein the car system is an autonomous driving system.

### Claim 7 (depends on 5)

The method of claim 5, wherein the behavior relates to changing parameters in accordance with a weather condition and driver behavior.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
receiving data from an additional source by an additional source interface; and
analyzing the data to by the stress level analysis engine, obtain an additional indicator; and
fusing the additional indicator with the telemetry information and the at least one indicator by the stress level analysis engine, to obtain the assessment.

### Claim 9 (depends on 8)

The method of claim 8, wherein the additional source is an image capture device or a voice capture device, and further comprising analyzing an image captured by the image capture device or voice captured by the voice capture device.

### Claim 10 (depends on 8)

The method of claim 8, wherein the additional source is at least one item selected from the group consisting of: a calendar of the driver, a global positioning system, a weather forecast source, a road condition report and a wearable device.

### Claim 11 (depends on 10)

The method of claim 10, further comprising determining by the stress level analysis engine, in accordance with the calendar and with a Global Positioning System whether the driver is late for a meeting.

### Claim 12

An apparatus for fusing information related to a driver of a vehicle, the apparatus comprising:
a hyper spectral camera capable of imaging body features invisible to a human;
car telemetry system providing vehicle-related telemetry information; and
a processor adapted to perform the steps of:
receiving vehicle-related telemetry information from a car telemetry system, wherein the vehicle-related telemetry information comprises at least one of vehicle speed, vehicle acceleration, lane passing and honking;
assessing a stress level of the driver from the vehicle-related telemetry information received from the car telemetry system;
receiving an image of the driver captured by the hyper spectral camera;
analyzing the image to receive at least one indicator to a clinical parameter of the driver;
learning stress-related parameters or thresholds from a behavior of the driver over time;
fusing the stress level assessed from the information received from the car telemetry system, with the at least one indicator received from analyzing the image;
using the stress-related parameters or thresholds as learned to obtain a combined assessment to a stress level of the driver;
based on the combined assessment, determining an action by an action selection engine; and
taking the action by an action activation engine, in response to the stress level exceeding a predetermined threshold.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the action is selected from the group consisting of: affecting a behavior of a car system, collecting information to be provided to an insurer or making an offer to the driver.

### Claim 14 (depends on 12)

The apparatus of claim 12, further comprising a connection to a server, wherein the server is adapted to collect information related to a multiplicity of drivers in a road, and issue an alert related to a road, subject to a multiplicity of drivers having stress levels exceeding a threshold.

### Claim 15 (depends on 12)

The apparatus of claim 12, wherein for learning the stress-related parameters or thresholds comprises the processor is adapted to:
transmit the measurements and stress assessments to a server; and
normalize the measurements and stress assessments with other drivers or normalizing measurements of the driver over time.

### Claim 16 (depends on 12)

The apparatus of claim 12, wherein the processor is further adapted to:
receive data from an additional source;
analyze the data to obtain an additional indicator; and
fuse the additional indicator with the telemetry information and the at least one indicator to obtain the assessment.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the additional source is selected from the group consisting of: an image capture device, a voice capture device, a calendar of the driver, a global positioning system, a weather forecast source, a road condition report, and a wearable device.

### Claim 18

A computer program product comprising a non-transitory computer readable storage medium retaining program instructions configured to cause a processor to perform actions, which program instructions implement:
receiving by a vehicle telemetry interface, vehicle-related telemetry information from a car telemetry system, wherein the vehicle-related telemetry information comprises at least one of vehicle speed, vehicle acceleration, lane passing and honking;
assessing by a telemetric analysis engine, a stress level of the driver from the vehicle-related telemetry information received from the car telemetry system;
receiving by an information fusing apparatus, an image of the driver captured by the hyper spectral camera;
analyzing the image by an image analysis engine, to receive at least one indicator to a clinical parameter of the driver;
learning by a learning engine, stress-related parameters or thresholds from a behavior of the driver over time; and
fusing by a stress level analysis engine, the stress level assessed from the information received from the car telemetry system, with the at least one indicator received from analyzing the image;
using the stress-related parameters or thresholds as learned to obtain a combined assessment to a stress level of the driver;
based on the combined assessment, determining an action by an action selection engine; and
taking the action by an action activation engine, in response to the stress level exceeding a predetermined threshold.