# US 12,216,466 B2 · Method of maintaining a favorable emotional state of a rider of a vehicle by a neural network to classify emotional state indicative wearable sensor data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,216,466 B2
- Document ID: US12216466B2
- Status: Granted
- Kind: B2
- Application number: 18/396,429
- Filed: 2023-12-26
- Published: 2024-05-02
- Granted: 2025-02-04
- Earliest claimed: 2018-09-30
- Assignees: Strong Force TP Portfolio 2022, LLC
- Applicants: Strong Force Tp Portfolio 2022, LLC
- Inventors: Charles Howard Cella
- Web page: https://patentgrep.com/patent/US12216466B2

## CPC

- G06Q 30/0265: {Vehicular advertisement}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/168: {Evaluating attention deficit, hyperactivity}
- A61B 5/6893: {Cars}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- B60W 40/08: related to drivers or passengers
- G01C 21/3438: {Rendezvous; Ride sharing}
- G01C 21/3461: {Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types or segments such as motorways, toll roads or ferries}
- G01C 21/3469: {Fuel consumption; Energy use; Emission aspects}
- G01C 21/3617: {using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement}
- G05B 13/027: {using neural networks only}
- G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)}
- G05D 1/0287: {involving a plurality of land vehicles, e.g. fleet or convoy travelling (traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16)}
- G05D 1/20: Control system inputs
- G05D 1/224: Output arrangements on the remote controller, e.g. displays, haptics or speakers
- G05D 1/225: operated by off-board computers
- G05D 1/226: Communication links with the remote-control arrangements
- G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements
- G05D 1/228: Command input arrangements located on-board unmanned vehicles
- G05D 1/229: Command input data, e.g. waypoints
- G05D 1/24: Arrangements for determining position or orientation
- G05D 1/43: Control of position or course in two dimensions [2D]
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G05D 1/69: Coordinated control of the position or course of two or more vehicles
- G05D 1/692: involving a plurality of disparate vehicles
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/0418: {using chaos or fractal principles}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 20/00: Machine learning
- G06Q 30/0208: Trade or exchange of goods or services in exchange for incentives or rewards
- G06Q 50/188: {Electronic negotiation}
- G06Q 50/40: Business processes related to the transportation industry (shipping G06Q10/083)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 20/64: Three-dimensional [3D] objects
- G07C 5/006: {Indicating maintenance}
- G07C 5/008: {communicating information to a remotely located station (transmission systems for measured values G08C)}
- G07C 5/02: Registering or indicating driving, working, idle, or waiting time only (apparatus forming part of taximeters G07B13/00)
- G07C 5/08: Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C 5/0808: {Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005)}
- G07C 5/0816: {Indicating performance data, e.g. occurrence of a malfunction}
- G07C 5/0866: {the electronic data carrier being a digital video recorder in combination with video camera}
- G07C 5/0891: {Video recorder in combination with video camera}
- G10L 15/16: using artificial neural networks
- G10L 25/63: for estimating an emotional state
- A61B 5/0059: {using light, e.g. diagnosis by transillumination, diascopy, fluorescence (photoacoustic A61B5/0093; optical measurement of heart rate A61B5/02416; optical measurement of blood flow A61B5/0261; optical measurement of analytes A61B5/1455)}
- A61B 5/0533: Measuring galvanic skin response
- A61B 5/1176: {Recognition of faces}
- A61B 5/163: {by tracking eye movement, gaze, or pupil change}
- A61B 5/369: Electroencephalography [EEG]
- A61B 2503/22: Motor vehicles operators, e.g. drivers, pilots, captains
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61B 2562/0223: Magnetic field sensors
- B60W 2040/0881: {Seat occupation; Driver or passenger presence}
- G01C 21/3484: {Personalized, e.g. from learned user behaviour or user-defined profiles}
- G05D 2101/10: using artificial intelligence [AI] techniques
- G06N 3/02: Neural networks
- G06N 3/048: Activation functions
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 10/42: Determination of affinities or common interests between users
- G06Q 10/44: Identification of trends within social networks, e.g. identification of trending topics
- G06Q 30/0281: {Customer communication at a business location, e.g. providing product or service information, consulting}
- G08G 1/0968: Systems involving transmission of navigation instructions to the vehicle
- G08G 1/20: {Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles}
- Y02T 10/62: Hybrid vehicles
- Y02T 10/70: Energy storage systems for electromobility, e.g. batteries
- Y02T 10/7072: Electromobility specific charging systems or methods for batteries, ultracapacitors, supercapacitors or double-layer capacitors
- Y02T 90/12: Electric charging stations
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

A method of maintaining a favorable emotional state of a rider of a vehicle. The method includes: receiving emotional state indicative wearable sensor data of the rider in the vehicle; classifying, at a rider emotional state determining neural network, the emotional state indicative wearable sensor data and generating output data indicative of a classification of the rider emotional state as one of a favorable emotional state of the rider or an unfavorable emotional state of the rider; converting the output data from the rider emotional state determining neural network into a vehicle operational state datum; and adjusting a vehicle operating parameter in response to the vehicle operational state datum.

## Claims

### Claim 1

A method of maintaining a favorable emotional state of a rider in a vehicle, the method comprising:
receiving emotional state indicative wearable sensor data of the rider;
classifying, at a rider emotional state determining neural network, the emotional state indicative wearable sensor data and generating output data indicative of a classification of an emotional state as one of: the favorable emotional state of the rider; or an unfavorable emotional state of the rider;
converting the output data from the rider emotional state determining neural network into a vehicle operational state datum; and
adjusting a vehicle operating parameter in response to the vehicle operational state datum.

### Claim 2 (depends on 1)

The method of claim 1 wherein the output data is indicative of at least one of the emotional state of the rider is changing, the emotional state of the rider is stable, a rate of change of the emotional state of the rider, a direction of change of the emotional state of the rider, or a polarity of a change of the emotional state of the rider.

### Claim 3 (depends on 1)

The method of claim 1 wherein the output data is indicative of one of: the emotional state of the rider is changing to an unfavorable state; or the emotional state of the rider is changing to a favorable state.

### Claim 4 (depends on 1)

The method of claim 1 wherein the vehicle operating parameter includes at least one of: a route of the vehicle, an in-vehicle audio content, a speed of the vehicle, an acceleration of the vehicle, a deceleration of the vehicle, a proximity to objects along the route, or a proximity to other vehicles along the route.

### Claim 5 (depends on 1)

The method of claim 1 wherein the receiving emotional state indicative wearable sensor data comprises:
receiving the emotional state indicative wearable sensor data from at least one of: a watch, a ring, a wrist band, an arm band, an ankle band, a torso band, a skin patch, a head-worn device, eyeglasses, foot wear, a glove, an in-ear device, clothing, headphones, a belt, a finger ring, a thumb ring, a toe ring, or a necklace.

### Claim 6 (depends on 1)

The method of claim 1 wherein the receiving emotional state indicative wearable sensor data comprises:
detecting, at the rider emotional state determining neural network, the emotional state of the rider through expert system-based processing of the emotional state indicative wearable sensor data of a plurality of wearable physiological condition sensors worn by the rider in the vehicle.

### Claim 7 (depends on 6)

The method of claim 6 wherein the plurality of wearable physiological condition sensors comprises at least two of: a watch, a ring, a wrist band, an arm band, an ankle band, a torso band, a skin patch, a head-worn device, eyeglasses, foot wear, a glove, an in-ear device, clothing, headphones, a belt, a finger ring, a thumb ring, a toe ring, or a necklace.

### Claim 8 (depends on 1)

The method of claim 1 wherein the vehicle is a self-driving vehicle.

### Claim 9 (depends on 1)

The method of claim 1 wherein classifying the emotional state indicative wearable sensor data includes classifying the emotional state of the rider as the favorable emotional state, and wherein the method further comprises
optimizing the vehicle operating parameter to maintain the favorable emotional state.

### Claim 10 (depends on 1)

The method of claim 1 wherein classifying the emotional state indicative wearable sensor data includes using at least one of: a training set of emotional state indicators of a set of riders or trainer-generated rider emotional state indicators.

### Claim 11

A data processing system for maintaining a favorable emotional state of a rider of a vehicle, the data processing system comprising:
a processor;
a wearable sensor worn by the rider configured to communicate emotional state indicative wearable sensor data;
a neural network configured to receive the emotional state indicative wearable sensor data of the rider; and
a rider emotional state determining neural network configured to classify the emotional state indicative wearable sensor data and to generate output data indicative of a classification as one of: the favorable emotional state of the rider; or an unfavorable emotional state of the rider;
wherein the neural network is configured to convert the output data from the rider emotional state determining neural network into vehicle operational state data, and to adjust a vehicle operating parameter in response to the vehicle operational state data.

### Claim 12 (depends on 11)

The data processing system of claim 11 wherein the output data is indicative of at least one of: an emotional state of the rider is changing, the emotional state of the rider is stable, a rate of change of the emotional state of the rider, a direction of change of the emotional state of the rider, or a polarity of a change of the emotional state of the rider.

### Claim 13 (depends on 11)

The data processing system of claim 11 wherein the output data is indicative of one of: an emotional state of the rider is changing to an unfavorable state; or the emotional state of the rider is changing to a favorable state.

### Claim 14 (depends on 11)

The data processing system of claim 11 wherein the vehicle operating parameter includes at least one of: a route of the vehicle, an in-vehicle audio content, a speed of the vehicle, an acceleration of the vehicle, a deceleration of the vehicle, a proximity to objects along the route, or a proximity to other vehicles along the route.

### Claim 15 (depends on 11)

The data processing system of claim 11 wherein the emotional state indicative wearable sensor data comprises:
the emotional state indicative wearable sensor data from at least one of: a watch, a ring, a wrist band, an arm band, an ankle band, a torso band, a skin patch, a head-worn device, eyeglasses, foot wear, a glove, an in-ear device, clothing, headphones, a belt, a finger ring, a thumb ring, a toe ring, or a necklace.

### Claim 16 (depends on 11)

The data processing system of claim 11 wherein the neural network detects an emotional state of the rider through expert system-based processing of the emotional state indicative wearable sensor data of a plurality of wearable physiological condition sensors worn by the rider.

### Claim 17 (depends on 16)

The data processing system of claim 16 wherein the plurality of wearable physiological condition sensors comprise at least two of: a watch, a ring, a wrist band, an arm band, an ankle band, a torso band, a skin patch, a head-worn device, eyeglasses, foot wear, a glove, an in-ear device, clothing, headphones, a belt, a finger ring, a thumb ring, a toe ring, or a necklace.

### Claim 18 (depends on 11)

The data processing system of claim 11 wherein the vehicle is a self-driving vehicle.

### Claim 19 (depends on 11)

The data processing system of claim 11 wherein the rider emotional state determining neural network is further configured to optimize the vehicle operating parameter to maintain the favorable emotional state.

### Claim 20 (depends on 11)

The data processing system of claim 11 wherein the rider emotional state determining neural network is configured to classify the emotional state indicative wearable sensor data by using at least one of: a training set of emotional state indicators of a set of riders; or trainer-generated rider emotional state indicators.