# US 2024/0004382 A1 · Augmented reality in a vehicle configured for changing an emotional state of a rider

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004382 A1
- Document ID: US20240004382A1
- Status: Published application
- Kind: A1
- Application number: 18/370,362
- Filed: 2023-09-19
- Published: 2024-01-04
- Earliest claimed: 2018-09-30
- Assignees: Strong Force TP Portfolio 2022, LLC
- Applicants: Strong Force Tp Portfolio 2022, LLC
- Inventors: Charles Howard Cella
- Web page: https://patentgrep.com/patent/US20240004382A1

## CPC

- G06Q 30/0265: {Vehicular advertisement}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/168: {Evaluating attention deficit, hyperactivity}
- A61B 5/6893: {Cars}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- B60W 40/08: related to drivers or passengers
- G01C 21/3438: {Rendezvous; Ride sharing}
- G01C 21/3461: {Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types or segments such as motorways, toll roads or ferries}
- G01C 21/3469: {Fuel consumption; Energy use; Emission aspects}
- G01C 21/3617: {using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement}
- G05B 13/027: {using neural networks only}
- G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)}
- G05D 1/0287: {involving a plurality of land vehicles, e.g. fleet or convoy travelling (traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16)}
- G05D 1/20: Control system inputs
- G05D 1/224: Output arrangements on the remote controller, e.g. displays, haptics or speakers
- G05D 1/225: operated by off-board computers
- G05D 1/226: Communication links with the remote-control arrangements
- G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements
- G05D 1/228: Command input arrangements located on-board unmanned vehicles
- G05D 1/229: Command input data, e.g. waypoints
- G05D 1/24: Arrangements for determining position or orientation
- G05D 1/43: Control of position or course in two dimensions [2D]
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G05D 1/69: Coordinated control of the position or course of two or more vehicles
- G05D 1/692: involving a plurality of disparate vehicles
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/0418: {using chaos or fractal principles}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 20/00: Machine learning
- G06Q 30/0208: Trade or exchange of goods or services in exchange for incentives or rewards
- G06Q 50/188: {Electronic negotiation}
- G06Q 50/40: Business processes related to the transportation industry (shipping G06Q10/083)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 20/64: Three-dimensional [3D] objects
- G07C 5/006: {Indicating maintenance}
- G07C 5/008: {communicating information to a remotely located station (transmission systems for measured values G08C)}
- G07C 5/02: Registering or indicating driving, working, idle, or waiting time only (apparatus forming part of taximeters G07B13/00)
- G07C 5/08: Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C 5/0808: {Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005)}
- G07C 5/0816: {Indicating performance data, e.g. occurrence of a malfunction}
- G07C 5/0866: {the electronic data carrier being a digital video recorder in combination with video camera}
- G07C 5/0891: {Video recorder in combination with video camera}
- G10L 15/16: using artificial neural networks
- G10L 25/63: for estimating an emotional state
- A61B 5/0059: {using light, e.g. diagnosis by transillumination, diascopy, fluorescence (photoacoustic A61B5/0093; optical measurement of heart rate A61B5/02416; optical measurement of blood flow A61B5/0261; optical measurement of analytes A61B5/1455)}
- A61B 5/0533: Measuring galvanic skin response
- A61B 5/1176: {Recognition of faces}
- A61B 5/163: {by tracking eye movement, gaze, or pupil change}
- A61B 5/369: Electroencephalography [EEG]
- A61B 2503/22: Motor vehicles operators, e.g. drivers, pilots, captains
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61B 2562/0223: Magnetic field sensors
- B60W 2040/0881: {Seat occupation; Driver or passenger presence}
- G01C 21/3484: {Personalized, e.g. from learned user behaviour or user-defined profiles}
- G05D 2101/10: using artificial intelligence [AI] techniques
- G06N 3/02: Neural networks
- G06N 3/048: Activation functions
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 10/42: Determination of affinities or common interests between users
- G06Q 10/44: Identification of trends within social networks, e.g. identification of trending topics
- G06Q 30/0281: {Customer communication at a business location, e.g. providing product or service information, consulting}
- G08G 1/0968: Systems involving transmission of navigation instructions to the vehicle
- G08G 1/20: {Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles}
- Y02T 10/62: Hybrid vehicles
- Y02T 10/70: Energy storage systems for electromobility, e.g. batteries
- Y02T 10/7072: Electromobility specific charging systems or methods for batteries, ultracapacitors, supercapacitors or double-layer capacitors
- Y02T 90/12: Electric charging stations
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

Vehicles and methods described herein include a vehicle that operates with a rider according to an operating parameter. The vehicle includes: a physiological monitoring sensor configured to measure a physiological parameter of the rider; an experience hybrid neural network trained on outcomes related to a rider in-vehicle experience associated with the physiological parameter to determine an emotional state of the rider; an augmented reality system configured to present augmented reality content to the rider of the vehicle based, at least in part, on the operating parameter; and an optimization hybrid neural network that identifies a variation in the operating parameter to change the emotional state of the rider and that generates a command to vary the operating parameter and the augmented reality content according to the variation.

## Claims

### Claim 1

A vehicle that operates with a rider according to an operating parameter, the vehicle comprising:
a physiological monitoring sensor configured to measure a physiological parameter of the rider;
an experience hybrid neural network trained on outcomes related to a rider in-vehicle experience associated with the physiological parameter to determine an emotional state of the rider;
an augmented reality system configured to present augmented reality content to the rider of the vehicle based, at least in part, on the operating parameter; and
an optimization hybrid neural network that identifies a variation in the operating parameter to change the emotional state of the rider and that generates a command to vary the operating parameter and the augmented reality content according to the variation.

### Claim 2 (depends on 1)

The vehicle of claim 1, wherein the optimization hybrid neural network identifies the variation, at least in part, by controlling a timing and an extent of the variation.

### Claim 3 (depends on 1)

The vehicle of claim 1, wherein the operating parameter indicates entertainment content presented to the rider.

### Claim 4 (depends on 3)

The vehicle of claim 3, wherein the optimization hybrid neural network identifies the variation based, at least in part, on what entertainment is most effective for the rider to change the emotional state of the rider.

### Claim 5 (depends on 4)

The vehicle of claim 4, wherein the optimization hybrid neural network identifies the variation based, at least in part, on social media data associated with the rider.

### Claim 6 (depends on 4)

The vehicle of claim 4, wherein the experience hybrid neural network and the optimization hybrid neural network collectively perform at least one of random variations or systematized variations of various combinations of operating parameters to predict the emotional state of the rider.

### Claim 7 (depends on 6)

The vehicle of claim 6, wherein the various combinations include combinations of entertainment settings, seat settings, suspension settings, and route types.

### Claim 8 (depends on 6)

The vehicle of claim 6, wherein the experience hybrid neural network and the optimization hybrid neural network include genetic programming that promotes favorable combinations of the various combinations and eliminates unfavorable combinations of the various combinations to predict the emotional state of the rider.

### Claim 9 (depends on 6)

The vehicle of claim 6, wherein at least one of the experience hybrid neural network or the optimization hybrid neural network includes perceptrons that mimic human senses to determine a likely emotional state of the rider after presenting a variation of the operating parameter.

### Claim 10 (depends on 1)

The vehicle of claim 1, wherein the vehicle further includes a location registration system that registers a location and an orientation of the vehicle, and wherein the augmented reality system is configured to retrieve the location and the orientation from the location registration system and to present the augmented reality content based on the location and the orientation.

### Claim 11

A method of operating a vehicle having a rider according to an operating parameter, the method comprising:
measuring a physiological parameter of the rider with a physiological monitoring sensor;
determining an emotional state of the rider using an experience hybrid neural network trained on outcomes related to a rider in-vehicle experience associated with the physiological parameter;
presenting augmented reality content to the rider of the vehicle based, at least in part, on the operating parameter using an augmented reality system;
identifying, using an optimization hybrid neural network, a variation in the operating parameter that changes the emotional state of the rider; and
generating a command to vary the operating parameter and the augmented reality content according to the variation.

### Claim 12 (depends on 11)

The method of claim 11, wherein identifying the variation is based, at least in part, on controlling a timing and an extent of the variation.

### Claim 13 (depends on 11)

The method of claim 11, wherein operating the vehicle according to the operating parameter includes indicating entertainment content presented to the rider.

### Claim 14 (depends on 13)

The method of claim 13, wherein identifying the variation is based, at least in part, on what entertainment is most effective for the rider to change the emotional state of the rider.

### Claim 15 (depends on 14)

The method of claim 14, wherein identifying the variation is based, at least in part, on social media data associated with the rider.

### Claim 16 (depends on 14)

The method of claim 14, wherein identifying the variation includes performing at least one of random variations or systematized variations of various combinations of operating parameters to predict the emotional state of the rider.

### Claim 17 (depends on 16)

The method of claim 16, wherein the various combinations include combinations of entertainment settings, seat settings, suspension settings, and route types.

### Claim 18 (depends on 16)

The method of claim 16, wherein identifying the variation further includes using genetic programming that promotes favorable combinations of the various combinations and eliminates unfavorable combinations of the various combinations to predict the emotional state of the rider.

### Claim 19 (depends on 16)

The method of claim 16, further comprising utilizing perceptrons that mimic human senses to determine a likely emotional state of the rider after presenting a variation of the operating parameter.

### Claim 20 (depends on 11)

The method of claim 11, further comprising;
registering, at the vehicle, a location and an orientation of the vehicle;
retrieving, at the augmented reality system, the location and the orientation from the vehicle; and
presenting the augmented reality content based on the location and the orientation.
