# US 12,222,714 B2 · Robotic process automation system trained to undertake actions with a vehicle based on user interactions with a user interface of the vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,222,714 B2
- Document ID: US12222714B2
- Status: Granted
- Kind: B2
- Application number: 18/395,424
- Filed: 2023-12-22
- Published: 2024-04-18
- Granted: 2025-02-11
- Earliest claimed: 2018-09-30
- Assignees: Strong Force TP Portfolio 2022, LLC
- Applicants: Strong Force Tp Portfolio 2022, LLC
- Inventors: Charles Howard Cella
- Web page: https://patentgrep.com/patent/US12222714B2

## CPC

- G06Q 30/0265: {Vehicular advertisement}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/168: {Evaluating attention deficit, hyperactivity}
- A61B 5/6893: {Cars}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- B60W 40/08: related to drivers or passengers
- G01C 21/3438: {Rendezvous; Ride sharing}
- G01C 21/3461: {Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types or segments such as motorways, toll roads or ferries}
- G01C 21/3469: {Fuel consumption; Energy use; Emission aspects}
- G01C 21/3617: {using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement}
- G05B 13/027: {using neural networks only}
- G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)}
- G05D 1/0287: {involving a plurality of land vehicles, e.g. fleet or convoy travelling (traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16)}
- G05D 1/20: Control system inputs
- G05D 1/224: Output arrangements on the remote controller, e.g. displays, haptics or speakers
- G05D 1/225: operated by off-board computers
- G05D 1/226: Communication links with the remote-control arrangements
- G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements
- G05D 1/228: Command input arrangements located on-board unmanned vehicles
- G05D 1/229: Command input data, e.g. waypoints
- G05D 1/24: Arrangements for determining position or orientation
- G05D 1/43: Control of position or course in two dimensions [2D]
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G05D 1/69: Coordinated control of the position or course of two or more vehicles
- G05D 1/692: involving a plurality of disparate vehicles
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/0418: {using chaos or fractal principles}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 20/00: Machine learning
- G06Q 30/0208: Trade or exchange of goods or services in exchange for incentives or rewards
- G06Q 50/188: {Electronic negotiation}
- G06Q 50/40: Business processes related to the transportation industry (shipping G06Q10/083)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 20/64: Three-dimensional [3D] objects
- G07C 5/006: {Indicating maintenance}
- G07C 5/008: {communicating information to a remotely located station (transmission systems for measured values G08C)}
- G07C 5/02: Registering or indicating driving, working, idle, or waiting time only (apparatus forming part of taximeters G07B13/00)
- G07C 5/08: Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
- G07C 5/0808: {Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005)}
- G07C 5/0816: {Indicating performance data, e.g. occurrence of a malfunction}
- G07C 5/0866: {the electronic data carrier being a digital video recorder in combination with video camera}
- G07C 5/0891: {Video recorder in combination with video camera}
- G10L 15/16: using artificial neural networks
- G10L 25/63: for estimating an emotional state
- A61B 5/0059: {using light, e.g. diagnosis by transillumination, diascopy, fluorescence (photoacoustic A61B5/0093; optical measurement of heart rate A61B5/02416; optical measurement of blood flow A61B5/0261; optical measurement of analytes A61B5/1455)}
- A61B 5/0533: Measuring galvanic skin response
- A61B 5/1176: {Recognition of faces}
- A61B 5/163: {by tracking eye movement, gaze, or pupil change}
- A61B 5/369: Electroencephalography [EEG]
- A61B 2503/22: Motor vehicles operators, e.g. drivers, pilots, captains
- A61B 2562/0219: Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
- A61B 2562/0223: Magnetic field sensors
- B60W 2040/0881: {Seat occupation; Driver or passenger presence}
- G01C 21/3484: {Personalized, e.g. from learned user behaviour or user-defined profiles}
- G05D 2101/10: using artificial intelligence [AI] techniques
- G06N 3/02: Neural networks
- G06N 3/048: Activation functions
- G06Q 10/40: Business processes related to social networking or social networking services
- G06Q 10/42: Determination of affinities or common interests between users
- G06Q 10/44: Identification of trends within social networks, e.g. identification of trending topics
- G06Q 30/0281: {Customer communication at a business location, e.g. providing product or service information, consulting}
- G08G 1/0968: Systems involving transmission of navigation instructions to the vehicle
- G08G 1/20: {Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles}
- Y02T 10/62: Hybrid vehicles
- Y02T 10/70: Energy storage systems for electromobility, e.g. batteries
- Y02T 10/7072: Electromobility specific charging systems or methods for batteries, ultracapacitors, supercapacitors or double-layer capacitors
- Y02T 90/12: Electric charging stations
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

A method of operating a vehicle in a transportation system includes: receiving data collected for each of a set of users based on interactions of the set of users with a user interface of the vehicle including at least one of a vehicle data system, a vision system, and a connected system; performing robotic process automation on the interactions; learning changes in driving style using an artificial intelligence system based on the robotic process automation; and undertaking an action within the vehicle on behalf of the user based on the learned changes in driving style.

## Claims

### Claim 1

A method of operating a vehicle in a transportation system, the method comprising:
receiving data collected for each of a set of users based on interactions of the set of users with a user interface of the vehicle including at least one of a vehicle data system, a vision system, and a connected system;
performing robotic process automation on the interactions to learn a driving style for at least one user of the set of users;
learning changes in the driving style using an artificial intelligence system based on the robotic process automation and based on feedback of the robotic process automation; and
optimizing a margin of safety and controlling the vehicle with the optimized margin of safety on behalf of the at least one user of the set of users based on the learned changes in the driving style, wherein the optimizing and the controlling affect at least one of: a route of the vehicle, a speed of the vehicle, an acceleration of the vehicle, a deceleration of the vehicle, a proximity to other vehicles along the route of the vehicle, a proximity to objects along the route of the vehicle, braking intensity, steering sensitivity, or route selection.

### Claim 2 (depends on 1)

The method of claim 1, wherein the data collected for each of the set of users includes at least one of: driving speed, braking patterns, steering patterns, or route preferences.

### Claim 3 (depends on 1)

The method of claim 1, wherein the artificial intelligence system is further trained using data from external sources including at least one of: traffic data, weather data, or road condition data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the artificial intelligence system uses machine learning algorithms to learn the changes in the driving style.

### Claim 5 (depends on 1)

The method of claim 1, wherein the user interface of the vehicle includes at least one of: a touch screen display, voice recognition system, or a gesture recognition system.

### Claim 6 (depends on 1)

The method of claim 1, wherein the learned changes in the driving style are stored in at least one user profile associated with the at least one user of the set of users.

### Claim 7 (depends on 1)

The method of claim 1, wherein a robotic process automation system provides audio or visual feedback to the at least one user regarding the learned changes in the driving style.

### Claim 8 (depends on 1)

The method of claim 1, wherein an action undertaken within the vehicle during the optimizing and the controlling is based on real-time data collected during a current driving session.

### Claim 9 (depends on 1)

The method of claim 1, wherein the artificial intelligence system is further trained to predict future changes in the driving style based on historical data.

### Claim 10

A method of robotic process automation to facilitate mimicking human operation of a vehicle, the method comprising:
tracking human interactions with a vehicle control-facilitating interface;
recording the tracked human interactions in a robotic process automation training data structure;
tracking vehicle operational state information of the vehicle, wherein the vehicle is controlled through the vehicle control-facilitating interface;
recording the vehicle operational state information in the robotic process automation training data structure;
training, through use of at least one neural network, an artificial intelligence system to operate the vehicle in a manner consistent with the human interactions based on the human interactions and the vehicle operational state information in the robotic process automation training data structure and based on feedback obtained from the robotic process automation training data structure,
wherein the training of the artificial intelligence system further involves training the artificial intelligence system to optimize a margin of safety and to control the vehicle on behalf of a user of the vehicle, wherein the optimizing and the controlling affect at least one of: a route of the vehicle, a speed of the vehicle, an acceleration of the vehicle, a deceleration of the vehicle, a proximity to other vehicles along the route of the vehicle, a proximity to objects along the route of the vehicle, braking intensity, steering sensitivity, or route selection; and
controlling, via the trained artificial intelligence system, the vehicle with the optimized margin of safety.

### Claim 11 (depends on 10)

The method of claim 10, wherein the human interactions with the vehicle control-facilitating interface include at least one of: steering inputs, braking inputs, acceleration inputs, or route selection inputs.

### Claim 12 (depends on 10)

The method of claim 10, wherein the vehicle operational state information includes at least one of: the speed of the vehicle, a location of the vehicle, a direction of the vehicle, a fuel level of the vehicle, or a battery level of the vehicle.

### Claim 13 (depends on 10)

The method of claim 10, wherein the at least one neural network is a deep learning neural network.

### Claim 14 (depends on 10)

The method of claim 10, wherein the vehicle control-facilitating interface includes at least one of: a touch screen display, a steering wheel, a brake pedal, an accelerator pedal, or a navigation system.

### Claim 15 (depends on 10)

The method of claim 10, wherein the robotic process automation training data structure is stored in a non-transitory computer-readable medium.

### Claim 16 (depends on 10)

The method of claim 10, wherein the training of the artificial intelligence system further includes reinforcement learning based on the feedback.

### Claim 17 (depends on 10)

The method of claim 10, further comprising providing a user interface that allows a human operator to override operation of the vehicle by the artificial intelligence system.

### Claim 18 (depends on 10)

The method of claim 10, wherein the vehicle is a self-driving vehicle and the human interactions are simulated based on data from a database of human driving behavior.