# US 2021/0279514 A1 · Vehicle manipulation with convolutional image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279514 A1
- Document ID: US20210279514A1
- Status: Published application
- Kind: A1
- Application number: 17/327,813
- Filed: 2021-05-24
- Published: 2021-09-09
- Earliest claimed: 2010-06-07
- Assignees: Affectiva, Inc.
- Applicants: Affectiva, Inc.
- Inventors: Panu James Turcot; Rana el Kaliouby; Abdelrahman N. Mahmoud; Seyedmohammad Mavadati
- Web page: https://patentgrep.com/patent/US20210279514A1

## CPC

- A61B 5/18: for vehicle drivers {or machine operators}
- A61B 5/7267: {involving training the classification device}
- B60W 60/00: Drive control systems specially adapted for autonomous road vehicles
- B60W 60/0013: {specially adapted for occupant comfort}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/593: {Recognising seat occupancy}
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/174: {Facial expression recognition}
- G06V 40/175: {Static expression}
- A61B 2503/22: Motor vehicles operators, e.g. drivers, pilots, captains
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2540/043: Identity of occupants
- B60W 2556/05: Big data
- B60W 2556/10: Historical data
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed embodiments provide for vehicle manipulation with convolutional image processing. The convolutional image processing uses a multilayered analysis engine. A plurality of images is obtained using an imaging device within a first vehicle. A multilayered analysis engine is trained using the plurality of images. The multilayered analysis engine includes multiple layers that include convolutional layers and hidden layers. The evaluating provides a cognitive state analysis. Further images are evaluated using the multilayered analysis engine. The further images include facial image data from one or more persons present in a second vehicle. Manipulation data is provided to the second vehicle based on the evaluating the further images. An additional plurality of images of one or more occupants of one or more additional vehicles is obtained. The additional images provide opted-in, crowdsourced image training. The crowdsourced image training enables retraining the multilayered analysis engine.

## Claims

### Claim 1

A computer-implemented method for vehicle manipulation comprising:
initializing a computer for convolutional processing;
obtaining, using an imaging device within a first vehicle, a plurality of images of an occupant of the first vehicle;
training, on the computer initialized for convolutional processing, a multilayered analysis engine using the plurality of images, wherein the multilayered analysis engine includes multiple layers that include one or more convolutional layers and one or more hidden layers, and wherein the multilayered analysis engine is used for cognitive state analysis;
evaluating further images, using the multilayered analysis engine, wherein the further images include facial image data from one or more persons present in a second vehicle; and
providing manipulation data to the second vehicle based on the evaluating.

### Claim 2 (depends on 1)

The method of claim 1 further comprising obtaining an additional plurality of images of one or more occupants of one or more additional vehicles.

### Claim 3 (depends on 2)

The method of claim 2 wherein the training includes the additional plurality of images.

### Claim 4 (depends on 2)

The method of claim 2 wherein the obtaining an additional plurality of images is accomplished using opted-in imaging from the one or more occupants of one or more additional vehicles.

### Claim 5 (depends on 4)

The method of claim 4 wherein the using opted-in imaging comprises crowdsourced image training.

### Claim 6 (depends on 5)

The method of claim 5 wherein the crowdsourced image training comprises retraining the multilayered analysis engine.

### Claim 7 (depends on 5)

The method of claim 5 wherein the crowdsourced image training is accomplished demographically.

### Claim 8 (depends on 7)

The method of claim 7 wherein the evaluating further images is weighted by a demographic assessment of the one or more persons present in the second vehicle.

### Claim 9 (depends on 8)

The method of claim 8 wherein the demographic assessment is provided by at least one of the one or more persons present in the second vehicle.

### Claim 10 (depends on 8)

The method of claim 8 wherein the demographic assessment is included in the evaluating.

### Claim 11 (depends on 1)

The method of claim 1 wherein the evaluating provides a cognitive state analysis.

### Claim 12 (depends on 11)

The method of claim 11 further comprising collecting voice data from the one or more persons present in the second vehicle and augmenting the cognitive state analysis.

### Claim 13 (depends on 12)

The method of claim 12 further comprising mapping cognitive state data from the cognitive state analysis to location data along a vehicle travel route.

### Claim 14 (depends on 13)

The method of claim 13 further comprising updating information about the vehicle travel route based on the cognitive state data.

### Claim 15 (depends on 1)

The method of claim 1 further comprising manipulating the second vehicle based on the manipulation data.

### Claim 16 (depends on 1)

The method of claim 1 further comprising generating a cognitive state profile for one or the one or more persons present in the second vehicle.

### Claim 17 (depends on 1)

The method of claim 1 wherein a last layer within the multiple layers provides output indicative of cognitive state.

### Claim 18 (depends on 17)

The method of claim 17 further comprising tuning the last layer within the multiple layers for a particular cognitive state.

### Claim 19 (depends on 1)

The method of claim 1 wherein the training comprises assigning weights to inputs on one or more layers within the multilayered analysis engine.

### Claim 20 (depends on 1)

The method of claim 1 wherein the occupant is a passenger within the vehicle.

### Claim 21 (depends on 1)

The method of claim 1 wherein the occupant is a driver within the vehicle.

### Claim 22 (depends on 1)

The method of claim 1 wherein the second vehicle is an autonomous or semi-autonomous vehicle.

### Claim 23 (depends on 1)

The method of claim 1 further comprising learning image descriptors, as part of the training, for emotional content.

### Claim 24 (depends on 23)

The method of claim 23 wherein the image descriptors are identified based on a temporal co-occurrence with an external stimulus.

### Claim 25 (depends on 1)

The method of claim 1 further comprising training an emotion classifier, as part of the training, for emotional content.

### Claim 26 (depends on 1)

The method of claim 1 wherein the multilayered analysis engine comprises a convolutional neural network.

### Claim 27

A computer program product embodied in a non-transitory computer readable medium for vehicle manipulation, the computer program product comprising code which causes one or more processors to perform operations of:
initializing a computer for convolutional processing;
obtaining, using an imaging device within a first vehicle, a plurality of images of an occupant of the first vehicle;
training, on the computer initialized for convolutional processing, a multilayered analysis engine using the plurality of images, wherein the multilayered analysis engine includes multiple layers that include one or more convolutional layers and one or more hidden layers, and wherein the multilayered analysis engine is used for cognitive state analysis;
evaluating further images, using the multilayered analysis engine, wherein the further images include facial image data from one or more persons present in a second vehicle; and
providing manipulation data to the second vehicle based on the evaluating.

### Claim 28

A computer system for vehicle manipulation comprising:
a memory which stores instructions;
one or more processors coupled to the memory wherein the one or more processors, when executing the instructions which are stored, are configured to:
initialize a computer for convolutional processing;
obtain, using an imaging device within a first vehicle, a plurality of images of an occupant of the first vehicle;
train a multilayered analysis engine on the computer using the plurality of images, wherein the multilayered analysis engine includes multiple layers that include one or more convolutional layers and one or more hidden layers, and wherein the multilayered analysis engine is used for cognitive state analysis;
evaluate further images, using the multilayered analysis engine, wherein the further images include facial image data from one or more persons present in a second vehicle; and
provide manipulation data to the second vehicle based on the evaluating.
