# US 2021/0081689 A1 · Method and System for Determining an Activity of an Occupant of a Vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081689 A1
- Document ID: US20210081689A1
- Status: Published application
- Kind: A1
- Application number: 17/006,652
- Filed: 2020-08-28
- Published: 2021-03-18
- Earliest claimed: 2019-09-17
- Assignees: Aptiv Technologies AG
- Applicants: Aptiv Technologies Limited
- Inventors: Patrick Weyers; Alexander Barth; David Schiebener
- Web page: https://patentgrep.com/patent/US20210081689A1

## CPC

- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- B60W 40/09: Driving style or behaviour
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06F 18/253: {of extracted features}
- G06N 3/02: Neural networks
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 40/107: {Static hand or arm}
- B60W 2540/01: Occupants other than the driver
- B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer implemented method for determining an activity of an occupant of a vehicle comprises the following steps carried out by computer hardware components: capturing sensor data of the occupant using at least one sensor; determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data; determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data.

## Claims

### Claim 1

A method for determining an activity of an occupant of a vehicle, the method comprising:
capturing sensor data of the occupant using at least one sensor;
determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data;
determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and
determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the sensor data comprises an image and further sensor data;
wherein the at least one portion of the sensor data comprises at least one portion of the image.

### Claim 3 (depends on 2)

The method of claim 2,
wherein the image comprises a plurality of intensity pixels, each intensity pixel indicating an intensity of light of a pre-determined wavelength or wavelength range received at the respective intensity pixel; and
wherein the further sensor data comprises a plurality of distance pixels, each distance pixel indicates a distance between the further sensor and an object corresponding to the respective distance pixel.

### Claim 4 (depends on 1)

The method of claim 1,
wherein the pre-determined body part of the occupant comprises one of the occupant's hands or the occupant's face.

### Claim 5 (depends on 1)

The method of claim 1,
wherein a plurality of images and a plurality of further sensor data are captured; and
wherein the two-dimensional or three-dimensional coordinates are determined based on at least one of the plurality of images and the plurality of further sensor data.

### Claim 6 (depends on 1)

The method of claim 1,
wherein the activity is determined based on a neural network.

### Claim 7 (depends on 1)

The method of claim 1,
wherein the activity is determined based on a softmax method.

### Claim 8 (depends on 1)

The method of claim 1,
wherein determining the activity comprises classifying an action of a driver into one or more classes of a plurality of classes of actions.

### Claim 9 (depends on 8)

The method of claim 8,
wherein classifying comprises determining a probability vector, wherein each entry of the probability vector indicates a respective probability for each class of the plurality of classes of actions.

### Claim 10 (depends on 8)

The method of claim 8,
wherein the plurality of classes of actions comprises one or more of the following actions: entering a car; leaving a car; inserting an object to a car; removing an object from a car; inserting a baby seat to a car; removing a baby seat from a car; inserting a child to a car; removing a child from a car; inserting a baby to a car; removing a baby from a car, buckeling up a seat belt; unbuckeling a seat belt; interacting with a phone; holding a phone; active control of a phone; typing on a phone; talking on a phone; interacting with an object; interacting with a book; interacting with a magazine; interacting with a laptop; interacting with a tablet; interacting with a steering wheel; smoking; eating; drinking; operating a vehicle infotainment system; operating vehicle controls.

### Claim 11

A system comprising computer hardware components configured to determine an activity of an occupant of a vehicle by at least:
capturing sensor data of the occupant from at least one sensor of the vehicle;
determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data;
determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and
determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data.

### Claim 12 (depends on 11)

The system of claim 11, further comprising an image sensor and at least one further sensor.

### Claim 13 (depends on 12)

The computer system of claim 12, wherein the at least one further sensor comprises a time of flight camera.

### Claim 14 (depends on 12)

The computer system of any one of claim 12, wherein the image sensor and the at least one further sensor are a combined sensor.

### Claim 15

A non-transitory computer readable medium comprising instructions that when executed, configure computer hardware components of a system to determine an activity of an occupant of a vehicle by at least:
capturing sensor data of the occupant from at least one sensor of the vehicle;
determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data;
determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and
determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data.
