# US 11,308,722 B2 · Method and system for determining an activity of an occupant of a vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,722 B2 - Document ID: US11308722B2 - Status: Granted - Kind: B2 - Application number: 17/006,652 - Filed: 2020-08-28 - Published: 2021-03-18 - Granted: 2022-04-19 - Earliest claimed: 2019-09-17 - Assignees: Aptiv Technologies AG - Applicants: Aptiv Technologies Limited - Inventors: Patrick Weyers; Alexander Barth; David Schiebener - Web page: https://patentgrep.com/patent/US11308722B2 ## CPC - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - B60W 40/09: Driving style or behaviour - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/253: {of extracted features} - G06N 3/02: Neural networks - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06V 40/107: {Static hand or arm} - B60W 2540/01: Occupants other than the driver - B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computer implemented method for determining an activity of an occupant of a vehicle comprises the following steps carried out by computer hardware components: capturing sensor data of the occupant using at least one sensor; determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data; determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data. ## Claims ### Claim 1 A method for determining an activity of an occupant of a vehicle, the method comprising: capturing sensor data, including data from a camera sensor, of the occupant using at least one sensor; determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data; determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data by classifying the activity into one or more classes of actions using an artificial neural network trained to evaluate temporal aspects of actions. ### Claim 2 (depends on 1) The method of claim 1, wherein the sensor data comprises an image and further sensor data; wherein the at least one portion of the sensor data comprises at least one portion of the image. ### Claim 3 (depends on 2) The method of claim 2, wherein the image comprises a plurality of intensity pixels, each intensity pixel indicating an intensity of light of a pre-determined wavelength or wavelength range received at the respective intensity pixel; and wherein the further sensor data comprises a plurality of distance pixels, each distance pixel indicates a distance between the further sensor and an object corresponding to the respective distance pixel. ### Claim 4 (depends on 1) The method of claim 1, wherein the pre-determined body part of the occupant comprises one of the occupant's hands or the occupant's face. ### Claim 5 (depends on 1) The method of claim 1, wherein a plurality of images and a plurality of further sensor data are captured; and wherein the two-dimensional or three-dimensional coordinates are determined based on at least one of the plurality of images and the plurality of further sensor data. ### Claim 6 (depends on 1) The method of claim 1, wherein the activity is determined based on a softmax method. ### Claim 7 (depends on 1) The method of claim 1, wherein classifying comprises determining a probability vector, wherein each entry of the probability vector indicates a respective probability for each class of the one or more classes of actions. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more classes of actions comprises one or more of the following actions: entering a car; leaving a car; inserting an object to a car; removing an object from a car; inserting a baby seat to a car; removing a baby seat from a car; inserting a child to a car; removing a child from a car; inserting a baby to a car; removing a baby from a car, buckling a seat belt; unbuckling a seat belt; interacting with a phone; holding a phone; active control of a phone; typing on a phone; talking on a phone; interacting with an object; interacting with a book; interacting with a magazine; interacting with a laptop; interacting with a tablet; interacting with a steering wheel; smoking; eating; drinking; operating a vehicle infotainment system; operating vehicle controls. ### Claim 9 (depends on 1) The method of claim 1, wherein determining the activity of the occupant is performed by further: inputting the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data into one or more other trained artificial neural networks; receiving feature vectors from the one or more other trained artificial neural networks; and inputting the feature vectors into the trained artificial neural network. ### Claim 10 (depends on 9) The method of claim 9, wherein the one or more other trained artificial neural networks comprise at least one of: a convolutional neural network (CNN); a recurrent neural network (RNN); a three-dimensional convolutional network; or a long short-term memory (LSTM). ### Claim 11 (depends on 9) The method of claim 9, wherein the one or more other trained artificial neural networks comprise a plurality of different types of artificial neural networks. ### Claim 12 (depends on 9) The method of claim 9, wherein each of the other trained artificial neural networks is trained using different training data. ### Claim 13 (depends on 9) The method of claim 9, wherein: the one or more trained artificial neural networks comprise at least two other trained artificial neural networks; and the method further comprises, prior to inputting the feature vectors into the trained artificial neural network, concatenating the feature vectors output from the at least two other trained artificial neural networks. ### Claim 14 A system comprising computer hardware components configured to determine an activity of an occupant of a vehicle by: capturing sensor data, including data from a camera sensor, of the occupant from at least one sensor of the vehicle; determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data; determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data by classifying the activity into one or more classes of actions using an artificial neural network trained to evaluate temporal aspects of actions. ### Claim 15 (depends on 14) The system of claim 14, further comprising an image sensor and at least one further sensor. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the at least one further sensor comprises a time of flight camera. ### Claim 17 (depends on 15) The computer system of any one of claim 15, wherein the image sensor and the at least one further sensor are a combined sensor. ### Claim 18 (depends on 14) The system of claim 14, wherein the computer hardware components are further configured to determine the activity of the occupant by: inputting the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data into one or more other trained artificial neural networks; and receiving feature vectors from the one or more other trained artificial neural networks; and inputting the feature vectors into the trained artificial neural network. ### Claim 19 A non-transitory computer readable medium comprising instructions that when executed, configure computer hardware components of a system to determine an activity of an occupant of a vehicle by: capturing sensor data, including data from a camera sensor, of the occupant from at least one sensor of the vehicle; determining respective two-dimensional or three-dimensional coordinates for a plurality of pre-determined portions of the body of the occupant based on the sensor data; determining at least one portion of the sensor data showing a pre-determined body part of the occupant based on the sensor data and the two-dimensional or three-dimensional coordinates; and determining the activity of the occupant based on the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data by classifying the activity into one or more classes of actions using an artificial neural network trained to evaluate temporal aspects of actions. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the instructions further configure the computer hardware components to determine the activity of the occupant by: inputting the two-dimensional or three-dimensional coordinates and the at least one portion of the sensor data into one or more other trained artificial neural networks; and receiving feature vectors from the one or more other trained artificial neural networks; and inputting the feature vectors into the trained artificial neural network.