# US 12,602,796 B2 · System, device, and methods for detecting and obtaining information on objects in a vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,796 B2 - Document ID: US12602796B2 - Status: Granted - Kind: B2 - Application number: 18/607,853 - Filed: 2024-03-18 - Published: 2024-08-08 - Granted: 2026-04-14 - Earliest claimed: 2019-02-17 - Assignees: Gentex Corporation - Applicants: Gentex Corporation - Inventors: Yuval Gronau - Web page: https://patentgrep.com/patent/US12602796B2 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - B60W 50/0098: {Details of control systems ensuring comfort, safety or stability not otherwise provided for} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/593: {Recognising seat occupancy} - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/161: {Detection; Localisation; Normalisation} - B60W 2540/223: Posture, e.g. hand, foot, or seat position, turned or inclined - B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30232: Surveillance - G06T 2207/30242: Counting objects in image - G06T 2207/30268: Vehicle interior - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract System and methods are provided for detecting occupancy state in a vehicle including analyzing data of the interior passenger compartment to yield at least one probability of a current occupancy state of the vehicle and further analyzing additional data of the interior passenger compartment to yield predicted probabilities of the occupancy state of the vehicle, wherein each probability of the predicted probabilities relate to the probability of changing the current occupancy state to different occupancy state combining the current at least one probability of the current occupancy state with the predicted probabilities of the occupancy state to yield an updated probability of an updated occupancy state of the vehicle; determining the current occupancy state based on the updated probability of an updated occupancy state of the vehicle and generating an output to control one or more devices or applications based on the determined occupancy state. ## Claims ### Claim 1 A system for detecting the number of occupants in a vehicle having an interior passenger compartment, the system comprising: an image sensor for capturing one or more images of the occupants; and at least one processor in communication with the image sensor, the at least one processor configured to: obtain data including the one or more images of the occupants; apply one or more pose detection algorithms on said one or more images to yield one or more skeleton models, respectively, for each or one or more occupants of said occupants; and generate a prediction model based on the one or more skeleton models to detect the number of occupants in said vehicle. ### Claim 2 (depends on 1) The system of claim 1, wherein said one or more images comprise two-dimensional (2D) images and wherein said one or more skeleton models are 2D skeleton models. ### Claim 3 (depends on 2) The system of claim 2, wherein said obtained data comprise two or more images and wherein said two or more images comprise two-dimensional (2D) images and three-dimensional (3D) images of said interior passenger compartment and wherein said processor is configured to: analyze the 3D images to generate one or more depth map representations of the occupants; combine the depth map representations with the generated depth map representations to yield one or more 3D skeleton models of said occupants; and analyze said one or more 3D skeleton models to detect the number of occupants in said vehicle. ### Claim 4 (depends on 3) The system of claim 3, wherein said prediction model is a fusing algorithm. ### Claim 5 (depends on 4) The system of claim 4, wherein said fusing algorithm comprises combining two or more types of data inputs. ### Claim 6 (depends on 5) The system of claim 5, wherein said two or more types of data inputs include said one or more skeleton models and one or more of optical flow data or face detection data. ### Claim 7 (depends on 1) The system of claim 1, wherein the processor is configured to analyze the skeleton models based on predefined filtering criteria to remove invalid skeleton models. ### Claim 8 (depends on 1) The system of claim 1, wherein the analysis of said one or more skeleton models to detect the number of occupants in said vehicle comprises inserting the generated skeleton models to the prediction model, said prediction model is configured to estimate the number of occupants in time (t) based on current and previous (t−1) occupancy measurements. ### Claim 9 (depends on 1) The system of claim 1, wherein said prediction model is a stochastic model. ### Claim 10 (depends on 9) The system of claim 9, wherein said stochastic model is in the form of a Markov chain, or Markov matrix. ### Claim 11 (depends on 1) The system of claim 1, wherein the processor is further configured to: analyze additional data of said interior passenger compartment to yield predicted probabilities of said occupancy state of said vehicle, wherein each probability of said predicted probabilities relate to the probability of changing said current occupancy state to different occupancy state; combine said current at least one probability of the current occupancy state with said predicted probabilities of said occupancy state to yield an updated probability of an updated occupancy state of said vehicle; determine said current occupancy state based on said updated probability of an updated occupancy state of said vehicle; and generate an output to control one or more devices or applications based on the determined occupancy state. ### Claim 12 (depends on 11) The system of claim 11, wherein the at least one probability of a current occupancy state of said vehicle is in the form of a probability vector and the predicted probabilities of said occupancy state of said vehicle are in the form of a matrix. ### Claim 13 (depends on 12) The system of claim 12, wherein said probability vector is multiplied by said matrix to yield the updated probability of an updated occupancy state of said vehicle. ### Claim 14 (depends on 11) The system of claim 11, wherein said additional data is obtained from one or more of: prior knowledge of possible movements in said vehicle; 2D or 3D images of the interior passenger compartment; RF images; identified vibrations of one or more objects; identified micro-vibrations; speckle pattern analysis. ### Claim 15 A method for detecting the number of occupants in a vehicle having an interior passenger compartment, the method comprising: using at least one processor to: obtain data including one or more images of said occupants; apply one or more pose detection algorithms on said one or more images to yield one or more skeleton models, respectively, for each or one or more occupants of said occupants; and generate a prediction model based on the one or more skeleton models to detect the number of occupants in said vehicle. ### Claim 16 (depends on 15) The method of claim 15, wherein said one or more images comprise two-dimensional (2D) images and wherein said one or more skeleton models are 2D skeleton models. ### Claim 17 (depends on 16) The method of claim 16, wherein said obtained data comprise two or more images and wherein said two or more images comprise two-dimensional (2D) images and three-dimensional (3D) images of said interior passenger compartment and wherein said processor is configured to: analyze the 3D images to generate one or more depth map representations of the occupants; combine the depth map representations with the generated depth map representations to yield one or more 3D skeleton models of said occupants; and analyze said one or more 3D skeleton models to detect the number of occupants in said vehicle. ### Claim 18 (depends on 15) The method of claim 15, wherein the analysis of said one or more skeleton models to detect the number of occupants in said vehicle comprises inserting the generated skeleton models to the prediction model, said prediction model is configured to estimate the number of occupants in time (t) based on current and previous (t−1) occupancy measurements. ### Claim 19 (depends on 15) The method of claim 15, wherein the processor is further configured to: analyze additional data of said interior passenger compartment to yield predicted probabilities of said occupancy state of said vehicle, wherein each probability of said predicted probabilities relates to the probability of changing said current occupancy state to different occupancy state; combine said current at least one probability of the current occupancy state with said predicted probabilities of said occupancy state to yield an updated probability of an updated occupancy state of said vehicle; determine said current occupancy state based on said updated probability of an updated occupancy state of said vehicle; and generate an output to control one or more devices or applications based on the determined occupancy state. ### Claim 20 A non-transitory computer readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to perform the steps of: obtain data including one or more images of said occupants; apply one or more pose detection algorithms on said one or more images to yield one or more skeleton models, respectively, for each or one or more occupants of said occupants; and generate a prediction model based on the one or more skeleton models to detect the number of occupants in a vehicle having an interior passenger compartment.