# US 12,762,869 B2 · Image-based three-dimensional occupant assessment for in-cabin monitoring systems and applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,869 B2 - Document ID: US12762869B2 - Status: Granted - Kind: B2 - Application number: 18/349,853 - Filed: 2023-07-10 - Published: 2025-01-16 - Granted: 2026-10-13 - Earliest claimed: 2023-07-10 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Sakthivel Sivaraman; Arjun Guru; Rajath Shetty; Umar Iqbal; Orazio Gallo; Hang Su; Abhishek Badki; Varsha Hedau - Web page: https://patentgrep.com/patent/US12762869B2 ## CPC - G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions - G06T 7/50: Depth or shape recovery - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30268: Vehicle interior - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In various examples, image-based three-dimensional occupant assessment for in-cabin monitoring systems and applications are provided. An evaluation function may determine a 3D representation of an occupant of a machine by evaluating sensor data comprising an image frame from an optical image sensor. The 3D representation may comprise at least one characteristic representative of a size of the occupant, (e.g., a 3D pose and/or 3D shape), which may be used to derive other characteristics such as, but not limited to weight, height, and/or age. A first processing path may generate a representation of one or more features corresponding to at least a portion of the occupant based on optical image data, and a second processing path may determine a depth corresponding to the one or more features based on depth data derived from the optical image data and ground truth depth data corresponding to the interior of the machine. ## Claims ### Claim 1 A system comprising: one or more processing units comprising circuitry to: detect one or more features representing at least a portion of an occupant of a machine based at least on two-dimensional optical image data; determine a depth corresponding to the one or more features based at least on the optical image data and a ground truth cabin depth image corresponding to an interior of the machine; generate a three-dimensional representation of the occupant based at least on the one or more features and the depth of the one or more features, wherein the three-dimensional representation comprises at least one characteristic representative of a size of the occupant; and control at least one operation of the machine based at least on the at least one characteristic. ### Claim 2 (depends on 1) The system of claim 1, wherein the at least one characteristic comprises either a three-dimensional shape estimate or a three-dimensional pose estimate. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more processing units are further to: execute at least one three-dimensional pose detection model to generate a scale-normalized three-dimensional pose estimate of the occupant based at least on the one or more features; and scale the scale-normalized three-dimensional pose estimate to a linear measurement scale three-dimensional pose estimate based at least on the depth corresponding to the one or more features. ### Claim 4 (depends on 3) The system of claim 3, wherein the one or more processing units are further to: generate the at least one characteristic representative of the size of the occupant based at least on the linear measurement scale three-dimensional pose estimate. ### Claim 5 (depends on 3) The system of claim 3, wherein the one or more processing units are further to: generate the scale-normalized three-dimensional pose estimate based at least on a set of kinematic elements of the occupant detected using the one or more features by the at least one three-dimensional pose detection model. ### Claim 6 (depends on 3) The system of claim 3, wherein the one or more processing units are further to: apply the two-dimensional optical image data to at least one person detection model to define the portion of an image frame representing the occupant; and generate the scale-normalized three-dimensional pose estimate based at least on the portion of the image frame representing the occupant. ### Claim 7 (depends on 1) The system of claim 1, wherein the one or more processing units are further to: perform a ray tracing to correlate at least a first pixel of the ground truth cabin depth image to the one or more features to determine the depth corresponding to the one or more features. ### Claim 8 (depends on 1) The system of claim 1, wherein the one or more processing units are further to: generate a scale-normalized three-dimensional pose estimate of the occupant based at least on the one or more features, wherein the scale-normalized three-dimensional pose estimate represents a set of kinematic elements of the occupant. ### Claim 9 (depends on 1) The system of claim 1, wherein the one or more processing units are further to: generate a range image based at least on the two-dimensional optical image data and the ground truth cabin depth image, wherein the depth corresponding to the one or more features is derived at least from the range image. ### Claim 10 (depends on 9) The system of claim 9, wherein the one or more processing units are further to: generate a vehicle interior mask based at least on a segmentation of the two-dimensional optical image data, wherein the vehicle interior mask comprises a boundary that outlines pixels that correspond to structural elements of the interior; and generate the range image further based on the vehicle interior mask. ### Claim 11 (depends on 1) The system of claim 1, wherein the one or more processing units are further to: generate a vehicle interior mask based at least on a segmentation of the two-dimensional optical image data, wherein the vehicle interior mask comprises a first boundary that outlines pixels corresponding to structural elements of the interior; generate an occupant mask based at least on the segmentation of the two-dimensional optical image data, wherein the occupant mask comprises a second boundary that outlines pixels corresponding to the one or more features; and generate a three-dimensional shape estimate of the occupant based at least on the occupant mask and a range image derived based at least on the two-dimensional optical image data, the ground truth cabin depth image, and the vehicle interior mask. ### Claim 12 (depends on 11) The system of claim 11, wherein the one or more processing units are further to: generate the three-dimensional shape estimate of the occupant further based at least on a pose estimate of the occupant generated from the two-dimensional optical image data. ### Claim 13 (depends on 1) The system of claim 1, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for three-dimensional assets; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system for performing deep learning operations; a system for performing real-time streaming; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; a system for performing generative AI operations; a system implemented at least partially using a language model; or a system implemented at least partially using cloud computing resources. ### Claim 14 A processor comprising: one or more processing units comprising circuitry to: generate a first representation of at least a portion of an occupant of a machine based at least on two-dimensional optical image data; determine a depth corresponding to one or more surface locations of the occupant based at least on the two-dimensional optical image data and a ground truth cabin depth image corresponding to an interior of the machine; scale the first representation of at least the portion of the occupant to a second representation of at least the portion of the occupant based at least on the depth corresponding to the one or more surface locations of the occupant, wherein the second representation is a three-dimensional representation; determine at least one characteristic representative of a size of the occupant based at least on the second representation; and control at least one operation of the machine based at least on the at least one characteristic of the size of the occupant. ### Claim 15 (depends on 14) The processor of claim 14, wherein the at least one characteristic comprises either a three-dimensional shape estimate or a three-dimensional pose estimate. ### Claim 16 (depends on 14) The processor of claim 14, wherein the one or more processing units are further to: execute at least one three-dimensional pose detection model to generate a scale-normalized three-dimensional pose estimate of the occupant based at least on one or more features representing at least the portion of the occupant of the machine based at least on the two-dimensional optical image data; and scale the scale-normalized three-dimensional pose estimate to a linear measurement scale three-dimensional pose estimate based at least on the depth corresponding to the one or more features. ### Claim 17 (depends on 14) The processor of claim 14, wherein the one or more processing units are further to: generate a range image based at least on the two-dimensional optical image data and the ground truth cabin depth image, wherein the depth corresponding to the one or more surface locations is derived at least from the range image. ### Claim 18 (depends on 14) The processor of claim 14, wherein the one or more processing units are further to: generate a vehicle interior mask based at least on a segmentation of the two-dimensional optical image data, wherein the vehicle interior mask comprises a first boundary that outlines pixels corresponding to structural elements of the interior; generate an occupant mask based at least on the segmentation of the two-dimensional optical image data, wherein the occupant mask comprises a second boundary that outlines pixels corresponding to the one or more surface locations; and generate a three-dimensional shape estimate of the occupant based at least on the occupant mask and a range image derived based at least on the two-dimensional optical image data, the ground truth cabin depth image, and the vehicle interior mask. ### Claim 19 (depends on 14) The processor of claim 14, wherein the processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for three-dimensional assets; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system for performing deep learning operations; a system for performing real-time streaming; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; a system for performing generative AI operations; a system implemented at least partially using a language model; or a system implemented at least partially using cloud computing resources. ### Claim 20 A method comprising: generating a three-dimensional representation of a size of an occupant of a machine based at least on one or more features representing at least a portion of the occupant derived from two-dimensional optical image data and a depth of the one or more features generated from the two-dimensional optical image data and a ground truth cabin depth image corresponding to an interior of the machine.