# US 11,308,312 B2 · System and method for reconstructing unoccupied 3D space > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,312 B2 - Document ID: US11308312B2 - Status: Granted - Kind: B2 - Application number: 16/277,469 - Filed: 2019-02-15 - Published: 2019-08-15 - Granted: 2022-04-19 - Earliest claimed: 2018-02-15 - Assignees: Dmai, Inc. - Applicants: DMAI, Inc. - Inventors: Nishant Shukla - Web page: https://patentgrep.com/patent/US11308312B2 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06T 7/73: using feature-based methods - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/165: {using facial parts and geometric relationships} - G06V 40/173: {face re-identification, e.g. recognising unknown faces across different face tracks} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06T 2207/30201: Face - G06T 2207/30241: Trajectory - G06T 2207/30242: Counting objects in image - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract The present teaching relates to method, system, medium, and implementations for understanding a three dimensional (3D) scene. Image data acquired by a camera at different time instances with respect to the 3D scene are received wherein the 3D scene includes a user or one or more objects. The face of the user is detected and tracked at different time instances. With respect to some of the time instances, a 2D user profile representing a region in the image data occupied by the user is generated based on a corresponding face detected and a corresponding 3D space in the 3D scene is estimated based on calibration parameters associated with the camera. Such estimated 3D space occupied by the user in the 3D scene is used to dynamically update a 3D space occupancy record of the 3D scene. ## Claims ### Claim 1 A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for understanding a three dimensional (3D) scene, the method comprising: receiving image data acquired, by a camera at different time instances, with respect to the 3D scene which includes at least one of a user and one or more objects present therein; detecting a face of the user at each of the different time instances; and with respect to each of at least some of the different time instances, generating a 2D user profile of the user based on the face detected at the time instance, wherein the 2D user profile represents a region in the image data occupied by the user at the time instance, obtaining a 3D position of the face of the user in the 3D scene corresponding to the region in the image data, determining, based on a face based human model, a 3D prism in the 3D scene at the 3D position, estimating a 3D volumetric space occupied by the user in the 3D scene based on the 3D prism, determining spatial relationships between the one or more objects and the 3D volumetric space, estimating, based on the 3D volumetric space and the spatial relationships, a 3D space occupancy of the 3D scene, and dynamically updating a 3D space occupancy record of the 3D scene to reflect the estimated 3D space occupancy of the 3D scene. ### Claim 2 (depends on 1) The method of claim 1, wherein the user is engaged in a human machine dialogue. ### Claim 3 (depends on 2) The method of claim 2, wherein the 3D scene is a dialogue scene where the user is engaged in the human machine dialogue. ### Claim 4 (depends on 1) The method of claim 1, further comprising: detecting, from the image data, the one or more objects; and estimating at least one feature associated with each of the one or more objects. ### Claim 5 (depends on 4) The method of claim 4, further comprising determining a 3D pose of each of the one or more objects. ### Claim 6 (depends on 5) The method of claim 5, further comprising updating the 3D space occupancy record based on the 3D pose of each of the detected one or more objects. ### Claim 7 (depends on 1) The method of claim 1, further comprising determining a trajectory of the user in the 3D scene based on the at least some 3D spaces estimated based on detected user face. ### Claim 8 Machine readable and non-transitory medium having information recorded thereon for understanding a three dimensional (3D) scene, wherein the information, when read by the machine, causes the machine to perform: receiving image data acquired, by a camera at different time instances, with respect to the 3D scene which includes at least one of a user and one or more objects present therein; detecting a face of the user at each of the different time instances; and with respect to each of at least some of the different time instances, generating a 2D user profile of the user based on the face detected at the time instance, wherein the 2D user profile represents a region in the image data occupied by the user at the time instance, obtaining a 3D position of the face of the user in the 3D scene corresponding to the region in the image data, determining, based on a face based human model, a 3D prism in the 3D scene at the 3D position, estimating a 3D volumetric space occupied by the user in the 3D scene based on the 3D prism, determining spatial relationships between the one or more objects and the 3D volumetric space, estimating, based on the 3D volumetric space and the spatial relationships, a 3D space occupancy of the 3D scene, and dynamically updating a 3D space occupancy record of the 3D scene to reflect the estimated 3D space occupancy of the 3D scene. ### Claim 9 (depends on 8) The medium of claim 8, wherein the user is engaged in a human machine dialogue. ### Claim 10 (depends on 9) The medium of claim 9, wherein the 3D scene is a dialogue scene where the user is engaged in the human machine dialogue. ### Claim 11 (depends on 8) The medium of claim 8, wherein the information, when read by the machine, further causes the machine to perform: detecting, from the image data, the one or more objects; and estimating at least one feature associated with each of the one or more objects. ### Claim 12 (depends on 11) The medium of claim 11, wherein the information, when read by the machine, further causes the machine to perform determining a 3D pose of each of the one or more objects. ### Claim 13 (depends on 12) The medium of claim 12, wherein the information, when read by the machine, further causes the machine to perform updating the 3D space occupancy record based on the 3D pose of each of the detected one or more objects. ### Claim 14 (depends on 8) The medium of claim 8, wherein the information, when read by the machine, further causes the machine to perform determining a trajectory of the user in the 3D scene based on the at least some 3D spaces estimated based on detected user face. ### Claim 15 A system for understanding a three dimensional (3D) scene, comprising: a face detection unit configured for receiving image data acquired, by a camera at different time instances, with respect to the 3D scene which includes at least one of a user and one or more objects present therein, and detecting a face of the user at each of the different time instances; and a faced based human tracking unit configured for, with respect to each of at least some of the different time instances, generating a 2D user profile of the user based on the face detected at the time instance, wherein the 2D user profile represents a region in the image data occupied by the user at the time instance; a human 3D occupancy estimator configured for, with respect to the at least some of the time instances, obtaining a 3D position of the face of the user in the 3D scene corresponding to the region in the image data, determining, based on a face based human model, a 3D prism in the 3D scene at the 3D position, estimating a 3D volumetric space occupied by the user in the 3D scene based on the 3D prism, and a spatial relationship identifier configured for determining spatial relationships between the one or more objects and the 3D volumetric space; and a dynamic occupancy updater configured for dynamically updating a 3D space occupancy record of the 3D scene to reflect a 3D space occupancy of the 3D scene that is estimated based on the 3D volumetric space and the spatial relationships. ### Claim 16 (depends on 15) The system of claim 15, wherein the user is engaged in a human machine dialogue. ### Claim 17 (depends on 16) The system of claim 16, wherein the 3D scene is a dialogue scene where the user is engaged in the human machine dialogue. ### Claim 18 (depends on 15) The system of claim 15, further comprising: a visual object detection unit configured for detecting, from the image data, the one or more objects; and an object property determiner configured for estimating at least one feature associated with each of the one or more objects. ### Claim 19 (depends on 18) The system of claim 18, further comprising an object pose estimator configured for determining a 3D pose of each of the one or more objects. ### Claim 20 (depends on 19) The system of claim 19, wherein the dynamic occupancy updater is further configured for updating the 3D space occupancy record based on the 3D pose of each of the detected one or more objects. ### Claim 21 (depends on 15) The system of claim 15, further comprising a face based human tracking unit configured for determining a trajectory of the user in the 3D scene based on the at least some 3D spaces estimated based on detected user face.