# US 10,311,589 B2 · Model-based three-dimensional head pose estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,589 B2 - Document ID: US10311589B2 - Status: Granted - Kind: B2 - Application number: 15/823,370 - Filed: 2017-11-27 - Published: 2018-03-15 - Granted: 2019-06-04 - Earliest claimed: 2015-08-12 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Gregory P. Meyer; Shalini Gupta; Iuri Frosio; Nagilla Dikpal Reddy; Jan Kautz - Web page: https://patentgrep.com/patent/US10311589B2 ## CPC - G06V 10/764: using classification, e.g. of video objects - G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification} - G06T 7/251: {involving models} - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 7/507: from shading (G06T7/586 takes precedence) - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/75: {involving models} - G06T 7/77: using statistical methods - G06T 2200/28: involving image processing hardware - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30201: Face - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract One embodiment of the present invention sets forth a technique for estimating a head pose of a user. The technique includes acquiring depth data associated with a head of the user and initializing each particle included in a set of particles with a different candidate head pose. The technique further includes performing one or more optimization passes that include performing at least one iterative closest point (ICP) iteration for each particle and performing at least one particle swarm optimization (PSO) iteration. Each ICP iteration includes rendering the three-dimensional reference model based on the candidate head pose associated with the particle and comparing the three-dimensional reference model to the depth data. Each PSO iteration comprises updating a global best head pose associated with the set of particles and modifying at least one candidate head pose. The technique further includes modifying a shape of the three-dimensional reference model based on depth data. ## Claims ### Claim 1 A non-transitory computer-readable medium including instructions that, when executed by a processor, cause the processor to perform the steps of: obtaining depth data associated with a head of a user; and performing at least one optimization pass that comprises: performing, based on the depth data, at least one iterative closest point (ICP) iteration operation for each particle included in a set of particles, wherein each particle included in the set of particles is associated with a different candidate head pose; performing at least one particle swarm optimization (PSO) iteration by modifying a candidate head pose associated with a particle based on a global best head pose; and modifying a shape of a three-dimensional reference model associated with the head of the user based on the candidate head pose. ### Claim 2 (depends on 1) The non-transitory computer-readable medium of claim 1, further comprising rendering the three-dimensional reference model based on a candidate head pose associated with a particle, and comparing the three-dimensional reference model to the depth data to determine at least one error value based on at least one weighting coefficient associated with at least one vertex included in the three-dimensional reference model. ### Claim 3 (depends on 2) The non-transitory computer-readable medium of claim 2, further comprising, after performing the at least one optimization pass, updating the at least one weighting coefficient based on the depth data and the global best head pose associated with the set of particles. ### Claim 4 (depends on 2) The non-transitory computer-readable medium of claim 2, wherein the three-dimensional reference model comprises a face region having a plurality of vertices, and the at least one weighting coefficient comprises a different weighting coefficient for each vertex included in the plurality of vertices. ### Claim 5 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein acquiring the depth data associated with the head of the user comprises scaling an adaptive matched filter based on at least one distance included in the depth data, and aligning the adaptive matched filter with the depth data. ### Claim 6 (depends on 5) The non-transitory computer-readable medium of claim 5, wherein the adaptive matched filter includes a head region and a shoulder region. ### Claim 7 (depends on 6) The non-transitory computer-readable storage medium of claim 6, wherein aligning the adaptive matched filter with the depth data comprises: analyzing the depth data to determine a silhouette of the user; positioning the adaptive matched filter at a plurality of locations relative to the silhouette of the user; for each of the plurality of locations: multiplying each of a plurality of values included in the adaptive matched filter by a corresponding value associated with the silhouette of the user to determine a plurality of products, and determining a sum from the plurality of products; and determining which location has the highest sum. ### Claim 8 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein modifying the shape of the three-dimensional reference model comprises estimating a difference between the shape of the three-dimensional reference model and the depth data, and updating a shape of at least a portion of the three-dimensional reference model based on the difference. ### Claim 9 (depends on 1) The non-transitory computer-readable medium of claim 1, further comprising initializing each particle included in the set of particles with a different candidate head pose associated with the three-dimensional reference model. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, further comprising: obtaining additional depth data associated with the head of the user; for each particle included in a first portion of the set of particles, initializing the particle with a different candidate head pose that is based on the global best head pose; and for each particle included in a second portion of the set of particles, initializing the particle with a different candidate head pose that is not based on the global best head pose. ### Claim 11 A method, comprising: performing at least one iterative closest point (ICP) iteration on a candidate head pose; performing at least one particle swarm optimization (PSO) iteration on the candidate head pose based on a global head pose; and modifying a shape of a three-dimensional reference model based on the candidate head pose. ### Claim 12 (depends on 11) The method of claim 11, wherein performing at least one iterative closest point (ICP) iteration on a candidate head pose comprises comparing the three-dimensional reference model to depth data associated with a head of a user to determine that at least one error value is based on at least one weighting coefficient associated with at least one vertex included in the three-dimensional reference model. ### Claim 13 (depends on 12) The method of claim 12, further comprising updating the at least one weighting coefficient based on the depth data and the global best head pose associated with a set of particles. ### Claim 14 (depends on 12) The method of claim 12, wherein the three-dimensional reference model comprises a face region having a plurality of vertices, and the at least one weighting coefficient comprises a different weighting coefficient for each vertex included in the plurality of vertices. ### Claim 15 (depends on 11) The method of claim 11, wherein modifying the shape of the three-dimensional reference model comprises estimating a difference between a shape of the three-dimensional reference model and the depth data, and updating a shape of at least a portion of the three-dimensional reference model based on the difference. ### Claim 16 (depends on 11) The method of claim 11, further comprising initializing each particle included in a set of particles with a different candidate head pose associated with the three-dimensional reference model. ### Claim 17 (depends on 16) The method of claim 16, wherein each different candidate head pose associated with the three-dimensional model comprises a rotation vector and a translation vector associated with the three-dimensional reference model. ### Claim 18 A system, comprising: at least one sensor configured to acquire depth data associated with a head of a user; and a processor that is coupled to the at least one sensor and, when executing instructions, is configured to: initialize each particle included in a set of particles with a different candidate head pose; perform at least one optimization pass that comprises: performing at least one iterative closest point (ICP) iteration for each particle included in the set of particles to modify the candidate head pose associated with each particle; performing at least one particle swarm optimization (PSO) iteration to modify at least one candidate head pose based on a global best head pose associated with the set of particles; and modifying a shape of a three-dimensional reference model based on the depth data. ### Claim 19 (depends on 18) The system of claim 18, wherein each different candidate head pose comprises a rotation vector and a translation vector associated with the three-dimensional reference model. ### Claim 20 (depends on 18) The system of claim 18, wherein: the at least one sensor is further configured to acquire additional depth data associated with the head of the user; and the processor is further configured to: for each particle included in a first portion of the set of particles, initialize the particle with a different candidate head pose that is based on the global best head pose; and for each particle included in a second portion of the set of particles, initialize the particle with a different candidate head pose that is not based on the global best head pose.