# US 12,112,569 B2 · Multi-stage neural network process for keypoint detection in an image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,569 B2 - Document ID: US12112569B2 - Status: Granted - Kind: B2 - Application number: 17/455,883 - Filed: 2021-11-19 - Published: 2022-03-10 - Granted: 2024-10-08 - Earliest claimed: 2019-12-02 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Upal Mahbub; Rakesh Nattoji Rajaram; Vasudev Bhaskaran - Web page: https://patentgrep.com/patent/US12112569B2 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06T 5/92: based on global image properties - G06T 7/11: Region-based segmentation - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Embodiments include systems and methods that may be performed by a processor of a computing device. Embodiments may be applied for keypoint detection in an image. In embodiments, the processor of the computing device may apply to an image a first-stage neural network to define and output a plurality of regions, apply to each of the plurality of regions a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions, and apply to the plurality of keypoints a third-stage neural network to determine a correction for each of the plurality of keypoints to provide corrected keypoints. ## Claims ### Claim 1 A method performed by a processor of a computing device for keypoint detection in an image, comprising: analyzing an image using a first-stage neural network to output a plurality of regions; analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions; and analyzing the plurality of keypoints using a third-stage neural network to determine a correction for the plurality of keypoints to provide corrected keypoints. ### Claim 2 (depends on 1) The method of claim 1, further comprising executing an image processing function based on the corrected keypoints. ### Claim 3 (depends on 1) The method of claim 1, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to determine one or more anchor points each comprising a center point for each of the plurality of regions. ### Claim 4 (depends on 1) The method of claim 1, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to determine a multi-dimensional global feature for the image. ### Claim 5 (depends on 1) The method of claim 1, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to derive a bounding box for each of the plurality of regions. ### Claim 6 (depends on 1) The method of claim 1, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: applying the first-stage neural network to define for each of the plurality of regions a smallest bounding box that includes keypoints to be determined from each region. ### Claim 7 (depends on 1) The method of claim 1, further comprising: identifying in the image a portion of a human body; and segmenting the portion of the human body into the plurality of regions. ### Claim 8 (depends on 1) The method of claim 1, wherein analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions comprises: applying the second-stage neural network to the image to determine a multi-dimensional local feature for each of the plurality of regions. ### Claim 9 (depends on 1) The method of claim 1, wherein analyzing the plurality of keypoints using a third-stage neural network to determine a correction for the plurality of keypoints to provide corrected keypoints comprises: applying the third-stage neural network to the plurality of keypoints and to one or more of a multi-dimensional global feature for the image and a multi-dimensional local feature for each of the plurality of regions to determine the correction for the plurality of keypoints. ### Claim 10 (depends on 1) The method of claim 1, wherein each second-stage neural network has been trained to output keypoints for its respective region. ### Claim 11 (depends on 1) The method of claim 1, wherein analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions is performed in parallel. ### Claim 12 A computing device, comprising: a memory; and a processor coupled to the memory and configured with processor-executable instructions to perform operations comprising: analyzing an image using a first-stage neural network to output a plurality of regions; analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions; and analyzing the plurality of keypoints using a third-stage neural network to determine a correction for the plurality of keypoints to provide corrected keypoints. ### Claim 13 (depends on 12) The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations further comprising executing an image processing function based on the corrected keypoints. ### Claim 14 (depends on 12) The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations such that analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to determine one or more anchor points each comprising a center point for each of the plurality of regions. ### Claim 15 (depends on 12) The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations such that analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to determine a multi-dimensional global feature for the image. ### Claim 16 (depends on 12) The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations such that analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to derive a bounding box for each of the plurality of regions. ### Claim 17 (depends on 12) The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations such that analyzing an image using a first-stage neural network to output a plurality of regions comprises: applying the first-stage neural network to define for each of the plurality of regions a smallest bounding box that includes keypoints to be determined from each region. ### Claim 18 (depends on 12) The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations further comprising: identifying in the image a portion of a human body; and segmenting the portion of the human body into the plurality of regions. ### Claim 19 (depends on 12) The computing device of claim 12, wherein the processor is configured with processor-executable instructions to perform operations such that analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions comprises: applying the second-stage neural network to the image to determine a multi-dimensional local feature for each of the plurality of regions. ### Claim 20 (depends on 12) The computing device of claim 12, wherein the processor configured with processor-executable instructions to perform operations such that analyzing the plurality of keypoints using a third-stage neural network to determine a correction for the plurality of keypoints to provide corrected keypoints comprises: applying the third-stage neural network to the plurality of keypoints and to one or more of a multi-dimensional global feature for the image and a multi-dimensional local feature for each of the plurality of regions to determine the correction for the plurality of keypoints. ### Claim 21 (depends on 12) The computing device of claim 12, wherein the processor configured with processor-executable instructions to perform operations such that analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions is performed in parallel. ### Claim 22 A non-transitory processor-readable medium having stored thereon processor-executable instructions configured to cause a processor of a computing device to perform operations comprising: analyzing an image using a first-stage neural network to output a plurality of regions; analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions; and analyzing the plurality of keypoints using a third-stage neural network to determine a correction for the plurality of keypoints to provide corrected keypoints. ### Claim 23 (depends on 22) The non-transitory processor-readable medium of claim 22, further comprising executing an image processing function based on the corrected keypoints. ### Claim 24 (depends on 22) The non-transitory processor-readable medium of claim 22, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to determine one or more anchor points each comprising a center point for each of the plurality of regions. ### Claim 25 (depends on 22) The non-transitory processor-readable medium of claim 22, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to determine a multi-dimensional global feature for the image. ### Claim 26 (depends on 22) The non-transitory processor-readable medium of claim 22, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: analyzing the image using the first-stage neural network to derive a bounding box for each of the plurality of regions. ### Claim 27 (depends on 22) The non-transitory processor-readable medium of claim 22, wherein analyzing an image using a first-stage neural network to output a plurality of regions comprises: applying the first-stage neural network to define for each of the plurality of regions a smallest bounding box that includes keypoints to be determined from each region. ### Claim 28 (depends on 22) The non-transitory processor-readable medium of claim 22, wherein analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions comprises: applying the second-stage neural network to the image to determine a multi-dimensional local feature for each of the plurality of regions. ### Claim 29 (depends on 22) The non-transitory processor-readable medium of claim 22, wherein analyzing the plurality of keypoints using a third-stage neural network to determine a correction for the plurality of keypoints to provide corrected keypoints function comprises: applying the third-stage neural network to the plurality of keypoints and to one or more of a multi-dimensional global feature for the image and a multi-dimensional local feature for each of the plurality of regions to determine the correction for the plurality of keypoints. ### Claim 30 A computing device, comprising: means for analyzing an image using a first-stage neural network to output a plurality of regions; means for analyzing each of the plurality of regions using a respective second-stage neural network to output a plurality of keypoints in each of the plurality of regions; and means for analyzing the plurality of keypoints using a third-stage neural network to determine a correction for the plurality of keypoints to provide corrected keypoints.