# US 2019/0172223 A1 · Optimizations for Dynamic Object Instance Detection, Segmentation, and Structure Mapping > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172223 A1 - Document ID: US20190172223A1 - Status: Published application - Kind: A1 - Application number: 15/972,035 - Filed: 2018-05-04 - Published: 2019-06-06 - Earliest claimed: 2017-12-03 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Peter Vajda; Peizhao Zhang; Fei Yang; Yanghan Wang - Web page: https://patentgrep.com/patent/US20190172223A1 ## CPC - G06T 7/73: using feature-based methods - G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation - G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06T 7/11: Region-based segmentation - G06T 7/75: {involving models} - G06V 10/764: using classification, e.g. of video objects - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20104: Interactive definition of region of interest [ROI] - G06T 2207/30196: Human being; Person - G06T 2210/12: Bounding box - G06V 20/20: in augmented reality scenes - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 2201/07: Target detection - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In one embodiment, a system accesses pose probability models for predetermined parts of a body depicted in an image. Each of the pose probability models is configured for determining a probability of the associated predetermined body part being at a location in the image. The system determines a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined body parts. The system further determines a first probability score for the candidate pose based on the pose probability models and the set of coordinates of the candidate pose. A pose representation is generated for the candidate pose using a transformation model and the candidate pose. The system determines a second probability score for the pose representation based on a pose-representation probability model. The system selects the candidate pose to represent a pose of the body based on at least the first and second probability scores. ## Claims ### Claim 1 A method comprising, by a computing system: accessing a plurality of pose probability models for a plurality of predetermined parts of a body that is depicted in an image, respectively, wherein each of the plurality of pose probability models is configured for determining a probability of the associated predetermined part of the body being at a location in the image, wherein the plurality of pose probability models is generated by a machine-learning model; determining a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined parts of the body in the image; determining a first probability score for the candidate pose based on the plurality of pose probability models and the set of coordinates of the candidate pose; generating a pose representation for the candidate pose using a transformation model and the candidate pose; determining a second probability score for the pose representation based on a pose-representation probability model; and selecting the candidate pose to represent a pose of the body depicted in the image based on at least the first probability score and the second probability score. ### Claim 2 (depends on 1) The method of claim 1, wherein each coordinate in the set of coordinates of the candidate pose is defined in a first coordinate system of the image; wherein the pose representation is defined in a first spatial dimension and is generated by applying the transformation model to a set of normalized coordinates that correspond to the set of coordinates of the candidate pose, respectively; and wherein each coordinate in the set of normalized coordinates is defined in a second coordinate system that is different from the first coordinate system. ### Claim 3 (depends on 2) The method of claim 2, further comprising: reprojecting the pose representation from the first spatial dimension into a second spatial dimension associated with the second coordinate system; and computing a reprojection error based on the reprojected pose representation and the normalized coordinates; wherein the selection of the candidate pose is further based on the reprojection error. ### Claim 4 (depends on 2) The method of claim 2, wherein the pose representation is generated based on differences between the set of normalized coordinates and an aggregate representation of a plurality of sets of normalized coordinates that are associated with a plurality of poses, respectively. ### Claim 5 (depends on 2) The method of claim 2, wherein the second coordinate system is defined relative to one or more of the predetermined parts of the body. ### Claim 6 (depends on 1) The method of claim 1, wherein each of the plurality of probability models is a probability heat map. ### Claim 7 (depends on 1) The method of claim 1, wherein the transformation model is generated using principal component analysis. ### Claim 8 (depends on 1) The method of claim 1, wherein at least one of the plurality of predetermined parts of the body corresponds to a joint of the body. ### Claim 9 A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising: accessing a plurality of pose probability models for a plurality of predetermined parts of a body that is depicted in an image, respectively, wherein each of the plurality of pose probability models is configured for determining a probability of the associated predetermined part of the body being at a location in the image, wherein the plurality of pose probability models is generated by a machine-learning model; determining a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined parts of the body in the image; determining a first probability score for the candidate pose based on the plurality of pose probability models and the set of coordinates of the candidate pose; generating a pose representation for the candidate pose using a transformation model and the candidate pose; determining a second probability score for the pose representation based on a pose-representation probability model; and selecting the candidate pose to represent a pose of the body depicted in the image based on at least the first probability score and the second probability score. ### Claim 10 (depends on 9) The system of claim 9, wherein each coordinate in the set of coordinates of the candidate pose is defined in a first coordinate system of the image; wherein the pose representation is defined in a first spatial dimension and is generated by applying the transformation model to a set of normalized coordinates that correspond to the set of coordinates of the candidate pose, respectively; and wherein each coordinate in the set of normalized coordinates is defined in a second coordinate system that is different from the first coordinate system. ### Claim 11 (depends on 10) The system of claim 10, wherein the processors are further operable when executing the instructions to perform operations comprising: reprojecting the pose representation from the first spatial dimension into a second spatial dimension associated with the second coordinate system; and computing a reprojection error based on the reprojected pose representation and the normalized coordinates; wherein the selection of the candidate pose is further based on the reprojection error. ### Claim 12 (depends on 10) The system of claim 10, wherein the pose representation is generated based on differences between the set of normalized coordinates and an aggregate representation of a plurality of sets of normalized coordinates that are associated with a plurality of poses, respectively. ### Claim 13 (depends on 10) The system of claim 10, wherein the second coordinate system is defined relative to one or more of the predetermined parts of the body. ### Claim 14 (depends on 9) The system of claim 9, wherein each of the plurality of probability models is a probability heat map. ### Claim 15 One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising: accessing a plurality of pose probability models for a plurality of predetermined parts of a body that is depicted in an image, respectively, wherein each of the plurality of pose probability models is configured for determining a probability of the associated predetermined part of the body being at a location in the image, wherein the plurality of pose probability models is generated by a machine-learning model; determining a candidate pose that is defined by a set of coordinates representing candidate locations of the predetermined parts of the body in the image; determining a first probability score for the candidate pose based on the plurality of pose probability models and the set of coordinates of the candidate pose; generating a pose representation for the candidate pose using a transformation model and the candidate pose; determining a second probability score for the pose representation based on a pose-representation probability model; and selecting the candidate pose to represent a pose of the body depicted in the image based on at least the first probability score and the second probability score. ### Claim 16 (depends on 15) The media of claim 15, wherein each coordinate in the set of coordinates of the candidate pose is defined in a first coordinate system of the image; wherein the pose representation is defined in a first spatial dimension and is generated by applying the transformation model to a set of normalized coordinates that correspond to the set of coordinates of the candidate pose, respectively; and wherein each coordinate in the set of normalized coordinates is defined in a second coordinate system that is different from the first coordinate system. ### Claim 17 (depends on 16) The media of claim 16, wherein the software is further operable when executed to cause the one or more processors to perform operations comprising: reprojecting the pose representation from the first spatial dimension into a second spatial dimension associated with the second coordinate system; and computing a reprojection error based on the reprojected pose representation and the normalized coordinates; wherein the selection of the candidate pose is further based on the reprojection error. ### Claim 18 (depends on 16) The media of claim 16, wherein the pose representation is generated based on differences between the set of normalized coordinates and an aggregate representation of a plurality of sets of normalized coordinates that are associated with a plurality of poses, respectively. ### Claim 19 (depends on 16) The media of claim 16, wherein the second coordinate system is defined relative to one or more of the predetermined parts of the body. ### Claim 20 (depends on 15) The media of claim 15, wherein each of the plurality of probability models is a probability heat map.