# US 2019/0171870 A1 · Optimizations for Dynamic Object Instance Detection, Segmentation, and Structure Mapping > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0171870 A1 - Document ID: US20190171870A1 - Status: Published application - Kind: A1 - Application number: 15/971,997 - Filed: 2018-05-04 - Published: 2019-06-06 - Earliest claimed: 2017-12-03 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Peter Vajda; Peizhao Zhang; Fei Yang; Yanghan Wang - Web page: https://patentgrep.com/patent/US20190171870A1 ## CPC - G06T 7/73: using feature-based methods - G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation - G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06T 7/11: Region-based segmentation - G06T 7/75: {involving models} - G06V 10/764: using classification, e.g. of video objects - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20104: Interactive definition of region of interest [ROI] - G06T 2207/30196: Human being; Person - G06T 2210/12: Bounding box - G06V 20/20: in augmented reality scenes - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 2201/07: Target detection - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In one embodiment, a method includes a system accessing an image and generating a feature map using a first neural network. The system identifies a plurality of regions of interest in the feature map. A plurality of regional feature maps may be generated for the plurality of regions of interest, respectively. Using a second neural network, the system may detect at least one regional feature map in the plurality of regional feature maps that corresponds to a person depicted in the image, and generate a target region definition associated with a location of the person using the regional feature map. Based on the target region definition associated with the location of the person, a target regional feature map may be generated by sampling the feature map for the image. The system may process the target regional feature map to generate a keypoint mask and an instance segmentation mask. ## Claims ### Claim 1 A method comprising, by a computing system: accessing an image; generating a feature map for the image using a first neural network; identifying a plurality of regions of interest in the feature map; generating a plurality of regional feature maps for the plurality of regions of interest, respectively, by sampling the feature map for the image; processing the plurality of regional feature maps using a second neural network to: detect at least one regional feature map in the plurality of regional feature maps that corresponds to a person depicted in the image; and generate a target region definition associated with a location of the person using the regional feature map; generating, based on the target region definition associated with the location of the person, a target regional feature map by sampling the feature map for the image; and generating: a keypoint mask associated with the person by processing the target regional feature map using a third neural network; or an instance segmentation mask associated with the person by processing the target regional feature map using a fourth neural network. ### Claim 2 (depends on 1) The method of claim 1, wherein the instance segmentation mask and keypoint mask are being generated concurrently. ### Claim 3 (depends on 1) The method of claim 1, wherein the first neural network comprises four or fewer convolutional layers. ### Claim 4 (depends on 3) The method of claim 3, wherein each of the convolutional layers uses a kernel size of 3×3 or less. ### Claim 5 (depends on 1) The method of claim 1, wherein the first neural network comprises a total of one pooling layer. ### Claim 6 (depends on 1) The method of claim 1, wherein the first neural network comprises three or fewer inception modules. ### Claim 7 (depends on 6) The method of claim 6, wherein each of the inception modules performs convolutional operations with kernel sizes of 5×5 or less. ### Claim 8 (depends on 1) The method of claim 1, wherein each of the second neural network, third neural network, and fourth neural network is configured to process an input regional feature map using a total of one inception module. ### Claim 9 A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising: accessing an image; generating a feature map for the image using a first neural network; identifying a plurality of regions of interest in the feature map; generating a plurality of regional feature maps for the plurality of regions of interest, respectively, by sampling the feature map for the image; processing the plurality of regional feature maps using a second neural network to: detect at least one regional feature map in the plurality of regional feature maps that corresponds to a person depicted in the image; and generate a target region definition associated with a location of the person using the regional feature map; generating, based on the target region definition associated with the location of the person, a target regional feature map by sampling the feature map for the image; and generating: a keypoint mask associated with the person by processing the target regional feature map using a third neural network; or an instance segmentation mask associated with the person by processing the target regional feature map using a fourth neural network. ### Claim 10 (depends on 9) The system of claim 9, wherein the instance segmentation mask and keypoint mask are being generated concurrently. ### Claim 11 (depends on 9) The system of claim 9, wherein the first neural network comprises four or fewer convolutional layers. ### Claim 12 (depends on 11) The system of claim 11, wherein each of the convolutional layers uses a kernel size of 3×3 or less. ### Claim 13 (depends on 9) The system of claim 9, wherein the first neural network comprises a total of one pooling layer. ### Claim 14 (depends on 9) The system of claim 9, wherein the first neural network comprises three or fewer inception modules. ### Claim 15 One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising: accessing an image; generating a feature map for the image using a first neural network; identifying a plurality of regions of interest in the feature map; generating a plurality of regional feature maps for the plurality of regions of interest, respectively, by sampling the feature map for the image; processing the plurality of regional feature maps using a second neural network to: detect at least one regional feature map in the plurality of regional feature maps that corresponds to a person depicted in the image; and generate a target region definition associated with a location of the person using the regional feature map; generating, based on the target region definition associated with the location of the person, a target regional feature map by sampling the feature map for the image; and generating: a keypoint mask associated with the person by processing the target regional feature map using a third neural network; or an instance segmentation mask associated with the person by processing the target regional feature map using a fourth neural network. ### Claim 16 (depends on 15) The media of claim 15, wherein the instance segmentation mask and keypoint mask are being generated concurrently. ### Claim 17 (depends on 15) The media of claim 15, wherein the first neural network comprises four or fewer convolutional layers. ### Claim 18 (depends on 17) The media of claim 17, wherein each of the convolutional layers uses a kernel size of 3×3 or less. ### Claim 19 (depends on 15) The media of claim 15, wherein the first neural network comprises a total of one pooling layer. ### Claim 20 (depends on 15) The media of claim 15, wherein the first neural network comprises three or fewer inception modules.