# US 12,008,792 B1 · Independently determining adjustments to bounding shapes for detected objects in image data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,792 B1 - Document ID: US12008792B1 - Status: Granted - Kind: B1 - Application number: 17/548,380 - Filed: 2021-12-10 - Granted: 2024-06-11 - Earliest claimed: 2021-12-10 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Min Bai; Jonathan Buck; Patrick Guy Haffner; Kumar Hemachandra Chellapilla; Li Erran Li - Web page: https://patentgrep.com/patent/US12008792B1 ## CPC - G06V 10/235: {based on user input or interaction} - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06N 3/08: Learning methods - G06T 7/13: Edge detection - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/82: using neural networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Adjustments to bounding shapes for detected objects in image data may be independently determined. A bounding shape for an object detected in image data may be obtained. Independently determined adjustments for one or more edges of the bounding shape may be determined according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the adjustments to the bounding shape and respective confidence scores for the adjustments. The confidence scores are evaluated with respect to a confidence threshold to determine whether to provide the adjustments to the one or more edges of the bounding shape. ## Claims ### Claim 1 A system, comprising: at least one processor; and a memory, storing program instructions that when executed by the at least one processor, causes the at least one processor to: receive a bounding box for an object detected in image data; independently determine one or more adjustments to one or more edges of the bounding box according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the one or more adjustments to the bounding box and generate respective confidence scores for the one or more adjustments to the bounding box; and determine that the confidence scores for the one or more adjustments to the one or more edges of the bounding box satisfies a confidence threshold; and display an updated version of the bounding box according to the one or more adjustments to the one or more edges of the bounding box after the determination that the confidence scores for the one or more adjustments to the one or more edges of the bounding box satisfy the confidence threshold. ### Claim 2 (depends on 1) The system of claim 1, wherein the memory stores further program instructions that when executed further cause the at least one processor to: determine that a difference between the bounding box and a version of the bounding described by the one or more adjustments is above a difference threshold; and wherein the display of the updated version of the bounding box according to the one or more adjustments to the one or more edges of the bounding box is performed after the determination that the difference between the bounding box and the version of the bounding box described by the one or more adjustments is above the difference threshold. ### Claim 3 (depends on 1) The system of claim 1, wherein the bounding box is received as draw operation via a graphical user interface, and wherein the display of the updated version of the bounding box according to the one or more adjustments to the one or more edges of the bounding box automatically updates the drawn bounding box in the graphical user interface. ### Claim 4 (depends on 1) The system of claim 1, wherein the at least one processor and the memory is implemented as part of a machine learning service offered by a provider network that displays the updated version of the bounding box via a labeling interface for the machine learning service. ### Claim 5 A method, comprising: obtaining, at an image labeling system, a bounding shape for an object detected in image data; independently determining, by the image labeling system, one or more adjustments to one or more edges of the bounding shape according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the one or more adjustments to the one or more edges of the bounding shape and generate respective confidence scores for the one or more adjustments to the bounding shape; and after determining that the respective confidence scores for the one or more adjustments to the one or more edges of the bounding shape satisfy a confidence threshold, providing, by the image labeling system, the one or more adjustments to the one or more edges of the bounding shape. ### Claim 6 (depends on 5) The method of claim 5, further comprising: determining that a difference between the bounding shape and a version of the bounding shape described by the one or more adjustments is above a difference threshold; and wherein the providing of the one or more adjustments to the one or more edges of the bounding shape is performed after the determination that the difference between the bounding box and the version of the bounding shape described by the one or more adjustments is above the difference threshold. ### Claim 7 (depends on 6) The method of claim 6, wherein the bounding shape for the object detected in image data is obtained via a validation interface. ### Claim 8 (depends on 5) The method of claim 5, wherein the bounding shape is obtained as draw operation via a graphical user interface of the image labeling system, and wherein providing the one or more adjustments to the one or more edges of the bounding shape comprises automatically updating the drawn bounding shape in the graphical user interface with an updated version of the bounding shape according to the one or more adjustments to the one or more edges of the bounding shape. ### Claim 9 (depends on 5) The method of claim 5, wherein another adjustment determined for another edge of the bounding shape different from the one or more edges is not provided after a determination that a confidence score for the other edge does not satisfy the confidence threshold. ### Claim 10 (depends on 5) The method of claim 5, wherein independently determining the one or more adjustments to the one or more edges of the second bounding shape comprises: cropping the image data, including the bounding shape for the detected object in the cropped image data; applying a convolutional neural network (CNN) machine learning model to extract features of the cropped image data at the different resolutions; combining the extracted features at the different resolutions to generate the multi-scale feature map; applying vertical pooling and a one dimensional decoder to the multi-scale feature map to generate new left and right edges for the bounding shape; applying horizontal pooling and another one dimensional decoder to the multi-scale feature map to generate new top and bottom edges for the bounding shape; and wherein the new left and right edges and the new top and bottom edges are the one or more adjustments to the bounding shape. ### Claim 11 (depends on 5) The method of claim 5, wherein the bounding shape is a polygon. ### Claim 12 (depends on 5) The method of claim 5, wherein providing the one or more adjustments to the one or more edges of the bounding shape comprises displaying an updated version of the bounding shape determined according to the one or more adjustments to the one or more edges. ### Claim 13 (depends on 5) The method of claim 5, wherein providing the one or more adjustments to the one or more edges of the bounding shape comprises storing the one or more adjustments to the one or more edges as a new version of the bounding shape as part of a training data set. ### Claim 14 One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement: obtaining a bounding shape for an object detected in image data; independently determining one or more adjustments to one or more edges of the bounding shape according to a multi-scale feature map generated from different resolutions of the image data that is provided as input to different dimension decoders to determine the one or more adjustments to the bounding shape and generate respective confidence scores for the one or more adjustments to the bounding shape; and determining that respective confidence scores for the one or more adjustments to the one or more edges of the bounding shape satisfies a confidence threshold; and displaying the one or more adjustments to the one or more edges of the bounding shape after determining that the confidence scores for the one or more adjustments to the one or more edges of the bounding shape satisfy the confidence threshold. ### Claim 15 (depends on 14) The one or more non-transitory, computer-readable storage media of claim 14, storing further instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement: determining that a difference between the bounding shape and a version of the bounding shape described by the one or more adjustments is above a difference threshold; and wherein the displaying the one or more adjustments to the one or more edges of the bounding shape is performed after the determination that the difference between the bounding shape and the version of the bounding shape described by the one or more adjustments is above the difference threshold. ### Claim 16 (depends on 14) The one or more non-transitory, computer-readable storage media of claim 14, wherein the bounding shape is a bounding box rotated such that it is not perpendicular to axes of the image data. ### Claim 17 (depends on 14) The one or more non-transitory, computer-readable storage media of claim 14, wherein the bounding shape is obtained as draw operation via a graphical user interface of the image labeling system, and wherein, in displaying the one or more adjustments to the different edges of the bounding shape, the programming instructions cause the one or more computing devices to implement automatically updating the drawn bounding shape in the graphical user interface with an updated version of the bounding shape according to the one or more adjustments to the one or more edges of the bounding shape. ### Claim 18 (depends on 14) The one or more non-transitory, computer-readable storage media of claim 14, storing further instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement: obtaining a second bounding shape for a second object detected in second image data; independently determining one or more adjustments to one or more edges of the second bounding shape according to a second multi-scale feature map generated from different resolutions of the second image data that is provided as input to the different dimension decoders to determine the one or more adjustments to the second bounding shape and generate respective second confidence scores for the one or more adjustments to the second bounding shape; and determining that a difference between the bounding shape and an adjusted version of the bounding shape determined from the one or more adjustments to the one or more edges of the bounding shape is below a difference threshold; and not displaying the one or more adjustments to the one or more edges of the second bounding shape after determining that the difference between the bounding shape and an adjusted version of the bounding shape determined from the one or more adjustments to the one or more edges of the bounding shape is below the difference threshold. ### Claim 19 (depends on 14) The one or more non-transitory, computer-readable storage media of claim 14, wherein, in independently determining the one or more adjustments to the one or more edges of the second bounding shape, the programming instructions cause the one or more computing devices to implement: cropping the image data, including the bounding shape for the detected object in the cropped image data; applying a convolutional neural network (CNN) machine learning model to extract features of the cropped image data at the different resolutions; combining the extracted features at the different resolutions to generate the multi-scale feature map; applying vertical pooling and a one dimensional decoder to the multi-scale feature map to generate new left and right edges for the bounding shape; applying horizontal pooling and another one dimensional decoder to the multi-scale feature map to generate new top and bottom edges for the bounding shape; and wherein the new left and right edges and the new top and bottom edges are the one or more adjustments to the bounding shape. ### Claim 20 (depends on 14) The one or more non-transitory, computer-readable storage media of claim 14, wherein the one or more computing devices are implemented as part of a machine learning service offered by a provider network that displays the one or more adjustments to the bounding shape via a labeling interface for the machine learning service.