# US 12,183,011 B2 · Confidence-based segmentation of multiple units > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,011 B2 - Document ID: US12183011B2 - Status: Granted - Kind: B2 - Application number: 17/161,344 - Filed: 2021-01-28 - Published: 2021-07-29 - Granted: 2024-12-31 - Earliest claimed: 2020-01-28 - Assignees: Embodied Intelligence Inc. - Applicants: Embodied Intelligence Inc. - Inventors: YuXuan Liu; Xi Chen; Nikhil Mishra - Web page: https://patentgrep.com/patent/US12183011B2 ## CPC - G06T 7/11: Region-based segmentation - B65G 47/905: {Control arrangements} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06V 10/82: using neural networks - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30242: Counting objects in image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Various embodiments of the present technology generally relate to robotic devices and artificial intelligence. More specifically, some embodiments relate to modeling uncertainty in neural network segmentation predictions of imaged scenes having a plurality of objects. In some embodiments, a computer vision system for guiding robotic picking utilizes a method for uncertainty modeling that comprises receiving one or more images of a scene comprising a plurality of distinct objects, generating a plurality of segmentation predictions each comprising one or more object masks, identifying a predefined confidence requirement, wherein the confidence requirement identifies a minimum amount of required agreement for a region, and outputting one or more object masks based on the confidence requirement. The systems and methods disclosed herein leverage the use of a plurality of hypotheses to create a distribution of possible segmentation outcomes in order model uncertainty associated with image segmentation. ## Claims ### Claim 1 A method of operating an image segmentation system, the method comprising: collecting at least one image of a scene, wherein the scene comprises a plurality of distinct objects; providing the at least one image of the scene as input to a machine learning model trained to generate segmentation predictions; receiving an output from the machine learning model comprising a plurality of segmentation predictions for the scene, wherein each segmentation prediction of the plurality of segmentation predictions includes one or more boundaries each representing a boundary predicted to correspond to a single object, and wherein at least one of the plurality of segmentation predictions includes a plurality of boundaries each predicted to correspond to a different object of the plurality of distinct objects; determining an uncertainty associated with one or more dimensions of an object associated with each boundary predicted, wherein determining the uncertainty associated with the one or more dimensions is based on agreement between the plurality of segmentation predictions; identifying a threshold level of uncertainty associated with the predicted boundaries at which a robotic device may attempt to pick up the objects corresponding to the predicted boundaries; and outputting a final segmentation prediction comprising at least one identified boundary, wherein the at least one identified boundary is chosen based on the determined uncertainty of the predicted boundaries not exceeding the threshold level of uncertainty. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more dimensions comprise sizes of at least a portion of the object corresponding to each boundary predicted. ### Claim 3 (depends on 1) The method of claim 1, wherein the machine learning model uses a random input to generate the plurality of segmentation predictions. ### Claim 4 (depends on 1) The method of claim 1, wherein the image segmentation system comprises: a computer-imaging system comprising one or more cameras; and the robotic device comprising at least one picking element. ### Claim 5 (depends on 4) The method of claim 4, further comprising, based on the final segmentation prediction, directing the robotic device to attempt to pick up an object of the plurality of distinct objects using the at least one picking element. ### Claim 6 (depends on 5) The method of claim 5, further comprising, determining that the robotic device successfully picked up the object using the picking element. ### Claim 7 A system comprising: one or more computer-readable storage media; a processing system operatively coupled to the one or more computer-readable storage media; and program instructions, stored on the one or more computer-readable storage media, wherein the program instructions, when read and executed by the processing system, direct the processing system to: collect at least one image of a scene, wherein the scene comprises a plurality of distinct objects; provide the at least one image of the scene as input to a machine learning model trained to generate segmentation predictions; receive an output from the machine learning model comprising a plurality of segmentation predictions for the scene, wherein each segmentation prediction of the plurality of segmentation predictions includes one or more boundaries each representing a boundary predicted to correspond to a single object, and wherein at least one of the plurality of segmentation predictions includes a plurality of boundaries each predicted to correspond to a different object of the plurality of distinct objects; determine an uncertainty associated with one or more dimensions of an object associated with each boundary predicted, wherein uncertainty is based on agreement between the plurality of segmentation predictions; identify a threshold level of uncertainty associated with the predicted boundaries at which a robotic device may attempt to pick up the objects corresponding to the predicted boundaries; and output a final segmentation prediction comprising at least one identified boundary, wherein the at least one identified boundary is chosen based on the determined uncertainty of the predicted boundaries not exceeding the threshold level of uncertainty. ### Claim 8 (depends on 7) The system of claim 7, wherein the one or more dimensions comprise sizes of at least a portion of the object corresponding to each boundary predicted. ### Claim 9 (depends on 7) The system of claim 7, wherein the machine learning model uses a random input to generate the plurality of segmentation predictions. ### Claim 10 (depends on 7) The system of claim 7, further comprising: a computer-imaging system comprising one or more cameras; and the robotic device comprising at least one picking element. ### Claim 11 (depends on 10) The system of claim 10, wherein the program instructions, when read and executed by the processing system, further direct the processing system to, based on the final segmentation prediction, direct the robotic device to attempt to pick up an object of the plurality of distinct objects using the at least one picking element. ### Claim 12 (depends on 11) The system of claim 11, wherein the program instructions, when read and executed by the processing system, further direct the processing system to determine that the robotic device successfully picked up the object using the at least one picking element. ### Claim 13 One or more non-transitory computer-readable storage media having program instructions stored thereon to generate segmentation predictions, wherein the program instructions, when read and executed by a processing system, direct the processing system to at least: collect at least one image of a scene, wherein the scene comprises a plurality of distinct objects; provide the at least one image of the scene as input to a machine learning model trained to generate segmentation predictions; receive an output from the machine learning model comprising a plurality of segmentation predictions for the scene, wherein each segmentation prediction of the plurality of segmentation predictions includes one or more boundaries each representing a boundary predicted to correspond to a single object, and wherein at least one of the plurality of segmentation predictions includes a plurality of boundaries each predicted to correspond to a different object of the plurality of distinct objects; determine an uncertainty associated with one or more dimensions of an object associated with each boundary predicted, wherein the uncertainty is based on agreement between the plurality of segmentation predictions; identify a threshold level of uncertainty associated with the predicted boundaries at which a robotic device may attempt to pick up the objects corresponding to the predicted boundaries; and output a final segmentation prediction comprising at least one identified boundary, wherein the at least one identified boundary is chosen based on the determined uncertainty of the predicted boundaries not exceeding the threshold level of uncertainty. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable storage media of claim 13, wherein the one or more dimensions comprise sizes of at least a portion of the object corresponding to each boundary predicted. ### Claim 15 (depends on 13) The one or more non-transitory computer-readable storage media of claim 13, wherein the machine learning model uses a random input to generate the plurality of segmentation predictions. ### Claim 16 (depends on 13) The one or more non-transitory computer-readable storage media of claim 13, wherein the program instructions, when read and executed by the processing system, further direct the processing system to, based on the final segmentation prediction, direct a robotic device to attempt to pick up an object of the plurality of distinct objects using at least one picking element of the robotic device. ### Claim 17 (depends on 16) The one or more non-transitory computer-readable storage media of claim 16, wherein the program instructions, when read and executed by the processing system, further direct the processing system to determine that the robotic device successfully picked up the object using the at least one picking element.