# US 2021/0334607 A1 · Systems and Methods for Image Classification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0334607 A1 - Document ID: US20210334607A1 - Status: Published application - Kind: A1 - Application number: 17/216,746 - Filed: 2021-03-30 - Published: 2021-10-28 - Earliest claimed: 2020-04-27 - Assignees: Molecular Devices, LLC - Applicants: Molecular Devices, LLC - Inventors: Yousef Al-Kofahi; Michael MacDonald; Asha Singanamalli; Mohammed Yousefhussien; Will Marshall - Web page: https://patentgrep.com/patent/US20210334607A1 ## CPC - G06V 10/82: using neural networks - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/2413: based on distances to training or reference patterns - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 7/11: Region-based segmentation - G06V 10/764: using classification, e.g. of video objects - G06V 10/776: Validation; Performance evaluation - G06V 10/945: {User interactive design; Environments; Toolboxes} - G06N 3/045: Combinations of networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In one aspect, a method for inspecting features of an image using an image inspection controller that includes a processor communicatively coupled to a memory is described. The method includes receiving, at the processor, an input image, performing, on the input image, one of a semantic segmentation process and an object classification process to generate an output image, and prompting a user to select between approving the displayed output image, and at least one of i) performing an additional semantic segmentation process on the displayed output image, and ii) performing an additional object classification process on the displayed output image. ## Claims ### Claim 1 A method for inspecting features of an image using an image inspection controller that includes a processor communicatively coupled to a memory, said method comprising: receiving, at the processor, an input image; performing, on the input image, one of a semantic segmentation process and an object classification process to generate an output image; wherein the semantic segmentation process comprises: annotating, by the processor, at least one segment of the input image to produce a semantic segmentation annotated image; generating, by the processor, a semantic segmentation model based on a semantic segmentation training vector derived from the semantic segmentation annotated image; applying, by the processor, the semantic segmentation model to each pixel of the input image to generate the output image; and displaying the output image; and wherein the object classification process comprises: annotating, by the processor, at least one object in an object mask to produce an object classification annotated image; generating, by the processor, an object classification model based on an object classification training vector derived from the object classification annotated image; applying, by the processor, the object classification model to the object mask to generate the output image; and displaying the output image; and prompting a user to select between approving the displayed output image, and at least one of i) performing an additional semantic segmentation process on the displayed output image, and ii) performing an additional object classification process on the displayed output image. ### Claim 2 (depends on 1) The method in accordance with claim 1, further comprising receiving user input indicating a selection of performing the additional semantic segmentation process on the displayed output image. ### Claim 3 (depends on 2) The method in accordance with claim 2, further comprising performing the additional semantic segmentation process on the displayed output image, the additional semantic segmentation process comprising generating and displaying an updated output image. ### Claim 4 (depends on 3) The method in accordance with claim 3, further comprising prompting the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image. ### Claim 5 (depends on 1) The method in accordance with claim 1, further comprising receiving user input indicating a selection of performing the additional object classification process on the displayed output image. ### Claim 6 (depends on 5) The method in accordance with claim 5, further comprising performing the additional object classification process on the displayed output image, the additional object classification process comprising generating and displaying an updated output image. ### Claim 7 (depends on 6) The method in accordance with claim 6, further comprising prompting the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image. ### Claim 8 (depends on 1) The method in accordance with claim 1, wherein annotating at least one segment of the input image or annotating at least one object in the object mask comprises annotating the input image or the object mask, based on user input, on a graphical user interface displayed on a display device communicatively coupled to the processor. ### Claim 9 An image inspection computing device comprising: a memory device; and at least one processor communicatively coupled to said memory device, wherein said at least one processor is configured to: receive an input image; perform, on the input image, one of a semantic segmentation process and an object classification process to generate an output image; wherein the semantic segmentation process comprises: annotating, by the processor, at least one segment of the input image to produce a semantic segmentation annotated image; generating, by the processor, a semantic segmentation model based on a semantic segmentation training vector derived from the semantic segmentation annotated image; applying, by the processor, the semantic segmentation model to each pixel of the input image to generate the output image; and displaying the output image; and wherein the object classification process comprises: annotating, by the processor, at least one object in an object mask to produce an object classification annotated image; generating, by the processor, an object classification model based on an object classification training vector derived from the object classification annotated image; applying, by the processor, the object classification model to the object mask to generate the output image; and displaying the output image; and prompt a user to select between approving the displayed output image, and at least one of i) performing an additional semantic segmentation process on the displayed output image, and ii) performing an additional object classification process on the displayed output image. ### Claim 10 (depends on 9) The image inspection computing device in accordance with claim 9, wherein said processor is further configured to receive user input indicating a selection of performing the additional semantic segmentation process on the displayed output image. ### Claim 11 (depends on 10) The image inspection computing device in accordance with claim 10, wherein said processor is further configured to perform the additional semantic segmentation process on the displayed output image, the additional semantic segmentation process including generating and displaying an updated output image. ### Claim 12 (depends on 11) The image inspection computing device in accordance with claim 11, wherein said processor is further configured to prompt the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image. ### Claim 13 (depends on 9) The image inspection computing device in accordance with claim 9, wherein said processor is further configured to receive user input indicating a selection of performing the additional object classification process on the displayed output image. ### Claim 14 (depends on 13) The image inspection computing device in accordance with claim 13, wherein said processor is further configured to perform the additional object classification process on the displayed output image, the additional object classification process including generating and displaying an updated revised output image. ### Claim 15 (depends on 14) The image inspection computing device in accordance with claim 14, wherein said processor is further configured to prompt the user to select between approving the displayed updated output image, and at least one of i) performing a further semantic segmentation process on the displayed updated output image, and ii) performing a further object classification process on the displayed updated output image. ### Claim 16 (depends on 9) The image inspection computing device in accordance with claim 9, wherein to annotate at least one segment of the input image or to annotate at least one object in the object mask, said processor is configured to annotate the input image or the object mask, based on user input, on a graphical user interface displayed on a display device communicatively coupled to said processor. ### Claim 17 A non-transitory computer-readable storage media having computer-executable instructions embodied thereon, wherein when executed by a computing device comprising at least one processor in communication with a memory, the computer-executable instructions cause the computing device to: receive, at the processor, an input image; perform, on the input image, one of a semantic segmentation process and an object classification process to generate an output image; wherein the semantic segmentation process comprises: annotating, by the processor, at least one segment of the input image to produce a semantic segmentation annotated image; generating, by the processor, a semantic segmentation model based on a semantic segmentation training vector derived from the semantic segmentation annotated image; applying, by the processor, the semantic segmentation model to each pixel of the input image to generate the output image; and displaying the output image; and wherein the object classification process comprises: annotating, by the processor, at least one object in an object mask to produce an object classification annotated image; generating, by the processor, an object classification model based on an object classification training vector derived from the object classification annotated image; applying, by the processor, the object classification model to the object mask to generate the output image; and displaying the output image; and prompt a user to select between approving the displayed output image, and at least one of i) performing an additional semantic segmentation process on the displayed output image, and ii) performing an additional object classification process on the displayed output image. ### Claim 18 (depends on 17) The non-transitory computer-readable storage media in accordance with claim 17, wherein to annotate at least one segment of the input image, the computer-executable instructions cause the computing device to receive user input indicating a selection of performing the additional semantic segmentation process on the displayed output image. ### Claim 19 (depends on 17) The non-transitory computer-readable storage media in accordance with claim 17, wherein the computer-executable instructions cause the computing device to convert the object mask to a semantic segmentation mask. ### Claim 20 (depends on 17) The non-transitory computer-readable storage media in accordance with claim 17, wherein the computer-executable instructions cause the computing device to convert a semantic segmentation mask to the object mask.