# US 12,112,557 B2 · Method and system for processing captured image data of a microbial culture medium and recognizing colony forming unit (CFU) from the captured image data using pre-learned deep learning model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,557 B2 - Document ID: US12112557B2 - Status: Granted - Kind: B2 - Application number: 17/390,000 - Filed: 2021-07-30 - Published: 2022-08-11 - Granted: 2024-10-08 - Earliest claimed: 2021-02-05 - Applicants: ByoungJin Park - Inventors: ByoungJin Park - Web page: https://patentgrep.com/patent/US12112557B2 ## CPC - G06V 20/695: {Preprocessing, e.g. image segmentation} - G06N 3/04: Architecture, e.g. interconnection topology - G06T 3/147: using affine transformations - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 5/20: using local operators - G06T 5/92: based on global image properties - G06T 7/0012: {Biomedical image inspection} - G06V 20/698: {Matching; Classification} - G06T 2207/10056: Microscopic image - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro - G06T 2207/30242: Counting objects in image - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract This application relates to a method for processing image data of a microbial culture medium to recognize colony forming unit (CFU). In one aspect, the method includes receiving, at a processor, captured image data of the microbial culture medium from a user device, and preprocessing, at the processor, the captured image data. The method may also include counting, at the processor, the number of CFUs included in the preprocessed image data to derive result data including the counted number of CFUs. The method may further include automatically inputting information included in the result data into a predetermined template to generate document data corresponding to the captured image data, and transmitting at least one of the result data or the document data to the user device. ## Claims ### Claim 1 A method for processing image data of a microbial culture medium to recognize colony forming unit (CFU), the method comprising: subjecting, at a processor, a plurality of deep learning models to learning to generate a plurality of pre-learned deep learning models; receiving, at the processor, captured image data of the microbial culture medium from a user device; preprocessing, at the processor, the captured image data, the preprocessing comprising one or more of: adjusting a size of the captured image data, adjusting a brightness and shading of the captured image data to be within a predetermined range, or filtering the captured image data using a filter stored in a memory; counting, at the processor, the number of CFUs included in the preprocessed image data using the plurality of pre-learned deep learning models, the counting comprising: receiving the plurality of pre-learned deep learning models, inputting the preprocessed image data into the plurality of pre-learned deep learning models, and deriving respective intermediate data for the plurality of pre-learned deep learning models, and comparing the respective intermediate data with each other and deriving result data, the result data including the counted number of CFUs, wherein the comparing comprises: determining whether result values outputted from the plurality of learned deep learning models are all identical or different from one another, and in response to the respective result values being different from one another, selecting a result value that satisfies a preset condition and deriving the result value as the result data; automatically inputting information included in the result data into a predetermined template to generate document data corresponding to the captured image data; and transmitting at least one of the result data or the document data to the user device, wherein the subjecting comprises: generating augmented image data by using one or more image transformation processes, based on the preprocessed image data; and subjecting the plurality of deep learning models to learning by using learning data including the augmented image data to generate the plurality of pre-learned deep learning models. ### Claim 2 (depends on 1) The method of claim 1, wherein the result data comprises at least one of the following: count data obtained by counting the counts of colonies formed in the microbial culture medium; coordinate data indicating locations where the colonies are formed in the microbial culture medium; or boundary data indicating boundaries of the colonies in the microbial culture medium. ### Claim 3 (depends on 2) The method of claim 2, wherein the plurality of pre-learned deep learning models receive the preprocessed image data as an input, and provide the count data or the coordinate data as an output thereof. ### Claim 4 (depends on 3) The method of claim 3, wherein each of the plurality of pre-learned deep learning models comprises: an input layer that uses the preprocessed image data as an input node; an output layer that uses the count data or the coordinate data as an output node; and one or more hidden layers arranged between the input layer and the output layer, and wherein weights of nodes and edges between the input node and the output node are updated by a learning process of the plurality of pre-learned deep learning models. ### Claim 5 (depends on 2) The method of claim 2, further comprising one or more of the following: overlapping boundaries of the colonies included in the boundary data onto the image for the microbial culture medium, and displaying them on a screen of the user device; or overlapping a bounding box created based on the coordinate data onto the image for the microbial culture medium, and displaying it on the screen of the user device. ### Claim 6 (depends on 1) The method of claim 1, further comprising: wherein the subjecting further comprises: receiving the result data corresponding to the captured image data for learning; and subjecting the plurality of deep learning models to learning by using learning data including the preprocessed image data for learning, the augmented image data and the result data to generate the plurality of pre-learned deep learning models. ### Claim 7 (depends on 6) The method of claim 6, further comprising: storing the learning data in an information collection memory; and storing the plurality of pre-learned deep learning models in a deep learning memory, wherein the counting comprises: deriving the number of the CFUs using one or more of the plurality of pre-learned deep learning models stored in the deep learning memory. ### Claim 8 (depends on 6) The method of claim 6, wherein the one or more transformation processes comprise at least one of: rotation, inversion, symmetry, or transformation of the preprocessed image data. ### Claim 9 (depends on 1) A non-transitory computer readable medium storing instructions, when executed by one or more processors, the instructions configured to cause the one or more processors to perform the method of claim 1. ### Claim 10 (depends on 9) The non-transitory computer readable medium of claim 9, wherein the result data comprises at least one of the following: count data obtained by counting the counts of colonies formed in the microbial culture medium; coordinate data indicating locations where the colonies are formed in the microbial culture medium; or boundary data indicating boundaries of the colonies in the microbial culture medium. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein each of the plurality of pre-learned deep learning models comprises: an input layer that uses the preprocessed image data as an input node; an output layer that uses the count data or the coordinate data as an output node; and one or more hidden layers arranged between the input layer and the output layer, and wherein weights of nodes and edges between the input node and the output node are updated by a learning process of the plurality of pre-learned deep learning models. ### Claim 12 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the method further comprises one or more of the following: overlapping boundaries of the colonies included in the boundary data onto the image for the microbial culture medium, and displaying them on a screen of the user device; or overlapping a bounding box created based on the coordinate data onto the image for the microbial culture medium, and displaying it on the screen of the user device. ### Claim 13 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the method further comprises: wherein the subjecting further comprises: receiving the result data corresponding to the captured image data, wherein the one or more transformation processes comprise at least one of: rotation, inversion, symmetry, or transformation of the preprocessed image data; and subjecting the plurality of deep learning models to learning by using learning data including the preprocessed image data for learning, the augmented image data and the result data to generate the plurality of pre-learned deep learning models. ### Claim 14 (depends on 13) The non-transitory computer readable medium of claim 13, wherein the method further comprises: storing the learning data in an information collection memory; and storing the plurality of pre-learned deep learning models in a deep learning memory, wherein the counting comprises deriving the number of the CFUs using one or more of the plurality of pre-learned deep learning models stored in the deep learning memory. ### Claim 15 A system for processing image data of a microbial culture medium to recognize colony forming unit (CFU), the system comprising: a processor configured to subject a plurality of deep learning models to learning to generate a plurality of pre-learned deep learning models; a preprocessing processor configured to receive and preprocess captured image data of the microbial culture medium from a user device, the preprocessing processor further configured to one or more of: adjust a size of the captured image data, adjust a brightness and shading of the captured image data to be within a predetermined range, or filter the captured image data using a filter stored in a memory; and a CFU counting processor configured to count the number of CFUs included in the preprocessed image data, the CFU counting processor further configured to: receive the plurality of pre-learned deep learning models; input the preprocessed image data into the plurality of pre-learned deep learning models, and derive respective intermediate data for the plurality of pre-learned deep learning models; compare the respective intermediate data with each other and derive result data, the result data including the counted number of CFUs, to perform the comparing, the CFU counting processor configured to: determine whether result values outputted from the plurality of pre-learned deep learning models are all identical or different from one another, and in response to the respective result values being different from one another, select a result value that satisfies a preset condition and deriving the result value as the result data; automatically input information included in the result data into a predetermined template to generate document data corresponding to the captured image data; and transmit at least one of the result data or the document data to the user device, the processor further configured to: generate augmented image data by using one or more image transformation processes, based on the preprocessed image data; and subject the plurality of deep learning models to learning by using learning data including the augmented image data to generate the plurality of pre-learned deep learning models. ### Claim 16 (depends on 15) The system of claim 15, wherein the result data comprises at least one of the following: count data obtained by counting the counts of colonies formed in the microbial culture medium; coordinate data indicating locations where the colonies are formed in the microbial culture medium; or boundary data indicating boundaries of the colonies in the microbial culture medium. ### Claim 17 (depends on 16) The system of claim 16, wherein the plurality of pre-learned deep learning models receive the preprocessed image data as an input, and provide the count data or the coordinate data as an output thereof. ### Claim 18 (depends on 16) The system of claim 16, further comprising a learning processor configured to: receive the result data corresponding to the captured image data; and subject the plurality of deep learning models to learning by using learning data including the preprocessed image data for learning, the augmented image data and the result data to generate the plurality of pre-learned deep learning models. ### Claim 19 (depends on 18) The system of claim 18, further comprising: an information collection memory configured to store the learning data; and a deep learning memory configured to store the plurality of pre-learned deep learning models, the CFU counting processor further configured to derive the number of the CFUs using one or more of the plurality of pre-learned deep learning models stored in the deep learning memory. ### Claim 20 (depends on 18) The system of claim 18, wherein the one or more transformation processes comprise at least one of: rotation, inversion, symmetry, or transformation of the preprocessed image data.