# US 2020/0401799 A1 · Determining experiments represented by images in documents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401799 A1 - Document ID: US20200401799A1 - Status: Published application - Kind: A1 - Application number: 16/450,490 - Filed: 2019-06-24 - Published: 2020-12-24 - Earliest claimed: 2019-06-24 - Assignees: Scinapsis Analytics Inc., dba BenchSci - Applicants: Scinapsis Analytics Inc. - Inventors: Anshuman Sahoo; Thomas Kai Him Leung; David Qixiang Chen; Elvis Mboumien Wianda - Web page: https://patentgrep.com/patent/US20200401799A1 ## CPC - G06V 30/19173: {Classification techniques} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 20/00: Machine learning - G06V 10/764: using classification, e.g. of video objects - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 20/695: {Preprocessing, e.g. image segmentation} - G06V 30/413: Classification of content, e.g. text, photographs or tables - G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G16B 20/00: ICT specially adapted for functional genomics or proteomics, e.g. genotype-phenotype associations - G16B 40/20: Supervised data analysis - G16C 20/70: Machine learning, data mining or chemometrics - G06V 30/10: Character recognition - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY - G16C: COMPUTATIONAL CHEMISTRY; CHEMOINFORMATICS; COMPUTATIONAL MATERIALS SCIENCE ## Abstract A method may include acquiring one or more image texts from an image of a document, segmenting the image into one or more sub-images using the one or more image texts, determining, by applying a machine learning model, one or more experimental techniques of one or more experiments for the one or more sub-images, and adding, to a knowledge base, one or more mappings of the one or more sub-images to the one or more experiments. ## Claims ### Claim 1 A method, comprising: acquiring one or more image texts from an image of a document; segmenting the image into one or more sub-images using the one or more image texts; determining, by applying a machine learning model, one or more experimental techniques of one or more experiments for the one or more sub-images; and adding, to a knowledge base, one or more mappings of the one or more sub-images to the one or more experiments. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining one or more bounding boxes within the image for the one or more sub-images. ### Claim 3 (depends on 2) The method of claim 2, further comprising: recognizing one or more sub-image labels in the one or more image texts; obtaining one or more bounding boxes within the image for the one or more image texts; and matching the one or more sub-images with the one or more sub-image labels using the one or more bounding boxes for the one or more sub-images and the one or more bounding boxes for the one or more sub-image labels. ### Claim 4 (depends on 3) The method of claim 3, further comprising: determining macromolecules and experimental contexts for the one or more experiments using the one or more image texts, the one or more bounding boxes for the one or more sub-images, and the one or more bounding boxes for the one or more image texts. ### Claim 5 (depends on 3) The method of claim 3, wherein recognizing the one or more sub-image labels in the one or more image texts comprises: identifying one or more sub-legend texts in legend text associated with the image; and associating the one or more sub-image labels with the one or more sub-legend texts. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining, using body text of the document, materials for the one or more experiments. ### Claim 7 (depends on 1) The method of claim 1, further comprising: classifying, by the machine learning model, a sub-image of the one or more sub-images as a filter class; and in response to classifying the sub-image as the filter class, determining that the sub-image has no recognized experimental technique. ### Claim 8 A system, comprising: a memory coupled to a computer processor; a repository configured to store: a document comprising an image comprising one or more image texts, a machine learning model, and a knowledge base; and an image analyzer, executing on the computer processor and using the memory, configured to: acquire the one or more image texts from the image, segment the image into one or more sub-images using the one or more image texts, determine, by applying the machine learning model, one or more experimental techniques of one or more experiments for the one or more sub-images, and add, to the knowledge base, one or more mappings of the one or more sub-images to the one or more experiments. ### Claim 9 (depends on 8) The system of claim 8, wherein the image analyzer is further configured to: determine one or more bounding boxes within the image for the one or more sub-images. ### Claim 10 (depends on 9) The system of claim 9, wherein the image analyzer is further configured to: recognize one or more sub-image labels in the one or more image texts; obtain one or more bounding boxes within the image for the one or more image texts; and match the one or more sub-images with the one or more sub-image labels using the one or more bounding boxes for the one or more sub-images and the one or more bounding boxes for the one or more sub-image labels. ### Claim 11 (depends on 10) The system of claim 10, wherein the image analyzer is further configured to: determine macromolecules and experimental contexts for the one or more experiments using the one or more image texts, the one or more bounding boxes for the one or more sub-images, and the one or more bounding boxes for the one or more image texts. ### Claim 12 (depends on 10) The system of claim 10, wherein the image analyzer is further configured to recognize the one or more sub-image labels in the one or more image texts by: identifying one or more sub-legend texts in legend text associated with the image; and associating the one or more sub-image labels with the one or more sub-legend texts. ### Claim 13 (depends on 8) The system of claim 8, wherein the image analyzer is further configured to: determine, using body text of the document, materials for the one or more experiments. ### Claim 14 (depends on 8) The system of claim 8, wherein the image analyzer is further configured to: classify, by the machine learning model, a sub-image of the one or more sub-images as a filter class; and in response to classifying the sub-image as the filter class, determine that the sub-image has no recognized experimental technique. ### Claim 15 A non-transitory computer readable medium comprising instructions that, when executed by a computer processor, perform: acquiring one or more image texts from an image of a document; segmenting the image into one or more sub-images using the one or more image texts; determining, by applying a machine learning model, one or more experimental techniques of one or more experiments for the one or more sub-images; and adding, to a knowledge base, one or more mappings of the one or more sub-images to the one or more experiments. ### Claim 16 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the instructions further perform: determining one or more bounding boxes within the image for the one or more sub-images. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the instructions further perform: recognizing one or more sub-image labels in the one or more image texts; obtaining one or more bounding boxes within the image for the one or more image texts; and matching the one or more sub-images with the one or more sub-image labels using the one or more bounding boxes for the one or more sub-images and the one or more bounding boxes for the one or more sub-image labels. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the instructions further perform: determining macromolecules and experimental contexts for the one or more experiments using the one or more image texts, the one or more bounding boxes for the one or more sub-images, and the one or more bounding boxes for the one or more image texts. ### Claim 19 (depends on 17) The non-transitory computer readable medium of claim 17, wherein recognizing the one or more sub-image labels in the one or more image texts comprises: identifying one or more sub-legend texts in legend text associated with the image; and associating the one or more sub-image labels with the one or more sub-legend texts. ### Claim 20 (depends on 15) The non-transitory computer readable medium of claim 15, wherein the instructions further perform: classifying, by the machine learning model, a sub-image of the one or more sub-images as a filter class; and in response to classifying the sub-image as the filter class, determining that the sub-image has no recognized experimental technique.