# US 2019/0318189 A1 · Method, System and Computer Readable Storage Medium for Identifying Information Carried on Sheet > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318189 A1 - Document ID: US20190318189A1 - Status: Published application - Kind: A1 - Application number: 16/100,940 - Filed: 2018-08-10 - Published: 2019-10-17 - Earliest claimed: 2018-04-13 - Assignees: Hangzhou Glority Software Limited - Applicants: Hangzhou Glority Software Limited - Inventors: Qingsong XU; Mingquan CHEN; Huan LUO - Web page: https://patentgrep.com/patent/US20190318189A1 ## CPC - G06V 10/82: using neural networks - G06V 30/1478: {of characters or characters lines} - G06V 30/153: {using recognition of characters or words} - G06V 30/19173: {Classification techniques} - G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G06T 2207/30176: Document - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method for identifying information carried on a sheet is disclosed. The method comprises: identifying, using one or more computing devices, each of one or more areas on the sheet based on an image of the sheet and a pre-trained first model, wherein each of the one or more areas is associated with all or part of the information carried on the sheet, and the first model is a neural network based model; and identifying, using one or more computing devices, characters in each of the one or more areas based on the image of the sheet, each of the one or more areas and a pre-trained second model so as to determine the information carried on the sheet, wherein the second model is a neural network based model. ## Claims ### Claim 1 A method for identifying information carried on a sheet, comprising: identifying, using one or more computing devices, each of one or more areas on a sheet based on an image of the sheet and a pre-trained first model, wherein each of the one or more areas is associated with all or part of information carried on the sheet, and the first model is a neural network based model; and identifying, using the one or more computing devices, characters in each of the one or more areas based on the image of the sheet, each of the one or more areas, and a pre-trained second model so as to determine the information carried on the sheet, wherein the pre-trained second model is a neural network based model. ### Claim 2 (depends on 1) The method according to claim 1, wherein identifying the characters in each of the one or more areas comprises: identifying the characters in each of the one or more areas based on an image of each of the one or more areas and the pre-trained second model. ### Claim 3 (depends on 1) The method according to claim 1, wherein identifying the characters in each of the one or more areas comprises: identifying the characters in each of the one or more areas based on the image of the sheet, a position of each of the one or more areas, and the pre-trained second model. ### Claim 4 (depends on 1) The method according to claim 1, the method further comprising: identifying, using the one or more computing devices, an information type of information associated with each of the one or more areas based on the image of the sheet and the first model; and determining, using the one or more computing devices, the information carried on the sheet based on the identified information type of the information associated with each of the one or more areas and the characters identified in each of the one or more areas. ### Claim 5 (depends on 1) The method according to claim 1, further comprising: before identifying each of the one or more areas on the sheet, identifying, using the one or more computing devices, a category of the sheet based on the image of the sheet and a pre-trained third model, wherein the third model is a neural network based model; and selecting, using the one or more computing devices, the first model and/or the second model to be used based on the category identified. ### Claim 5 (depends on 5) The method according to claim 5, wherein the category comprises a language. ### Claim 7 (depends on 1) The method according to claim 1, wherein the pre-trained first model is obtained by: labeling, using the one or more computing devices, for each sheet image sample in a first set of sheet image samples, each of one or more areas in each sheet image sample, wherein each of the one or more areas in each sheet image sample is associated with all or part of information carried on the sheet image sample; and training, using the one or more computing devices, a second neural network based on the labeled first set of sheet image samples to obtain the pre-trained first model. ### Claim 8 (depends on 1) The method according to claim 1, wherein the pre-trained second model is obtained by: labeling, using the one or more computing devices, for each sheet image sample in a first set of sheet image samples, characters in each of one or more areas in each sheet image sample, wherein each of the one or more areas in each sheet image sample is associated with all or part of information carried on the sheet image sample; and training, using the one or more computing devices, a second neural network based on the labeled first set of sheet image samples to obtain the pre-trained second model. ### Claim 9 (depends on 5) The method according to claim 5, wherein the pre-trained third model is obtained by: labeling, using the one or more computing devices, for each sheet image sample in a first set of sheet image samples, a category of each sheet image sample; and training, using the one or more computing devices, a second neural network based on the labeled first set of sheet image samples to obtain the pre-trained third model. ### Claim 10 (depends on 2) The method according to claim 2, wherein an image of each of the one or more areas is outlined by a rectangle parallel to a horizontal line or a rectangle that tilts relative to a horizontal line. ### Claim 11 (depends on 10) The method according to claim 10, wherein in the case that each image of at least one of the one or more areas is outlined by a rectangle that tilts relative to a horizontal line, the method further comprising performing a tilt correction for each image of the at least one of the one or more areas before identifying the characters, wherein identifying the characters in each of the one or more areas comprising: identifying characters in the at least one of the one or more areas based on each image of the at least one of the one or more areas after the tilt correction and the pre-trained second model. ### Claim 12 (depends on 7) The method according to claim 7, wherein the second neural network is established based on a deep residual network. ### Claim 13 (depends on 8) The method according to claim 8, wherein the second neural network is established based on a recurrent neural network. ### Claim 14 (depends on 9) The method according to claim 9, wherein the second neural network is established based on a deep convolutional neural network. ### Claim 15 A system for identifying information carried on a sheet, comprising: a first model that is a neural network based model; a second model that is a neural network based model; and one or more first devices configured to: identify each of one or more areas on a sheet based on an image of the sheet and the first model, wherein each of the one or more areas is associated with all or part of information carried on the sheet; and identify characters in each of the one or more areas based on the image of the sheet, each of the one or more areas, and the second model so as to determine the information carried on the sheet. ### Claim 16 (depends on 15) The system according to claim 15, wherein identifying the characters in each of the one or more areas comprises: identifying the characters in each of the one or more areas based on an image of each of the one or more areas and the second model. ### Claim 17 (depends on 15) The system according to claim 15, wherein identifying the characters in each of the one or more areas comprises: identifying the characters in each of the one or more areas based on the image of the sheet, a position of each of the one or more areas, and the second model. ### Claim 18 (depends on 16) The system according to claim 16, wherein, an image of each of the one or more areas is outlined by a rectangle parallel to a horizontal line or a rectangle that tilts relative to a horizontal line, and the one or more first devices are further configured to perform a tilt correction for each image of the at least one of the one or more areas before identifying the characters in the case that each image of at least one of the one or more areas is outlined by a rectangle that tilts relative to a horizontal line, wherein identifying the characters in each of the one or more areas comprising: identifying characters in the at least one of the one or more areas based on each image of the at least one of the one or more areas after the tilt correction and the second model. ### Claim 19 (depends on 15) The system according to claim 15, the system further comprising: one or more second devices configured to: transmit the image of the sheet to the one or more first devices; and acquire digitalized identified information carried on the sheet from the one or more first devices. ### Claim 20 A non-transitory computer readable storage medium having computer executable instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: identify each of one or more areas on a sheet based on an image of the sheet and a pre-trained first model, wherein each of the one or more areas is associated with all or part of information carried on the sheet, and the pre-trained first model is a neural network based model; and identify characters in each of the one or more areas based on the image of the sheet, each of the one or more areas, and a pre-trained second model so as to determine the information carried on the sheet, wherein the pre-trained second model is a neural network based model.