# US 10,503,995 B2 · Optical character recognition of series of images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,995 B2 - Document ID: US10503995B2 - Status: Granted - Kind: B2 - Application number: 15/854,348 - Filed: 2017-12-26 - Published: 2019-06-27 - Granted: 2019-12-10 - Earliest claimed: 2017-12-22 - Assignees: Abbyy Production LLC - Applicants: ABBYY Development LLC - Inventors: Aleksey Ivanovich Kalyuzhny - Web page: https://patentgrep.com/patent/US10503995B2 ## CPC - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06F 18/23: Clustering techniques - G06F 18/24: Classification techniques - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06V 10/20: Image preprocessing - G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images - G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables - G06V 30/418: Document matching, e.g. of document images - G06V 30/10: Character recognition - G06V 30/1607: {Correcting image deformation, e.g. trapezoidal deformation caused by perspective} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods for performing OCR of a series of images depicting text symbols. An example method comprises performing OCR a series of images to produce symbol sequences and corresponding symbol sequence quadrangles; identifying median string; calculating transformation of the symbol sequence quadrangles into a common coordinate system; determining distances between the transformed symbol sequence quadrangles; identifying a median symbol sequence quadrangle; displaying, using the median symbol sequence quadrangle, a resulting OCR text representing at least a portion of the original document. ## Claims ### Claim 1 A method, comprising: receiving, by a processing device, a current image of a series of images of an original document, wherein the current image at least partially overlaps with a previous image of the series of images; performing optical character recognition (OCR) of the current image to produce a current symbol sequence and a corresponding text layout, wherein the text layout of the current image comprises coordinates of a current symbol sequence quadrangle containing the symbol sequence in the current image; associating, using a coordinate transformation, the current symbol sequence with a first cluster of a plurality of clusters of previous symbol sequences, wherein each previous symbol sequence in the first cluster is produced by processing one of a plurality of previously received images of the series of images, wherein each of the previously received images has a corresponding previous image text layout comprising coordinates of a previous symbol sequence quadrangle containing the previous symbol sequence; merging the symbol sequences from the first cluster by identifying a first median string representing the first cluster of symbol sequences based on a first subset of images of the series of images, wherein the first median string is a resulting OCR text representing at least a portion of the original document; calculating transformations of the previous symbol sequence quadrangles to the current image to produce transformed previous symbol sequence quadrangles; determining distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle; identifying a median symbol sequence quadrangle for the previous symbol sequence quadrangles and the current symbol sequence quadrangle based on the distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle; and displaying, using the median symbol sequence quadrangle, a resulting OCR text representing at least a portion of the original document. ### Claim 2 (depends on 1) The method of claim 1, wherein a distance between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle represents a measure of difference in geometry of the symbol sequence quadrangles. ### Claim 3 (depends on 1) The method of claim 1 wherein the determining distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle comprises: determining projection coefficients for the current symbol sequence quadrangle into a unit square; and applying the determined projective transformation to the transformed previous symbol sequence quadrangles to obtain a converted transformed previous symbol sequence quadrangles. ### Claim 4 (depends on 3) The method of claim 3 wherein the determining distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle further comprises determining lengths of line segments connecting vertices of the converted transformed previous symbol sequence quadrangles and vertices of the unit square. ### Claim 5 (depends on 1) The method of claim 1 wherein the identifying of the median symbol sequence quadrangle is a transformed symbol sequence quadrangle, which distance from the current symbol sequence quadrangle is a median to the other distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle. ### Claim 6 (depends on 4) The method of claim 4 wherein the lengths of line segments between vertices of the transformed previous symbol sequence quadrangles and vertices of the unit square are weighted. ### Claim 7 (depends on 1) The method of claim 1 wherein for each recognized symbol of the current symbol sequence the text layout of the current image comprises coordinates of a current symbol rectangle containing the symbol in the current image. ### Claim 8 (depends on 7) The method of claim 7 wherein the determining distances between the transformed symbol sequence quadrangles and the current symbol sequence quadrangle comprises normalizing the distances between the transformed symbol sequence quadrangles and the current symbol sequence quadrangles by number of symbols in the symbol sequences. ### Claim 9 (depends on 1) The method of claim 1, wherein displaying, using the median symbol sequence quadrangle, a resulting OCR text further comprises: determining symbol quadrangles for symbols of the first median string based on the median symbol sequence quadrangle; determining base lines based on the symbol quadrangles; and determining a median string quadrangle based on the symbol quadrangles. ### Claim 10 A system, comprising: a memory; a processing device, coupled to the memory, the processing device configured to: receive, by a processing device, a current image of a series of images of an original document, wherein the current image at least partially overlaps with a previous image of the series of images; perform optical character recognition (OCR) of the current image to produce a current symbol sequence and a corresponding text layout, wherein the text layout of the current image comprises coordinates of a current symbol sequence quadrangle containing the symbol sequence in the current image; associate, using a coordinate transformation, the current symbol sequence with a first cluster of a plurality of clusters of previous symbol sequences, wherein each previous symbol sequence in the first cluster is produced by processing one of a plurality of previously received images of the series of images, wherein each of the previously received images has a corresponding previous image text layout comprising coordinates of a previous symbol sequence quadrangle containing the previous symbol sequence; merge the symbol sequences from the first cluster by identifying a first median string representing the first cluster of symbol sequences based on a first subset of images of the series of images, wherein the first median string is a resulting OCR text representing at least a portion of the original document; calculate transformations of the previous symbol sequence quadrangles to the current image to produce transformed previous symbol sequence quadrangles; determine distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle; identify a median symbol sequence quadrangle for the previous symbol sequence quadrangles and the current symbol sequence quadrangle based on the distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle; and display, using the median symbol sequence quadrangle, a resulting OCR text representing at least a portion of the original document. ### Claim 11 (depends on 10) The system of claim 10, wherein a distance between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle represents a measure of difference in geometry of the symbol sequence quadrangles. ### Claim 12 (depends on 10) The system of claim 10 wherein the determining distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle comprises: determining projection coefficients for the current symbol sequence quadrangle into a unit square; and applying the determined projective transformation to the transformed previous symbol sequence quadrangles to obtain a converted transformed previous symbol sequence quadrangles. ### Claim 13 (depends on 12) The system of claim 12 wherein the determining distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle further comprises determining lengths of line segments connecting vertices of the converted transformed previous symbol sequence quadrangles and vertices of the unit square. ### Claim 14 (depends on 10) The system of claim 10 wherein the identifying of the median symbol sequence quadrangle is a transformed symbol sequence quadrangle, which distance from the current symbol sequence quadrangle is a median to the other distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle. ### Claim 15 (depends on 13) The system of claim 13 wherein the lengths of line segments between vertices of the transformed previous symbol sequence quadrangles and vertices of the unit square are weighted. ### Claim 16 (depends on 10) The system of claim 10 wherein for each recognized symbol of the current symbol sequence the text layout of the current image comprises coordinates of a current symbol rectangle containing the symbol in the current image. ### Claim 17 (depends on 16) The system of claim 16 wherein the determining distances between the transformed symbol sequence quadrangles and the current symbol sequence quadrangle comprises normalizing the distances between the transformed symbol sequence quadrangles and the current symbol sequence quadrangles by number of symbols in the symbol sequences. ### Claim 18 (depends on 10) The system of claim 10, wherein displaying, using the median symbol sequence quadrangle, a resulting OCR text further comprises: determining symbol quadrangles for symbols of the first median string based on the median symbol sequence quadrangle; determining base lines based on the symbol quadrangles; and determining a median string quadrangle based on the symbol quadrangles. ### Claim 19 A computer-readable non-transitory storage medium comprising executable instructions that, when executed by a processing device, cause the processing device to: receive, by a processing device, a current image of a series of images of an original document, wherein the current image at least partially overlaps with a previous image of the series of images; perform optical character recognition (OCR) of the current image to produce a current symbol sequence and a corresponding text layout, wherein the text layout of the current image comprises coordinates of a current symbol sequence quadrangle containing the symbol sequence in the current image; associate, using a coordinate transformation, the current symbol sequence with a first cluster of a plurality of clusters of previous symbol sequences, wherein each previous symbol sequence in the first cluster is produced by processing one of a plurality of previously received images of the series of images, wherein each of the previously received images has a corresponding previous image text layout comprising coordinates of a previous symbol sequence quadrangle containing the previous symbol sequence; merge the symbol sequences from the first cluster by identifying a first median string representing the first cluster of symbol sequences based on a first subset of images of the series of images, wherein the first median string is a resulting OCR text representing at least a portion of the original document; calculate transformations of the previous symbol sequence quadrangles to the current image to produce transformed previous symbol sequence quadrangles; determine distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle; identify a median symbol sequence quadrangle for the previous symbol sequence quadrangles and the current symbol sequence quadrangle based on the distances between the transformed previous symbol sequence quadrangles and the current symbol sequence quadrangle; and display, using the median symbol sequence quadrangle, a resulting OCR text representing at least a portion of the original document. ### Claim 20 (depends on 19) The storage medium of claim 19, wherein displaying, using the median symbol sequence quadrangle, a resulting OCR text further comprises: determining symbol quadrangles for symbols of the first median string based on the median symbol sequence quadrangle; determining base lines based on the symbol quadrangles; and determining a median string quadrangle based on the symbol quadrangles.