# US 2018/0025222 A1 · Optical character recognition (ocr) accuracy by combining results across video frames > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025222 A1 - Document ID: US20180025222A1 - Status: Published application - Kind: A1 - Application number: 15/218,907 - Filed: 2016-07-25 - Published: 2018-01-25 - Earliest claimed: 2016-07-25 - Assignees: Intuit Inc. - Applicants: Intuit Inc. - Inventors: Vijay YELLAPRAGADA; Peijun CHIANG; Sreeneel K. MADDIKA - Web page: https://patentgrep.com/patent/US20180025222A1 ## CPC - G06Q 40/123: {Tax preparation or submission} - G06F 40/279: Recognition of textual entities - G06T 7/11: Region-based segmentation - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G06T 2207/10004: Still image; Photographic image - G06T 2207/30176: Document - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure relates to optical character recognition using captured video. According to one embodiment, using a first image in stream of images depicting a document, the device extracts text data in a portion of the document depicted in the first image and determines a first confidence level regarding an accuracy of the extracted text data. If the first confidence level satisfies a threshold value, the device saves the extracted text data as recognized content of the source document. Otherwise, the device extracts the text data from the portion of the document as depicted in one or more second images in the stream and determines a second confidence level for the text data extracted from each second image until identifying one of the second images where the second confidence level associated with the text data extracted from the identified second image satisfies the threshold value. ## Claims ### Claim 1 A method for evaluating text depicted in images of a source document, comprising: receiving a stream of digital images depicting the source document; for a first digital one of the images: extracting text data in a portion of the document as depicted in the first image, and determining a first confidence level regarding an accuracy of the extracted text data; upon determining that the first confidence level satisfies a threshold value, saving the extracted text data as recognized content of the source document; and upon determining that the first confidence level does not satisfy the threshold value: extracting the text data from the portion of the document as depicted in one or more second images in the stream of digital images, and determining a second confidence level for the text data extracted from each second image until identifying one of the second images such that the second confidence level associated with the text data extracted from the identified second image satisfies the threshold value. ### Claim 2 (depends on 1) The method of claim 1, wherein the document comprises a semi-structured document with a plurality of fields. ### Claim 3 (depends on 2) The method of claim 2, wherein extracting text data from the document comprises associating an extracted name of a field with data entered in the field. ### Claim 4 (depends on 2) The method of claim 2, further comprising: identifying, in the semi-structured document, one or more key data fields, wherein the extracting text data from the document comprises extracting text data from at least the one or more key data fields. ### Claim 5 (depends on 4) The method of claim 4, further comprising: obtaining an authoritative copy of the document from a document repository based on the extracted text data from at least the one or more key data fields. ### Claim 6 (depends on 5) The method of claim 5, further comprising: comparing a value of a first field in the authoritative copy of the document to extracted text data for the first field of the one or more fields from the document; and upon determining that the extracted text data for the first field does not match the value of the first field in the authoritative copy, generating feedback to an optical character recognition (OCR) engine identifying at least a source image, the extracted text data designated as incorrect data, and the value of the first field in the authoritative copy designated as correct data. ### Claim 7 (depends on 2) The method of claim 2, further comprising: evaluating text extracted from the source document for a first one of the fields across the first digital image and the one or more second digital images; and identifying an extracted text as a most likely value of the first field based, at least in part, on a frequency in which the identified extracted text was extracted from the first one of the fields and an average confidence level associated with the identified extracted text. ### Claim 8 (depends on 1) The method of claim 1, further comprising: requesting, from a user, a value of the recognized content; and upon determining that the user-provided value of the recognized content is different from the recognized content, generating feedback to an optical character recognition (OCR) engine identifying at least a source image, the recognized content designated as incorrect data, and the user-provided value designated as correct data. ### Claim 9 An apparatus, comprising: a processor; and a memory having instructions which, when executed by the processor, performs an operation for evaluating text depicted in images of a source document, the operation comprising: receiving a stream of digital images depicting the source document; for a first digital one of the images: extracting text data in a portion of the document as depicted in the first image, and determining a first confidence level regarding an accuracy of the extracted text data; upon determining that the first confidence level satisfies a threshold value, saving the extracted text data as recognized content of the source document; and upon determining that the first confidence level does not satisfy the threshold value: extracting the text data from the portion of the document as depicted in one or more second images in the stream of digital images, and determining a second confidence level for the text data extracted from each second image until identifying one of the second images such that the second confidence level associated with the text data extracted from the identified second image satisfies the threshold value. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the document comprises a semi-structured document with a plurality of fields. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein the operation further comprises: identifying, in the semi-structured document, one or more key data fields, wherein the extracting text data from the document comprises extracting text data from at least the one or more key data fields. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the operation further comprises: obtaining an authoritative copy of the document from a document repository based on the extracted text data from at least the one or more key data fields. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the operation further comprises: comparing a value of a first field in the authoritative copy of the document to extracted text data for the first field of the one or more fields from the document; and upon determining that the extracted text data for the first field does not match the value of the first field in the authoritative copy, generating feedback to an optical character recognition (OCR) engine identifying at least a source image, the extracted text data designated as incorrect data, and the value of the first field in the authoritative copy designated as correct data. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein the operation further comprises: evaluating text extracted from the source document for a first one of the fields across the first digital image and the one or more second digital images; and identifying an extracted text as a most likely value of the first field based, at least in part, on a frequency in which the identified extracted text was extracted from the first one of the fields and an average confidence level associated with the identified extracted text. ### Claim 15 A computer-readable medium comprising instructions which, when executed on one or more processors, performs an operation for evaluating text depicted in images of a source document, the operation comprising: receiving a stream of digital images depicting the source document; for a first digital one of the images: extracting text data in a portion of the document as depicted in the first image, and determining a first confidence level regarding an accuracy of the extracted text data; upon determining that the first confidence level satisfies a threshold value, saving the extracted text data as recognized content of the source document; and upon determining that the first confidence level does not satisfy the threshold value: extracting the text data from the portion of the document as depicted in one or more second images in the stream of digital images, and determining a second confidence level for the text data extracted from each second image until identifying one of the second images such that the second confidence level associated with the text data extracted from the identified second image satisfies the threshold value. ### Claim 16 (depends on 15) The computer-readable medium of claim 15, wherein the document comprises a semi-structured document with a plurality of fields. ### Claim 17 (depends on 16) The computer-readable medium of claim 16, wherein the operation further comprises: identifying, in the semi-structured document, one or more key data fields, wherein the extracting text data from the document comprises extracting text data from at least the one or more key data fields. ### Claim 18 (depends on 17) The computer-readable medium of claim 17, wherein the operation further comprises: obtaining an authoritative copy of the document from a document repository based on the extracted text data from at least the one or more key data fields. ### Claim 19 (depends on 18) The computer-readable medium of claim 18, wherein the operation further comprises: comparing a value of a first field in the authoritative copy of the document to extracted text data for the first field of the one or more fields from the document; and upon determining that the extracted text data for the first field does not match the value of the first field in the authoritative copy, generating feedback to an optical character recognition (OCR) engine identifying at least a source image, the extracted text data designated as incorrect data, and the value of the first field in the authoritative copy designated as correct data. ### Claim 20 (depends on 16) The computer-readable medium of claim 16, wherein the operation further comprises: evaluating text extracted from the source document for a first one of the fields across the first digital image and the one or more second digital images; and identifying an extracted text as a most likely value of the first field based, at least in part, on a frequency in which the identified extracted text was extracted from the first one of the fields and an average confidence level associated with the identified extracted text.