# US 10,455,128 B2 · User feedback for real-time checking and improving quality of scanned image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,455,128 B2 - Document ID: US10455128B2 - Status: Granted - Kind: B2 - Application number: 15/173,355 - Filed: 2016-06-03 - Published: 2016-12-08 - Granted: 2019-10-22 - Earliest claimed: 2013-12-03 - Assignees: ML Netherlands C.V. - Applicants: ML Netherlands C.V. - Inventors: Alexander Ilic - Web page: https://patentgrep.com/patent/US10455128B2 ## CPC - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 1/60: Memory management - G06T 11/60: Creating or editing images; Combining images with text - G06V 30/224: of printed characters having additional code marks or containing code marks - G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text - H04N 23/62: Control of parameters via user interfaces - H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters} - H04N 23/635: {Region indicators; Field of view indicators} - H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image} - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - H04N 23/74: by influencing the scene brightness using illuminating means - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A smartphone may be freely moved in three dimensions as it captures a stream of images of an object. Multiple image frames may be captured in different orientations and distances from the object and combined into a composite image representing an image of the object. The image frames may be formed into the composite image based on representing features of each image frame as a set of points in a three dimensional point cloud. Inconsistencies between the image frames may be adjusted when projecting respective points in the point cloud into the composite image. Quality of the image frames may be improved by processing the image frames to correct errors. Reflections and shadows may be detected and removed. Further, optical character recognition may be applied. As the scan progresses, a direction for capturing subsequent image frames is provided to a user as a real-time feedback. ## Claims ### Claim 1 A method of operating a portable electronic device, comprising a camera, to form a composite image of an object from a plurality of image frames captured with the camera, the method comprising: with a processor, processing at least a first image frame of the plurality of image frames to: identify a portion of the first image frame comprising text; perform an optical character recognition process on the portion of the first image frame; and generate a quality metric associated with the portion of the first image frame based on the optical character recognition process, wherein the quality metric is based on at least one likelihood associated with identifying one or more characters in the portion of the first image frame; based on the quality metric, taking a corrective action to impact a quality of the composite image formed from the plurality of image frames, wherein the corrective action comprises replacing the portion of the first image frame with a portion of a second image frame of the plurality of image frames in a representation of the composite image; and identifying low quality areas of the composite image, wherein identifying the portion of the first image frame comprising text is performed for the areas of the composite image other than the low quality areas, but not for the low quality areas. ### Claim 2 (depends on 1) The method of claim 1, wherein the corrective action comprises presenting on a user interface directions to a user to adjust a position of the portable electronic device relative to the object. ### Claim 3 (depends on 1) The method of claim 1, wherein: the method further comprises sequentially processing image frames of the plurality of image frames by, for a processed image frame, incorporating the processed image frame in the representation of the composite image. ### Claim 4 (depends on 3) The method of claim 3, wherein: the corrective action comprises adjusting a camera acquisition condition for a subsequent image frame incorporated in the representation of the composite image. ### Claim 5 (depends on 4) The method of claim 4, wherein adjusting the camera capture condition comprises adjusting a camera zoom setting. ### Claim 6 (depends on 4) The method of claim 4, wherein adjusting the camera capture condition comprises adjusting a resolution of the subsequent image frame. ### Claim 7 (depends on 1) The method of claim 1, wherein: the plurality of image frames comprise successive image frames of a document; and processing at least the first image frame to identify the portion of the first image frame comprises identifying a plurality of segments in the document. ### Claim 8 (depends on 2) The method of claim 2, wherein: the plurality of image frames comprise successive image frames of a document; and processing at least the first image frame to identify the portion of the first image frame comprises identifying a plurality of segments in the document. ### Claim 9 (depends on 5) The method of claim 5, wherein the second image frame is the subsequent image frame. ### Claim 10 (depends on 1) The method of claim 1, wherein: processing at least the first image frame to perform an optical character recognition process comprises generating the at least one likelihood associated with identifying one or more characters in the portion of the first image frame; and processing at least the first image frame to generate the quality metric associated with the portion of the first image frame is based, in part, on the at least one likelihood. ### Claim 11 (depends on 1) The method of claim 1, wherein the optical character recognition process is a first optical character recognition process and the quality metric is a first quality metric, the method further comprising: with the processor, processing the second image frame to: identify a portion of the second image frame comprising text; perform a second optical character recognition process on the portion of the first image frame; and generate a second quality metric associated with the portion of the second image frame; and replacing the portion of the first image frame with the portion of the second image frame only if the second quality metric is greater than the first quality metric. ### Claim 12 A portable electronic device, comprising: a camera configured to capture a plurality of image frames; a display configured to display a composite image of an object; a processor, in communication with the camera and the display, configured to form the composite image of the object based on the plurality of image frames by: identifying a portion of a first image frame of the plurality of image frames comprising text; performing an optical character recognition process on the portion of the first image frame, wherein performing the optical character recognition process comprises generating at least one likelihood associated with identifying one or more characters in the portion of the first image frame; generating a quality metric associated with the portion of the first image frame based, in part, on the at least one likelihood; based on the quality metric, taking a corrective action to impact a quality of the composite image formed from the plurality of image frames, wherein the corrective action comprises replacing the portion of the first image frame with a portion of a second image frame of the plurality of image frames in a representation of the composite image, wherein the processor is further configured to control the display to display a visual indicator of the quality metric associated with the portion of the first image frame; and identifying low quality areas of the composite image, wherein identifying the portion of the first image frame of the plurality of image frames comprising text is performed for the areas of the composite image other than the low quality areas, but not for the low quality areas. ### Claim 13 (depends on 12) The portable electronic device of claim 12, further comprising a user interface, wherein the corrective action comprises presenting on the user interface directions to a user to adjust a position of the portable electronic device relative to the object. ### Claim 14 (depends on 12) The portable electronic device of claim 12, wherein the corrective action comprises adjusting a camera acquisition condition. ### Claim 15 (depends on 12) The portable electronic device of claim 12, wherein: the plurality of image frames comprise successive image frames of a document; and identifying the portion of the first image frame comprises identifying a plurality of segments in the document. ### Claim 16 At least one non-transitory, tangible computer readable storage medium having stored instructions for forming a composite image of an object from a plurality of image frames captured by a camera of a portable electronic device, the instructions, when executed by a processor, causing the processor to perform a method comprising: identifying low quality areas of the composite image; for the areas of the composite image other than the low quality areas, but not for the low quality areas, identifying a portion of a first image frame of the plurality of image frames comprising text; performing an optical character recognition process on the portion of the first image frame; generating a quality metric associated with the portion of the first image frame based on the optical character recognition process; and based on the quality metric, taking a corrective action to impact a quality of the composite image formed from the plurality of image frames, wherein the corrective action comprises replacing the portion of the first image frame with a portion of a second image frame of the plurality of image frames in a representation of the composite image. ### Claim 17 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein the corrective action comprises presenting on a user interface of the portable electronic device directions to a user to adjust a position of the portable electronic device relative to the object. ### Claim 18 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein the corrective action comprises adjusting a camera acquisition condition. ### Claim 19 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein: performing the optical character recognition process comprises generating at least one likelihood associated with identifying one or more characters in the portion of the first image frame; and generating the quality metric associated with the portion of the first image frame is based, in part, on the at least one likelihood.