# US 10,375,279 B2 · User feedback for real-time checking and improving quality of scanned image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,375,279 B2 - Document ID: US10375279B2 - Status: Granted - Kind: B2 - Application number: 15/173,334 - Filed: 2016-06-03 - Published: 2016-12-01 - Granted: 2019-08-06 - Earliest claimed: 2013-12-03 - Assignees: ML Netherlands C.V. - Applicants: ML Netherlands C.V. - Inventors: Alexander Ilic - Web page: https://patentgrep.com/patent/US10375279B2 ## CPC - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 1/60: Memory management - G06T 11/60: Creating or editing images; Combining images with text - G06V 30/224: of printed characters having additional code marks or containing code marks - G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text - H04N 23/62: Control of parameters via user interfaces - H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters} - H04N 23/635: {Region indicators; Field of view indicators} - H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image} - H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming - H04N 23/74: by influencing the scene brightness using illuminating means - H04N 23/80: Camera processing pipelines; Components thereof - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A smartphone may be freely moved in three dimensions as it captures a stream of images of an object. Multiple image frames may be captured in different orientations and distances from the object and combined into a composite image representing an image of the object. The image frames may be formed into the composite image based on representing features of each image frame as a set of points in a three dimensional point cloud. Inconsistencies between the image frames may be adjusted when projecting respective points in the point cloud into the composite image. Quality of the image frames may be improved by processing the image frames to correct errors. Reflections and shadows may be detected and removed. Further, optical character recognition may be applied. As the scan progresses, a direction for capturing subsequent image frames is provided to a user as a real-time feedback. ## Claims ### Claim 1 A method of operating a portable electronic device, comprising a camera, to form an image of an object from a plurality of image frames captured with the camera, the method comprising: with the camera, capturing a first image frame and a second image frame at different times; with a processor, determining a value of a quality metric associated with the image of the object before a corrective action; with the processor, processing at least the first image frame of the plurality of image frames, to identify a feature in the first image frame representing an artifact associated with a relative position of the camera and a source of light; and when the artifact is detected in the first image frame, taking the corrective action, the corrective action comprising forming the image of the object with at least a portion of the second image frame, the portion of the second image frame representing a portion of the object depicted in a portion of the first image frame containing the artifact, wherein a value of the quality metric associated with the image of the object after taking the corrective action is greater than the value of the quality metric associated with the image of the object before the corrective action. ### Claim 2 (depends on 1) The method of claim 1, wherein the artifact is a reflection. ### Claim 3 (depends on 2) The method of claim 2, wherein identifying the feature comprises detecting a specular highlight. ### Claim 4 (depends on 1) The method of claim 1, wherein the artifact is a shadow. ### Claim 5 (depends on 1) The method of claim 1, wherein the corrective action comprises presenting on a user interface directions to a user to position the portable electronic device to reduce the artifact. ### Claim 6 (depends on 1) The method of claim 1, wherein: the method further comprises sequentially processing image frames of the plurality of image frames by, for a processed image frame, incorporating the processed image frame in a representation of a composite image; and the corrective action comprises replacing the first image frame with the second image frame in the representation of the composite image. ### Claim 7 (depends on 1) The method of claim 1, wherein: the method further comprises sequentially processing image frames of the plurality of image frames by, for a processed image frame, incorporating the processed image frame in a representation of a composite image; and the corrective action comprises adjusting a camera acquisition condition for a subsequent image frame incorporated in the composite image. ### Claim 8 (depends on 7) The method of claim 7 wherein adjusting the camera capture condition comprises adjusting a camera flash setting. ### Claim 9 A method of operating a portable electronic device, comprising a camera, to form an image of an object from a plurality of image frames captured with the camera, the method comprising: with the camera, capturing a first image frame and a second image frame at different times; with a processor, processing at least the first image frame of the plurality of image frames, to identify a feature in the first image frame representing an artifact associated with a relative position of the camera and a source of light; and when the artifact is detected in the first image frame, taking a corrective action, the corrective action comprising forming the image of the object with at least a portion of the second image frame, the portion of the second image frame representing a portion of the object depicted in a portion of the first image frame containing the artifact, wherein a value of a quality metric associated with the image of the object after taking the corrective action is greater than a value of the quality metric associated with the image of the object before the corrective action, wherein: processing at least the first image frame comprises processing image frames of the plurality of image frames in a sequence representing a scan of the object by: correlating the image frames in the sequence to detect a portion of the image frames representing the same item in the image frames in the sequence; and identifying the item as the artifact based on changes of shape and/or position of the item with respect to a common frame of reference from frame to frame of the image frames in the sequence. ### Claim 10 A portable electronic device, comprising: a camera configured to capture a plurality of image frames comprising a first image frame captured with the portable electronic device at a first location relative to an object and a second image frame captured with the portable electronic device at a second location relative to the object different from the first location; and a processor configured to: form an image of the object from the plurality of image frames captured with the camera; process at least the first image frame of a plurality of image frames, to identify a feature in the first image frame representing an artifact associated with a relative position of the camera and a source of light, wherein the artifact comprises a specular highlight; and when the artifact is detected in the first image frame, take a corrective action, the corrective action comprising forming the image of the object with at least a portion of the second image frame not containing a feature representing the artifact, the portion of the second image frame representing a portion of the object depicted in a portion of the first image frame containing the artifact, the corrective action further comprising presenting on a user interface directions to a user to position the portable electronic device to reduce the artifact. ### Claim 11 (depends on 10) The portable electronic device of claim 10, wherein the artifact comprises a shadow. ### Claim 12 (depends on 10) The portable electronic device of claim 10, wherein: the processor is further configured to sequentially process image frames of the plurality of image frames by, for a processed image frame, incorporating the processed image frame in a representation of a composite image; and the corrective action comprises replacing the first image frame with the second image frame in the representation of the composite image. ### Claim 13 (depends on 10) The portable electronic device of claim 10, wherein: the processor is further configured to sequentially process image frames of the plurality of image frames by, for a processed image frame, incorporating the processed image frame in a representation of a composite image; and the corrective action comprises adjusting a camera acquisition condition for a subsequent image frame incorporated in the composite image. ### Claim 14 (depends on 13) The portable electronic device of claim 13, wherein adjusting the camera capture condition comprises adjusting a camera flash setting. ### Claim 15 A portable electronic device, comprising: a camera configured to capture a plurality of image frames; and a processor configured to: form an image of an object from the plurality of image frames captured with the camera; process at least a first image frame of a plurality of image frames, to identify a feature in the first image frame representing an artifact associated with a relative position of the camera and a source of light, wherein the artifact comprises a specular highlight; and when the artifact is detected in the first image frame, take a corrective action, the corrective action comprising forming the image of the object with at least a portion of a second image frame not containing a feature representing the artifact, the portion of the second image frame representing a portion of the object depicted in a portion of the first image frame containing the artifact, the corrective action further comprising presenting on a user interface directions to a user to position the portable electronic device to reduce the artifact, wherein: processing at least the first image frame comprises processing image frames of the plurality of image frames in a sequence representing a scan of the object by: correlating the image frames in the sequence to detect a portion of the image frames representing the same item in the image frames in the sequence; and identifying the item as the artifact based on changes of shape and/or position of the item with respect to a common frame of reference from frame to frame of the image frames in the sequence. ### Claim 16 At least one non-transitory, tangible computer readable storage medium having computer-executable instructions, that when executed by a processor, perform a method of forming an image of an object from a plurality of image frames captured with a camera of a portable electronic device, the method comprising: processing at least a first image frame of the plurality of image frames, to identify a feature in the first image frame representing an artifact associated with a relative position of the camera and a source of light; and when the artifact is detected in the first image frame, taking a corrective action, the corrective action comprising forming the image of the object with at least a portion of a second image frame, the portion of the second image frame representing a portion of the object depicted in a portion of the first image frame containing the artifact, wherein processing at least the first image frame comprises processing image frames of the plurality of image frames in a sequence representing a scan of the object by: correlating the image frames in the sequence to detect a portion of the image frames representing the same item in the image frames in the sequence; and identifying the item as the artifact based on changes of shape and/or position of the item with respect to a common frame of reference from frame to frame of the image frames in the sequence. ### Claim 17 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein the artifact is a reflection. ### Claim 18 (depends on 17) The at least one non-transitory, tangible computer readable storage medium of claim 17, wherein identifying the feature comprises detecting a specular highlight. ### Claim 19 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein the artifact is a shadow. ### Claim 20 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein the corrective action comprises presenting on a user interface directions to a user to position the portable electronic device to reduce the image artifact. ### Claim 21 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein: the method further comprises sequentially processing image frames of the plurality of image frames by, for a processed image frame, incorporating the processed image frame in a representation of a composite image; and the corrective action comprises replacing the first image frame with the second image frame in the representation of the composite image. ### Claim 22 (depends on 16) The at least one non-transitory, tangible computer readable storage medium of claim 16, wherein: the method further comprises sequentially processing image frames of the plurality of image frames by, for a processed image frame, incorporating the processed image frame in a representation of a composite image; and the corrective action comprises adjusting a camera acquisition condition for a subsequent image frame incorporated in the composite image. ### Claim 23 (depends on 22) The at least one non-transitory, tangible computer readable storage medium of claim 22, wherein adjusting the camera capture condition comprises adjusting a camera flash setting.