# US 2020/0007720 A1 · User feedback for real-time checking and improving quality of scanned image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0007720 A1
- Document ID: US20200007720A1
- Status: Published application
- Kind: A1
- Application number: 16/569,531
- Filed: 2019-09-12
- Published: 2020-01-02
- Earliest claimed: 2013-12-03
- Assignees: ML Netherlands C.V.
- Applicants: ML Netherlands C.V.
- Inventors: Alexander Ilic
- Web page: https://patentgrep.com/patent/US20200007720A1

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 1/60: Memory management
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 30/224: of printed characters having additional code marks or containing code marks
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- H04N 23/62: Control of parameters via user interfaces
- H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters}
- H04N 23/635: {Region indicators; Field of view indicators}
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- H04N 23/74: by influencing the scene brightness using illuminating means
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A smartphone may be freely moved in three dimensions as it captures a stream of images of an object. Multiple image frames may be captured in different orientations and distances from the object and combined into a composite image representing an image of the object. The image frames may be formed into the composite image based on representing features of each image frame as a set of points in a three dimensional point cloud. Inconsistencies between the image frames may be adjusted when projecting respective points in the point cloud into the composite image. Quality of the image frames may be improved by processing the image frames to correct errors. Reflections and shadows may be detected and removed. Further, optical character recognition may be applied. As the scan progresses, a direction for capturing subsequent image frames is provided to a user as a real-time feedback.

## Claims

### Claim 1

A method of forming a composite image of a scene using a portable electronic device, the method comprising:
capturing a stream of image frames of a scene with the portable electronic device;
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from multiple image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences, wherein the image frames are incorporated into initial positions in the three dimensional point cloud; and
adjusting the points in the point cloud based on a bundle adjustment for a plurality of the sets of points.

### Claim 2 (depends on 1)

The method of claim 1, further comprising associating the sets of points with a common frame of reference.

### Claim 3 (depends on 2)

The method of claim 2, wherein associating the sets of points with a common frame of reference comprises projecting the sets of points into a common plane.

### Claim 4 (depends on 3)

The method of claim 3, wherein sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences comprises:
adjusting a three dimensional position associated with each set of points of the sets of points to achieve coincidence in the common plane between points of the sets of points representing the same image features.

### Claim 5 (depends on 2)

The method of claim 2, further comprising reconstructing a surface of the scene using the three-dimensional point cloud, wherein reconstructing a surface comprises one of de-warping and reflection removal.

### Claim 6 (depends on 5)

The method of claim 5, wherein reconstructing a surface of the scene further comprises determining a geometry of the scene.

### Claim 7 (depends on 6)

The method of claim 6, wherein rendering the composite image of the scene further comprises:
determining a viewpoint of the composite image based on the determined geometry of the scene; and
projecting the extracted one or more image features into the common frame of reference.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining correspondences between the multiple image frames comprises identifying overlapping portions of image frames and adjusting the relative position of said image frames so that the overlapping portions are aligned.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
rendering the composite image of the scene based on the three dimensional point cloud.

### Claim 10 (depends on 1)

The method of claim 1, wherein determining correspondences between sets of points associated with the one or more image features comprises:
comparing a set of points associated with a succeeding image frame with sets of points associated with one or more respective previously captured image frames; and
identifying corresponding points of the sets of points between each succeeding image frame and the one or more respective previously captured image frames.

### Claim 11 (depends on 1)

The method of claim 1, wherein determining correspondences between sets of points associated with the one or more image features further comprises:
estimating an epipolar geometry of each pair of image frames; and
searching, for each point of the sets of points, for a corresponding point in another image frame along an epipolar line.

### Claim 12

A portable electronic device, comprising:
computer memory;
a camera configured to capture a stream of image frames; and
a processor, in communication with the camera, configured to form the composite image of the object based on the stream of image frames by:
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from pairs of image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences; and
reconstructing surfaces based on the three dimensional point cloud so as to provide in the computer memory a three dimensional representation of the scene.

### Claim 13 (depends on 12)

The portable electronic device of claim 12, wherein sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences comprises:
adjusting a three dimensional position associated with each set of points of the sets of points to achieve coincidence in a common plane between points of the sets of points representing the same image features.

### Claim 14 (depends on 13)

The portable electronic device of claim 13, further comprising determining a geometry of scanned object based on the reconstructed surfaces.

### Claim 15 (depends on 14)

The portable electronic device of claim 14, wherein reconstructing a surface of the scene further comprises determining a geometry of the scene.

### Claim 16 (depends on 15)

The portable electronic device of claim 15, further comprising: rendering a composite image of the scene by:
determining a viewpoint; and
reprojecting the three dimensional point cloud based on the viewpoint.

### Claim 17 (depends on 12)

The portable electronic device of claim 12, wherein incorporating image frames of the stream of image frames into a three dimensional point cloud comprises incorporating image frames of the stream of image frames into the three dimensional point cloud based on position information of the portable electronic device at the time the image frames were captured.

### Claim 18

At least one non-transitory, computer-readable storage medium having stored instructions for forming a composite image of an object from a stream of image frames captured by a camera of a portable electronic device, the instructions, when executed by an at least one processor, cause the processor to perform a method comprising steps of:
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from pairs of image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences; and
rendering the composite image of the scene based on the three dimensional point cloud.

### Claim 19 (depends on 18)

The at least one non-transitory, computer-readable storage medium of claim 18, wherein determining correspondences between sets of points associated with the one or more image features comprises:
comparing a set of points associated with a succeeding image frame with sets of points associated with one or more previously captured image frames; and
identifying corresponding points of the sets of points between each pair of image frames.

### Claim 20 (depends on 19)

The at least one non-transitory, computer-readable storage medium of claim 19, wherein determining correspondences between sets of points associated with the one or more image features further comprises:
estimating an epipolar geometry of each pair of image frames; and
searching, for each point of the sets of points, for a corresponding point in another image frame along an epipolar line.
