# US 2016/0259971 A1 · Systems and methods for detection and high-quality capture of documents on a cluttered tabletop with an automatically controlled camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259971 A1
- Document ID: US20160259971A1
- Status: Published application
- Kind: A1
- Application number: 14/637,391
- Filed: 2015-03-03
- Published: 2016-09-08
- Earliest claimed: 2015-03-03
- Assignees: Fuji Xerox Co., Ltd.
- Applicants: Fuji Xerox Co., Ltd.
- Inventors: Chelhwon Kim; Patrick Chiu; Hao Tang
- Web page: https://patentgrep.com/patent/US20160259971A1

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 30/224: of printed characters having additional code marks or containing code marks
- G06V 30/40: Document-oriented image-based pattern recognition
- H04N 1/387: Composing, repositioning or otherwise {geometrically} modifying originals
- H04N 1/3876: {Recombination of partial images to recreate the original image}
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Described are systems and methods for recognizing paper documents on a tabletop using an overhead camera mounted on pan-tilt servos. The described automated system first finds paper documents on a cluttered desk based on a text probability map, constructed using multiple images acquired at fixed grid positions, and then captures a sequence of high-resolution overlapping frames of the located document(s), which are then fused together and perspective-rectified, using computed homography, to reconstruct a high quality and fronto-parallel document image that is of sufficient quality required for optical character recognition. The extracted textual information may be used, for example, for indexing and search, document repository and/or language translation applications.

## Claims

### Claim 1

A computer-implemented method being performed in a computerized system comprising a processing unit, a memory and a camera, the camera being mounted on a turret operatively coupled to the processing unit, the computer-implemented method comprising:
a. scanning a surface with the camera to acquire a first plurality of images of the surface;
b. using the acquired first plurality of images of the surface to determine at least one location of a text on the surface;
c. capturing a second plurality of images using the camera based on the determined location of the text on the surface; and
d. extracting the text using the second captured plurality of images.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the processing unit is configured to cause the turret to move the camera during the scanning and capturing.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein in a. the first plurality of images is acquired by moving the camera to a plurality of fixed positions along a predetermined path, wherein each image of the first plurality of images corresponds to a fixed position of the plurality of fixed positions, where the image was acquired.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein a. further comprises, for each image in the first plurality of images of the surface, computing feature points and a text response map.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the each text response map is computed using a probability histogram, the probability histogram being pre-computed based on the feature points.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein a. further comprises stitching the text response maps corresponding to the first plurality of images into a single text response map based on the plurality of fixed positions corresponding to the first plurality of images.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein a. further comprises detecting text blobs in the single text response map.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein a. further comprises identifying at least one large rectangular shaped blob from the detected text blobs as the location of the text on the surface.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein in b. the second plurality of images is captured by moving the camera to a plurality of fixed positions along a predetermined path, wherein each image of the second plurality of images corresponds to a fixed position of the plurality of fixed positions, where the image was captured.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the determined location of the text on the surface is a location of a text blob and wherein the predetermined path is a center line through a bounding box of the corresponding text blob.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein the images of the second plurality of images overlap with one another.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, further comprising stitching and fusing the images of the second plurality of images to obtain a second stitched image.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, further comprising fitting lines around a boundary of a text blob in the second stitched image and estimating vanishing points.

### Claim 14 (depends on 12)

The computer-implemented method of claim 12, further comprising performing a perspective rectification on the second stitched image based on a computed homography.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein in d. the text is extracted using an optical character recognition performed on the perspective rectified second stitched image.

### Claim 16

A non-transitory computer-readable medium embodying a set of computer-executable instructions, which, when executed in a computerized system comprising a processing unit, a memory and a camera, the camera being mounted on a turret operatively coupled to the processing unit, cause the computerized system to perform a method comprising:
a. scanning a surface with the camera to acquire a first plurality of images of the surface;
b. using the acquired first plurality of images of the surface to determine at least one location of a text on the surface;
c. capturing a second plurality of images using the camera based on the determined location of the text on the surface; and
d. extracting the text using the second captured plurality of images.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the set of computer-executable instructions configures the processing unit to cause the turret to move the camera during the scanning and capturing.

### Claim 18 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein in a. the first plurality of images is acquired by moving the camera to a plurality of fixed positions along a predetermined path, wherein each image of the first plurality of images corresponds to a fixed position of the plurality of fixed positions, where the image was acquired.

### Claim 19 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein a. further comprises, for each image in the first plurality of images of the surface, computing feature points and a text response map.

### Claim 20

A computerized system comprising a processing unit, a memory and a camera, the camera being mounted on a turret operatively coupled to the processing unit, the memory storing a set of computer-executable instructions causing the computerized system to perform a method comprising:
a. scanning a surface with the camera to acquire a first plurality of images of the surface;
b. using the acquired first plurality of images of the surface to determine at least one location of a text on the surface;
c. capturing a second plurality of images using the camera based on the determined location of the text on the surface; and
d. extracting the text using the second captured plurality of images.
