# US 2010/0073735 A1 · Camera-based document imaging

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0073735 A1
- Document ID: US20100073735A1
- Status: Published application
- Kind: A1
- Application number: 12/436,775
- Filed: 2009-05-06
- Published: 2010-03-25
- Earliest claimed: 2008-05-06
- Assignees: Compulink Management Center, Inc.
- Inventors: Martin G. Hunt; Maria A. Pavlovskaia; Logan M.K. Gordon; William W. Tipton; Trang T. Pham; Darryl H. Yong; Weiqing Gu; James O. Egan; Liangnan Wu; Kin-Chung Wong
- Web page: https://patentgrep.com/patent/US20100073735A1

## CPC

- H04N 1/00251: {with an apparatus for taking photographic images, e.g. a camera}
- G06T 3/06: Topological mapping of higher dimensional structures onto lower dimensional surfaces
- G06V 30/1463: {Orientation detection or correction, e.g. rotation of multiples of 90 degrees}
- G06V 30/1478: {of characters or characters lines}
- G06V 30/10: Character recognition
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A process and system to transform a digital photograph of a text document into a scan-quality image is disclosed. By extracting the document text from the image, and analyzing visual clues from the text, a grid is constructed over the image representing the distortions in the image. Transforming the image to straighten this grid removes distortions introduced by the camera image-capture process. Variations in lighting, the extraction of text line information, and the modeling of curved lines in the image may be corrected.

## Claims

### Claim 1

A method for processing a photographed image containing text lines comprising text characters having vertical strokes comprising:
(a) binarization using pixel normalized thresholding to identify pixels in the image that make up the text;
(b) detecting typographical features indicative of the orientation of text;
(c) fitting one or more curves to a text line;
(d) building a grid of quadrilaterals using vectors that are parallel to the direction of the text lines and vectors parallel to the direction of the vertical stroke lines;
(e) dewarping the document by stretching the image so that vectors parallel to the text lines and vectors parallel to the direction of the vertical stroke lines become orthogonal; and
(f) processing the dewarped document with an optical character recognition software.

### Claim 2 (depends on 1)

The method of claim 1, wherein the binarization process includes artifact removal that discards whole connected regions of black pixels if such a region exceeds a maximum area parameter.

### Claim 3 (depends on 1)

The method of claim 1, wherein the binarization process includes artifact removal that discards whole connected regions of black pixels if such a region is less than a minimum area parameter.

### Claim 4

A method for processing a photographed image containing text lines, the text lines comprise text characters having vertical strokes and top and bottom tip points, the method comprising:
(a) detecting the top and bottom tip points of the text lines;
(b) fitting one curve to the top tip points and one curve to the bottom tip points for each of the text lines;
(c) determining the page orientation of the photographed image by distinguishing the top and bottom portions of text lines;
(d) computing approximate orientation for each text line and removing outliners among text lines;
(e) finding vertical paragraph boundaries by determining whether the start points or end points of text lines are lined up;
(f) detecting vertical strokes in text characters by scanning in local vertical direction to obtain vertical blocks of pixels at each of the intersection point of a centroid spline of a text line with the text pixels of text characters;
(g) building a grid of quadrilaterals using vectors that are parallel to the direction of the text lines and vectors parallel to the direction of the vertical stroke lines; and
(h) dewarping the document by stretching the image so that vectors parallel to the text lines and vectors parallel to the direction of the vertical stroke lines become orthogonal.

### Claim 5 (depends on 4)

The method of claim 4 wherein the determining the page orientation of the photographed image by distinguishing the top and bottom portions of text lines step further includes choosing a representative sample of text lines whose length is close to the median length of all text lines and, for each text line in the sample, checking which side has more outliers.

### Claim 6

A method for processing a photographed image containing text lines comprising text characters having vertical strokes comprising:
(a) detecting typographical features indicative of the orientation of text;
(b) fitting one or more curves to a text line;
(c) building a grid of quadrilaterals using vectors that are parallel to the direction of the text lines and vectors parallel to the direction of the vertical stroke lines; and
(d) dewarping the document by computing for each pixel location of the output image, its corresponding location in the input image; and its pixel color and/or intensity by using one or more pixels near the corresponding location in the input image.

### Claim 7 (depends on 6)

The method of claim 6 wherein the corresponding location in the input image in step (d) is computed by modeling its x-coordinate with one mathematical function and its y-coordinate with another mathematical function.

### Claim 8 (depends on 7)

The method of claim 7 wherein the two mathematical functions are generated using a Thin Plate Splines technique.

### Claim 9 (depends on 6)

The method of claim 6 wherein the computation of correspondence for every pixel location is preceded by the generation of control points in which the correspondence is computed for a subset of pixel locations.

### Claim 10 (depends on 9)

The method of claim 9 wherein the subset of pixel locations consists of one or more points lying on one or more text lines.

### Claim 11 (depends on 9)

The method of claim 9 wherein the subset of pixel locations consists of the left and right endpoints of one or more text lines.

### Claim 12 (depends on 6)

The method of claim 6 wherein the output pixel color or intensity is computed from the four nearest pixels in the input image.

### Claim 13

A method for processing a photographed image containing text lines comprising text characters having tip points and vertical strokes comprising:
(a) detecting text regions by finding a set of pixels in the photographed image that correspond to the text characters and creating a binary image containing only said set of pixels, the set of pixels are grouped into character regions, the characters regions are grouped into text lines;
(b) detecting shape by identifying the tip points and vertical strokes of the text characters;
(c) detecting orientation of the text; and
(d) transforming based on a grid building process where the identified tip points and vertical strokes are used as a basis to identify the warping of the document.

### Claim 14 (depends on 13)

The method of claim 13 wherein the detecting shape step fits splines to the top and bottom of text lines to approximate the original document shape.

### Claim 15 (depends on 13)

The method of claim 13 wherein the detecting text regions step further comprising the following steps:
(a1) estimating the foreground text by a standard or naïve thresholding method;
(a2) removing these foreground pixels from the original image;
(a3) filling the holes left by the removal by interpolating from the remaining values that provides a new estimate for the background by removing the initial thresholding and interpolating over the holes;
(a4) thresholding based on the improved estimate of the background.

### Claim 16 (depends on 13)

The method of claim 13 wherein the transform step relies on a grid building process where the extracted features are used as a basis to identify the warping of the document.

### Claim 17 (depends on 13)

The method of claim 13 wherein the transform step relies on an optimization-problem.

### Claim 18

A computer system for processing a photographed image containing text lines comprising text characters having vertical strokes, the computer system carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause the one or more processors to perform the computer-implemented steps of:
(a) binarization using pixel normalized thresholding to identify pixels in the image that make up the text;
(b) detecting typographical features indicative of the orientation of text;
(c) fitting one or more curves to a text line;
(d) building a grid of quadrilaterals using vectors that are parallel to the direction of the text lines and vectors parallel to the direction of the vertical stroke lines;
(e) dewarping the document by stretching the image so that vectors parallel to the text lines and vectors parallel to the direction of the vertical stroke lines become orthogonal; and
(f) processing the dewarped document with an optical character recognition software.

### Claim 19

A computer system for processing a photographed image containing text lines comprising text characters having vertical strokes, the computer system carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause the one or more processors to perform the computer-implemented steps of:
(a) detecting the top and bottom tip points of the text lines;
(b) fitting one curve to the top tip points and one curve to the bottom tip points for each of the text lines;
(c) determining the page orientation of the photographed image by distinguishing the top and bottom portions of text lines;
(d) computing approximate orientation for each text line and removing outliners among text lines;
(e) finding vertical paragraph boundaries by determining whether the start points or end points of text lines are lined up;
(f) detecting vertical strokes in text characters by scanning in local vertical direction to obtain vertical blocks of pixels at each of the intersection point of a centroid spline of a text line with the text pixels of text characters;
(g) building a grid of quadrilaterals using vectors that are parallel to the direction of the text lines and vectors parallel to the direction of the vertical stroke lines; and
(h) dewarping the document by stretching the image so that vectors parallel to the text lines and vectors parallel to the direction of the vertical stroke lines become orthogonal.

### Claim 20

A computer system for processing a photographed image containing text lines comprising text characters having vertical strokes, the computer system carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause the one or more processors to perform the computer-implemented steps of:
(a) detecting text regions by finding a set of pixels in the photographed image that correspond to the text characters and creating a binary image containing only said set of pixels, the set of pixels are grouped into character regions, the characters regions are grouped into text lines;
(b) detecting shape by identifying the tip points and vertical strokes of the text characters;
(c) detecting orientation of the text; and
(d) transforming based on a grid building process where the identified tip points and vertical strokes are used as a basis to identify the warping of the document.
