# US 2020/0394452 A1 · Object detection and image cropping using a multi-detector approach

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394452 A1
- Document ID: US20200394452A1
- Status: Published application
- Kind: A1
- Application number: 17/006,636
- Filed: 2020-08-28
- Published: 2020-12-17
- Earliest claimed: 2017-11-30
- Assignees: Kofax, Inc.
- Applicants: Kofax, Inc.
- Inventors: Jiyong Ma; Stephen M. Thompson; Jan W. Amtrup
- Web page: https://patentgrep.com/patent/US20200394452A1

## CPC

- G06V 30/147: {Determination of region of interest}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/23: Clustering techniques
- G06F 18/254: {of classification results, e.g. of results related to same input data}
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06N 7/00: Computing arrangements based on specific mathematical models
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/11: Region-based segmentation
- G06T 7/12: Edge-based segmentation
- G06T 7/13: Edge detection
- G06T 7/194: involving foreground-background segmentation
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/90: Determination of colour characteristics
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 30/162: Quantising the image signal
- G06V 30/18105: {related to colour}
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/19173: {Classification techniques}
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/30176: Document
- G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
- G06V 30/10: Character recognition
- G06V 30/2253: {Recognition of characters printed with magnetic ink (G06V30/2247 takes precedence)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Computer-implemented methods for detecting objects within digital image data based on color transitions include: receiving or capturing a digital image depicting an object; sampling color information from a first plurality of pixels of the digital image, wherein each of the first plurality of pixels is located in a background region of the digital image; optionally sampling color information from a second plurality of pixels of the digital image, wherein each of the second plurality of pixels is located in a foreground region of the digital image; assigning each pixel a label of either foreground or background using an adaptive label learning process; binarizing the digital image based on the labels assigned to each pixel; detecting contour(s) within the binarized digital image; and defining edge(s) of the object based on the detected contour(s). Corresponding systems and computer program products configured to perform the inventive methods are also described.

## Claims

### Claim 1

A computer-implemented method of detecting objects within digital image data based at least in part on color transitions within the digital image data, the method comprising:
receiving or capturing a digital image depicting an object;
analyzing the digital image data using one or more color transition detectors to generate a plurality of analysis results, each color transition detector being independently configured to detect one or more objects within digital images according to a unique set of analysis parameters;
selecting an optimum object location result among the plurality of analysis results; and
either or both of:
outputting, based on the optimum object location result, a projected location of one or more edges of the object to a memory; and
rendering, based on the optimum object location result, a projected location of the one or more edges of the object on a display.

### Claim 2

A computer-implemented method of detecting objects within digital image data based at least in part on color transitions within the digital image data, the method comprising:
receiving or capturing a digital image depicting an object;
sampling color information from a first plurality of pixels of the digital image, wherein each of the first plurality of pixels is located in a background region of the digital image;
optionally sampling color information from a second plurality of pixels of the digital image, wherein each of the second plurality of pixels is located in a foreground region of the digital image;
assigning each pixel within the digital image a label of either foreground or background using an adaptive label learning process;
binarizing the digital image based on the labels assigned to each pixel;
detecting one or more contours within the binarized digital image; and
defining one or more edges of the object based on the detected contour(s).

### Claim 3 (depends on 2)

The method as recited in claim 2, wherein the adaptive label learning process comprises:
selecting or estimating at least one initial Gaussian model of a representative foreground color profile of the digital image and/or a representative background color profile of the digital image; and
performing a maximum likelihood analysis of un-labeled pixels of the digital image using the at least one initial Gaussian model.

### Claim 4 (depends on 3)

The method as recited in claim 3, wherein the adaptive learning process comprises a plurality of iterations; and
wherein, for each iteration of the adaptive learning process, one or more Gaussian models of the representative foreground color profile of the digital image and/or one or more Gaussian models of the representative background color profile of the digital image is/are updated based on a set of labels assigned to a set of pixels in an immediately previous iteration of the adaptive learning process.

### Claim 5 (depends on 4)

The method as recited in claim 4, wherein the adaptive learning process comprises performing the plurality of iterations until parameters of the one or more Gaussian models achieve convergence.

### Claim 6 (depends on 5)

The method as recited in claim 5, wherein the method is characterized by achieving convergence within about 4 to about 8 iterations of the adaptive learning process.

### Claim 7 (depends on 3)

The method as recited in claim 3, wherein the maximum likelihood analysis comprises minimizing a total potential energy across all pixels within the digital image based on a representative foreground color profile of the digital image and a representative background color profile of the digital image, wherein a potential energy of each pixel comprises:
a negative log likelihood of a Gaussian model; and
an interaction energy β describing a probability of adjacent pixels exhibiting a transition from one color to another.

### Claim 8 (depends on 7)

The method as recited in claim 7, wherein the potential energy of each pixel is defined as:
LocalEnergy(xp)=SingletonEnergy(xp)+Σq∈Neighborhood(p)DoubletonEnergy(xp,xq);
wherein the SingletonEnergy(xp) is defined as −log(PDF(xp));
wherein PDF(x) is a probability distribution function; and
wherein the DoubletonEnergy(x, y) is defined as either −β or β.

### Claim 9 (depends on 8)

The method as recited in claim 8, wherein PDF(x) is defined as:
f(x)=|2πΣ|−1/2 exp(−½(x−μ)′Σ−1(x−μ));
wherein x is a 3D color Luv vector,
wherein μ is a mean vector; and
wherein Σ is a covariance matrix.

### Claim 10 (depends on 2)

The method as recited in claim 2, further comprising performing a color space transformation on the digital image.

### Claim 11 (depends on 10)

The method as recited in claim 10, wherein the color space transformation comprises a RGB to CIELUV transformation.

### Claim 12 (depends on 10)

The method as recited in claim 10, wherein the color space transformation produces a plurality of Luv vectors; and wherein each Luv vector is modeled as a random vector in a 3D CIELUV color space.

### Claim 13 (depends on 2)

The method as recited in claim 2, comprising generating a representative foreground color profile of the digital image based on the color information sampled from the first plurality of pixels, wherein generating the representative foreground color profile comprises inverting color values of the first plurality of pixels.

### Claim 14 (depends on 2)

The method as recited in claim 2, wherein the foreground region comprises a central region of the digital image.

### Claim 15 (depends on 14)

The method as recited in claim 14, wherein the central region comprises approximately 20% of a total area of the digital image.

### Claim 16 (depends on 2)

The method as recited in claim 2, further comprising downscaling the received or captured digital image, wherein the downscaling preserves an aspect ratio of the received or captured digital image.

### Claim 17 (depends on 2)

The method as recited in claim 2, wherein the object is surrounded by either at least 2 rows of background pixels or at least 2 columns of background pixels on each side.

### Claim 18 (depends on 2)

The method as recited in claim 2, further comprising computing a segmentation confidence score for the defined edge(s) of the object using one or more measures selected from the group consisting of: edge strength, angle between adjacent edges of the object, angle between opposite edges of the object, color contrast between foreground and background of the image, a least mean squares fitness, and combinations thereof.

### Claim 19 (depends on 2)

The method as recited in claim 2, wherein a first edge of the object is defined based on a largest of the detected contours; and
wherein additional edges of the object are derived by a least mean squares fitting process.

### Claim 20

A computer program product for detecting objects within digital image data based at least in part on color transitions within the digital image data, the computer program product comprising a computer readable storage medium having embodied therewith computer readable program instructions configured to cause a processor, upon execution of the computer readable program instructions, to perform a method comprising:
receiving or capturing a digital image depicting an object;
sampling, using the processor, color information from a first plurality of pixels of the digital image, wherein each of the first plurality of pixels is located in a background region of the digital image;
optionally sampling, using the processor, color information from a second plurality of pixels of the digital image, wherein each of the second plurality of pixels is located in a foreground region of the digital image;
assigning, using the processor, each pixel within the digital image a label of either foreground or background using an adaptive label learning process;
binarizing, using the processor, the digital image based on the labels assigned to each pixel;
detecting, using the processor, one or more contours within the binarized digital image; and
defining, using the processor, one or more edges of the object based on the detected contour(s).
