# US 2017/0286764 A1 · Content-based detection and three dimensional geometric reconstruction of objects in image and video data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286764 A1
- Document ID: US20170286764A1
- Status: Published application
- Kind: A1
- Application number: 15/234,969
- Filed: 2016-08-11
- Published: 2017-10-05
- Earliest claimed: 2016-04-01
- Assignees: Kofax, Inc.
- Applicants: Kofax, Inc.
- Inventors: Jiyong Ma; Stephen Michael Thompson; Jan W. Amtrup
- Web page: https://patentgrep.com/patent/US20170286764A1

## CPC

- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06T 5/70: Denoising; Smoothing
- G06T 7/11: Region-based segmentation
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 30/418: Document matching, e.g. of document images
- G06T 2207/30176: Document
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems, computer program products, and techniques for detecting objects depicted in digital image data are disclosed, according to various exemplary embodiments. The inventive concepts uniquely utilize internal features to accomplish object detection, thereby avoiding reliance on detecting object edges and/or transitions between the object and other portions of the digital image data, e.g. background textures or other objects. The inventive concepts thus provide an improvement over conventional object detection since objects may be detected even when edges are obscured or not depicted in the digital image data. In one aspect, a computer-implemented method of detecting an object depicted in a digital image includes: detecting a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and projecting a location of one or more edges of the object based at least in part on the plurality of identifying features.

## Claims

### Claim 1

A computer-implemented method of detecting an object depicted in a digital image, the method comprising:
detecting, using a hardware processor, a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and
projecting, using the hardware processor, a location of one or more edges of the object based at least in part on the plurality of identifying features; and
outputting the projected location of the one or more edges of the object to at least one of: a display of a computer, and a non-transitory computer readable medium.

### Claim 2 (depends on 1)

The computer-implemented method as recited in claim 1, wherein detecting the plurality of identifying features comprises analyzing a plurality of feature vectors each corresponding to pixels within a patch of the digital image to determine whether the patch includes a sharp transition in intensity.

### Claim 3 (depends on 1)

The computer-implemented method as recited in claim 1,
wherein detecting the plurality of identifying features comprises automatic feature zone discovery; and
wherein automatic feature zone discovery comprises:
matching a plurality of pixels in the digital image to a plurality of corresponding pixels in a plurality of reference images to form a set of matching pairs, each matching pair including one pixel from the digital image and one pixel from one of the plurality of reference images; and
determining a subset of the matching pairs exhibiting a frequency within the set of matching pairs that is greater than a predetermined frequency threshold.

### Claim 4 (depends on 1)

The computer-implemented method as recited in claim 1, comprising transforming the digital image to display the projected location of the one or more edges of the object.

### Claim 5 (depends on 1)

The computer-implemented method as recited in claim 1, wherein the plurality of identifying features comprise boilerplate content.

### Claim 6 (depends on 1)

The computer-implemented method as recited in claim 1, comprising identifying a plurality of distinctive pixels within the plurality of identifying features of the object, wherein the distinctive pixels are located at positions within the digital image characterized by a sharp transition in intensity.

### Claim 7

(canceled)

### Claim 8 (depends on 1)

The computer-implemented method as recited in claim 1, comprising matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type, wherein the matching comprises determining whether the object includes distinctive pixels that correspond to distinctive pixels present in one or more of the plurality of reference images.

### Claim 9 (depends on 1)

The computer-implemented method as recited in claim 1, comprising:
matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type; and
designating as an outlier a candidate match between a distinctive pixel in the digital image and one or more candidate corresponding distinctive pixels present in one of the plurality of reference images;
wherein the outlier is designated in response to determining a distance ratio is greater than a predetermined distance ratio threshold, wherein the distance ratio is a ratio describing:
a first distance between the distinctive pixel in the digital image and a first of the one or more candidate corresponding distinctive pixels; and
a second distance between the distinctive pixel in the digital image and a second of the one or more candidate corresponding distinctive pixels.

### Claim 10 (depends on 1)

The computer-implemented method as recited in claim 1, comprising:
matching the digital image depicting the object to one of a plurality of reference images each depicting a known object type; and
designating as an outlier a candidate match between a distinctive pixel in the digital image and a candidate corresponding distinctive pixel present in one of the plurality of reference images in response to determining the candidate match is not unique.

### Claim 11 (depends on 1)

The computer-implemented method as recited in claim 1, wherein at least a portion of one or more edges of the object for which the location is projected is missing in the digital image.

### Claim 12 (depends on 1)

The computer-implemented method as recited in claim 1, wherein projecting the location of the one or more edges of the object is based on a mapping of key points within some or all of the plurality of identifying features to key points of a reference image depicting an object belonging to a same class as the object depicted in the digital image.

### Claim 13 (depends on 1)

The computer-implemented method as recited in claim 1, comprising cropping the digital image based at least in part on the projected location of the one or more edges of the object;
wherein the cropped digital image depicts a portion of a background of the digital image surrounding the object; and
wherein the method comprises detecting one or more transitions between the background and the object within the cropped digital image.

### Claim 14 (depends on 1)

The computer-implemented method as recited in claim 1, comprising:
cropping the digital image based at least in part on the projected location of the one or more edges of the object; and
classifying the object depicted within the cropped digital image.

### Claim 15 (depends on 1)

The computer-implemented method as recited in claim 1, comprising:
generating a plurality of scaled images based on the digital image, each scaled image being characterized by a different resolution;
extracting one or more feature vectors from each scaled image; and
matching one or more of the scaled images to one of a plurality of reference images, each reference image depicting a known object type and being characterized by a known resolution.

### Claim 16 (depends on 1)

The computer-implemented method as recited in claim 1, comprising:
attempting to detect the object within the digital image using a plurality of predetermined object detection models each corresponding to a known object type; and
determining a classification of the object based on a result of attempting to detect the object within the digital image using the plurality of predetermined object detection models; and
wherein the classification of the object is determined to be the known object type corresponding to one of the object detection models for which the attempt to detect the object within the digital image was successful.

### Claim 17

(canceled)

### Claim 18

A computer program product for detecting an object depicted in a digital image, comprising a non-transitory computer readable medium having stored thereon computer readable program instructions configured to cause a processor, upon execution thereof, to:
generate, using the processor, a plurality of scaled images based on the digital image, each scaled image being characterized by a different resolution;
extract, using the processor, one or more feature vectors from each scaled image;
match, using the processor, one or more of the scaled images to one of a plurality of reference images based on the one or more feature vectors, each reference image depicting a known object type and being characterized by a known resolution;
detect, using the processor, a plurality of identifying features of the object within the scaled image matched to the one of the plurality of reference images, wherein the plurality of identifying features are located internally with respect to the object;
and
project, using the processor, a location of one or more edges of the object based at least in part on the plurality of identifying features.

### Claim 19 (depends on 18)

The computer program product as recited in claim 18, comprising computer readable program instructions configured to cause the processor, upon execution thereof, to: match the digital image depicting the object to one of a plurality of reference images each depicting a known object type.

### Claim 20

A system for detecting an object depicted in a digital image, comprising a processor and logic embodied with and/or executable by the processor, the logic being configured to cause the processor, upon execution thereof, to:
detect a plurality of identifying features of the object, wherein the plurality of identifying features are located internally with respect to the object; and
project a location of one or more edges of the object based at least in part on the plurality of identifying features, wherein at least portions of at least one of the one or more edges of the object displayed in the digital image are missing from the digital image.

### Claim 21 (depends on 4)

The computer-implemented method as recited in claim 4, wherein at least portions of at least one of the one or more edges displayed in the transformed digital image are missing from the digital image.

### Claim 22 (depends on 4)

The computer-implemented method as recited in claim 4, wherein the digital image is characterized by a complex background comprising a plurality of sharp intensity transitions not corresponding to edges of the object.
