# US 2022/0044055 A1 · Analyzing content of digital images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044055 A1
- Document ID: US20220044055A1
- Status: Published application
- Kind: A1
- Application number: 17/506,006
- Filed: 2021-10-20
- Published: 2022-02-10
- Earliest claimed: 2014-11-26
- Assignees: DST Technologies, Inc.
- Applicants: DST Technologies, Inc.
- Inventors: Huguens Jean; Yoriyasu Yano; Hui Peng Hu; Kuang Chen
- Web page: https://patentgrep.com/patent/US20220044055A1

## CPC

- G06F 16/5838: {using colour}
- G06F 16/5846: {using extracted text}
- G06F 16/5854: {using shape and object relationship}
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/464: {using a plurality of salient features, e.g. bag-of-words [BoW] representations}
- G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
- G06V 10/817: {by voting}
- G06V 30/418: Document matching, e.g. of document images
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, apparatuses, and embodiments related to analyzing the content of digital images. A computer extracts multiple sets of visual features, which can be keypoints, based on an image of a selected object. Each of the multiple sets of visual features is extracted by a different visual feature extractor. The computer further extracts a visual word count vector based on the image of the selected object. An image query is executed based on the extracted visual features and the extracted visual word count vector to identify one or more candidate template objects of which the selected object may be an instance. When multiple candidate template objects are identified, a matching algorithm compares the selected object with the candidate template objects to determine a particular candidate template of which the selected object is an instance.

## Claims

### Claim 1

A method for training a machine learning system on templates of paper form comprising:
receiving a digital image of an instance of a form of a plurality of form templates;
extracting, by a processor, a plurality of pixel clusters from the instance of the form based on a nearest neighbors threshold with respect to individual pixels;
generating a visual vocabulary of form features based on the plurality of pixel clusters;
generating a histogram of the instance of the form using the visual vocabulary; and
indexing the histogram in a training model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first and the second visual feature extractors are any of Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Oriented Features from Accelerated Segment Test and Rotated Binary Robust Independent Elementary Features (ORB).

### Claim 3 (depends on 1)

The method of claim 1, wherein the clustering algorithm is any of a k-means algorithm or an unsupervised learning algorithm.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first, the second, and the third classifiers are any of a SIFT classifier, a SIFT-ORB ensemble classifier, an ORB classifier, or a WORD classifier.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the histogram includes:
generating a vector the visual vocabulary; and
converting the vector into the histogram.

### Claim 6

A method comprising:
assigning a first template prediction based on a multi-dimensional descriptor vector, the multi-dimensional descriptor vector being associated with a keypoint of a first plurality of keypoints identified based on a digital image of a physical object and by use of a computer vision algorithm;
assigning a second template predictions based on a histogram of the digital image of a physical object using a visual vocabulary, the visual vocabulary is based on a keypoint of a second plurality of keypoints identified based on the digital image by use of a second computer vision algorithm;
reconciling each respective template prediction wherein the template predictions are merged where each respective template prediction is in agreement and eliminating each fewest vote receiving prediction where each respective template prediction is not in agreement; and
in response to said reconciling, classifying the physical object based on a remaining template prediction.

### Claim 7 (depends on 6)

The method of claim 6, wherein the assigning the first set of template predictions includes:
generating, by the processor, the first plurality of keypoints,
wherein the keypoint of the first plurality of keypoints includes a multi-dimensional descriptor vector that represents an image gradient at a location of the digital image associated with the keypoint of the first plurality of keypoints;
comparing, by the processor, the multi-dimensional descriptor vector that represents the image gradient to a nearest neighbor multi-dimensional descriptor vector found in a set of previously classified example digital images; and
based on the comparison, assigning, by the processor, a template prediction to the multi-dimensional descriptor vector that represents the image gradient.

### Claim 8 (depends on 6)

The method of claim 6, further comprising: assigning, by the processor implementing the first classifier, a third set of template predictions based on a keypoint of a third plurality of keypoints identified based on the digital image by use of a third computer vision algorithm, wherein the third computer vision algorithm is different from the first and the second computer vision algorithms.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first and the second computer vision algorithms are a same computer vision algorithm.

### Claim 10 (depends on 6)

The method of claim 6, wherein the assigning the second set of template predictions includes:
partitioning, by the processor, an instance of the digital image into a plurality of partitions;
generating, by the processor, the second plurality of keypoints based on a partition of the plurality of partitions,
wherein the keypoint of the second plurality of keypoints includes a multi-dimensional descriptor vector that represents an image gradient at a location of the digital image associated with the keypoint of the second plurality of keypoints;
comparing, by the processor, the multi-dimensional descriptor vector that represents the image gradient to a plurality of clusters of multi-dimensional descriptor vectors associated with keypoints in previously classified example digital images;
mapping, by the processor, the multi-dimensional descriptor vector that represents the image gradient to a nearest cluster of the plurality of clusters, based on the comparison;
generating, by the processor, the histogram that represents a histogram of most frequently mapped clusters; and
assigning a template prediction to the partition of the plurality of partitions based on a most frequently mapped cluster of the histogram.

### Claim 11 (depends on 10)

The method of claim 10, wherein the plurality of partitions include one or more of the following: the whole image, four quarters of the image, left and right halves, or top and bottom halves.

### Claim 12 (depends on 6)

The method of claim 6, wherein the classifying the physical object includes:
aggregating, by the processor, the first set of template predictions to form a first aggregate template prediction, wherein the first aggregate template prediction is based on a majority template prediction of the first set of template predictions;
aggregating, by the processor, the second set of template predictions to form a second aggregate template prediction, wherein the second aggregate template prediction is based on a majority template prediction of the second set of template predictions;
comparing, by the processor, the first and the second aggregate template predictions; and
when the first and the second aggregate template predictions match:
outputting, by the processor, the first or the second aggregate template prediction as a final template prediction, and
when the first and the second aggregate template predictions do not match:
identifying, by the processor, a set of most frequent template predictions from the first and the second set of template predictions,
retrieving a plurality of digital images of candidate object templates associated with the set of the most frequent template predictions,
comparing, by the processor, each of a plurality of digital images of candidate object templates based on keypoints identified based on the digital images,
identifying, by the processor, a top candidate template prediction from the candidate object templates based on the comparison,
calculating, by the processor, a confidence interval of the identified top candidate template prediction, and
when the confidence interval is below a threshold value, discarding the top candidate template prediction, and
when the confidence interval is at or above the threshold value, outputting, by the processor, the top candidate template prediction as the final template prediction.

### Claim 13 (depends on 6)

The method of claim 6, further comprising, receiving, via an image capture device, the digital image of the physical object.

### Claim 14 (depends on 6)

The method of claim 6, wherein the keypoints identified based on the digital image are associated with scale and rotation invariant features of the physical object depicted in the digital image.

### Claim 15 (depends on 6)

The method of claim 6, wherein the first and the second computer vision algorithms are each any one of: Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented Features from Accelerated Segment Test and Rotated Binary Robust Independent Elementary Features (ORB).

### Claim 16 (depends on 6)

The method of claim 6, wherein the physical object is a paper form document.

### Claim 17 (depends on 6)

The method of claim 6, wherein the plurality of identified keypoints includes between 64 and 512 keypoints.

### Claim 18 (depends on 6)

The method of claim 6, wherein the multi-dimensional descriptor vector is a 128-dimension descriptor vector that represents an image gradient at a location of the digital image associated with the keypoint of the first plurality of keypoints.

### Claim 19

A system comprising:
a processor;
a communication interface, coupled to the processor, through which to communicate over a network with remote devices; and
a memory coupled to the processor, the memory storing instructions which when executed by the processor cause the system to perform operations including:
assign a first template prediction based on a multi-dimensional descriptor vector, the multi-dimensional descriptor vector being associated with a keypoint of a first plurality of keypoints identified based on a digital image of a physical object and by use of a computer vision algorithm;
assign a second template predictions based on a histogram, the histogram is based on a keypoint of a second plurality of keypoints identified based on the digital image by use of a second computer vision algorithm;
reconcile each respective template prediction wherein the template predictions are merged where each respective template prediction is in agreement and eliminating each fewest vote receiving prediction where each respective template prediction is not in agreement; and
in response to said reconciliation, classify the physical object based on a remaining template prediction.

### Claim 20 (depends on 19)

The system of claim 19, wherein the first plurality of keypoints and the second plurality of keypoints are a same plurality of keypoints, and wherein the first computer vision algorithm and the second computer vision algorithm are a same computer vision algorithm.

### Claim 21 (depends on 19)

The system of claim 19, wherein the operations further include:
extracting, by use of a first visual feature extractor, a first plurality of visual features of the digital image;
extracting, by use of a second visual feature extractor, a second plurality of visual features of the digital image;
determining, by use of a clustering algorithm, a plurality of clusters of visual features based on any of the first plurality or the second plurality of visual features; and
determining, by use of a histogram algorithm, the histogram based on the plurality of clusters of visual features.

### Claim 22 (depends on 19)

The system of claim 19, wherein the operations further include:
receiving the digital image from a mobile device after the mobile device captured the digital image of the object.

### Claim 23 (depends on 19)

The system of claim 19, wherein the digital image is a frame of a video that includes the object.
