# US 11,308,319 B2 · Computer vision image feature identification via multi-label few-shot model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,319 B2
- Document ID: US11308319B2
- Status: Granted
- Kind: B2
- Application number: 16/698,465
- Filed: 2019-11-27
- Published: 2021-05-13
- Granted: 2022-04-19
- Earliest claimed: 2019-11-08
- Assignees: DST Technologies, Inc.
- Applicants: DST Technologies, Inc.
- Inventors: Hui Peng Hu; Ramesh Sridharan
- Web page: https://patentgrep.com/patent/US11308319B2

## CPC

- G06N 20/00: Machine learning
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24317: {Piecewise classification, i.e. whereby each classification requires several discriminant rules}
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06V 10/40: Extraction of image or video features
- G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A technique making use of a few-shot model to determine graphical features present in an image based on a small set of examples with known graphical features. Where a support set including a number of images that each have a known combination of graphical features, the image recognition can identify unknown combinations of those graphical features in any number of query images. In an embodiment of the present disclosure examples of a filled-out form are used to interpret any number of additional filled out versions of the form.

## Claims

### Claim 1

A multi-model image content recognition system comprising:
a first model configured to receive an input set including a query image and a support set of images, wherein first model is trained to output a descriptive vector for each image of the input set including a descriptive query vector and descriptive support vectors;
a second model configured to receive each descriptive support vector concatenated with the descriptive query vector, wherein the second model is trained to output a comparison vector for each input, the comparison vector indicating a degree of similarity between the query image and a respective member of the support set of images; and
a third model trained to output a solution vector indicating graphic features present in the query image based on the comparison vectors.

### Claim 2 (depends on 1)

The system of claim 1, wherein the third model further bases the solution vector on a matrix of support set graphic features.

### Claim 3 (depends on 1)

The system of claim 1, wherein the first model, the second model, and the third model are neural network models.

### Claim 4 (depends on 3)

The system of claim 3, wherein the first model, the second model, and the third model are few-shot models.

### Claim 5 (depends on 1)

The system of claim 1, wherein each image of the support set of images and the query image are each a representation of a filled-out form document, the filled-out form documents adhering to a same form template, and the graphic features are check boxes positioned at predetermined locations on the form template.

### Claim 6 (depends on 1)

The system of claim 1, wherein the support set of images includes at least one instance of each of a set of graphic features and the query image includes an unknown combination of the graphic features that is not present within the support set of images.

### Claim 7

A method comprising:
receiving, by a first model, an input set including a query image and a support set of images;
outputting, by the first model, a descriptive vector for each image of the input set including a descriptive query vector and descriptive support vectors;
receiving, by a second model, each descriptive support vector concatenated with the descriptive query vector;
outputting, by the second model, a comparison vector for each descriptive support vector concatenated with the descriptive query vector, the comparison vector indicating a degree of similarity between the query image and a respective member of the support set of images; and
projecting, by a third model, a solution vector indicating graphic features present in the query image based on the comparison vectors.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
computing an unprojected solution based on the comparison vectors and an inverted matrix indicating graphic features of the support set of images, wherein the unprojected solution is used by the third model to project the solution vector.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
training the first model on a plurality of images including metadata indicating present graphic features.

### Claim 10 (depends on 7)

The method of claim 7, wherein the third model further bases the solution vector on a matrix of support set graphic features.

### Claim 11 (depends on 7)

The method of claim 7, wherein the first model, the second model, and the third model are neural network models.

### Claim 12 (depends on 11)

The method of claim 11, wherein the first model, the second model, and the third model are few-shot models.

### Claim 13 (depends on 7)

The method of claim 7, wherein each image of the support set of images and the query image are each a representation of a filled-out form document, the filled-out form documents adhering to a same form template, and the graphic features are check boxes positioned at predetermined locations on the form template.

### Claim 14 (depends on 7)

The method of claim 7, wherein the support set of images includes at least one instance of each of a set of graphic features and the query image includes an unknown combination of the graphic features that is not present within the support set of images.

### Claim 15

A method comprising:
receiving, by a series of neural network models, an input set including a query image and a support set of images, wherein the support set of images includes at least one instance of each of a set of graphic features and the query image includes an unknown combination of the graphic features that is not present within the support set of images;
generating, by the series of neural network models, a set of partial products that progressively develop a list of graphic features included in the unknown combination of graphic features of the query image, wherein the series of neural network models vectorizes the input set and then compares and projects the partial products; and
outputting, by the series of neural network models, the list of graphic features included in the unknown combination of graphic features of the query image.

### Claim 16 (depends on 15)

The method of claim 15, wherein the series of neural network models are few-shot models.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
training the series of neural network models on a plurality of images including metadata indicating present graphic features.

### Claim 18 (depends on 15)

The method of claim 15, wherein the input set further includes a matrix indicating the instances of each of graphic feature present in each respective image of the support set of images.

### Claim 19 (depends on 18)

The method of claim 18, wherein the partial products further include an inverted version of the matrix.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
executing computations including partial products external to and interspersed between the series of neural network models.