# US 2019/0244059 A1 · Machine vision system for recognizing novel objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244059 A1
- Document ID: US20190244059A1
- Status: Published application
- Kind: A1
- Application number: 16/267,033
- Filed: 2019-02-04
- Published: 2019-08-08
- Earliest claimed: 2018-02-06
- Assignees: HRL Laboratories, LLC
- Applicants: HRL Laboratories, LLC
- Inventors: Soheil Kolouri; Charles E. Martin; Kyungnam Kim; Heiko Hoffmann
- Web page: https://patentgrep.com/patent/US20190244059A1

## CPC

- G06N 3/08: Learning methods
- G06V 10/82: using neural networks
- G06F 18/23213: with fixed number of clusters, e.g. K-means clustering
- G06F 18/41: {Interactive pattern learning with a human teacher}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/096: Transfer learning
- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06F 18/2133: based on naturality criteria, e.g. with non-negative factorisation or negative correlation
- G06V 10/464: {using a plurality of salient features, e.g. bag-of-words [BoW] representations}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Described is a system for classifying novel objects in imagery. In operation, the system extracts salient patches from a plurality of unannotated images using a multi-layer network. Activations of the multi-layer network are clustered into key attribute, with the key attributes being displayed to a user on a display, thereby prompting the user to annotate the key attributes with class label. An attribute database is then generated based on user prompted annotations of the key attributes. A test image can then be passed through the system, allowing the system to classify at least one object in the test image by identifying an object class in the attribute database. Finally, a device can be caused to operate or maneuver based on the classification of the at least one object in the test image.

## Claims

### Claim 1

A system for classifying novel objects in imagery, the system comprising:
one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of:
extracting salient patches from a plurality of unannotated images using a multi-layer network;
clustering activations of the multi-layer network into key attributes; and
displaying the key attributes to a user on a display and prompting the user to annotate the key attributes with class label;
generating an attribute database based on user generated annotations of the key attributes;
extracting salient patches from a test image, the test image having at least one object associated with the salient patches;
mapping the salient patches onto a binary attribute vector, the binary attribute vector indicating an existence or absence of key attributes in the test image;
classifying at least one object in the test image by identifying an object class in the attribute database corresponding to the binary attribute vector; and
controlling a device based on the classification of the at least one object in the test image.

### Claim 2 (depends on 1)

The system as set forth in claim 1, wherein the at least one object is classified based on an optimization routine that minimizes a distance between the binary attribute vector of textual attributes of a class and the binary attribute vector of the key attributes of the test image.

### Claim 3 (depends on 1)

The system as set forth in claim 1, wherein controlling the device includes causing the device to maneuver based on the classification of the at least one object in the test image.

### Claim 4 (depends on 1)

The system as set forth in claim 1, wherein the attribute database includes a library of object classes and corresponding textual attributes describing each object.

### Claim 5 (depends on 1)

The system as set forth in claim 1, wherein the multi-layer network is a neural network and salient patches are extracted based on the neural network's activation for each image pixel or group of pixels in an image.

### Claim 6

A computer program product for classifying novel objects in imagery, the computer program product comprising:
a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of:
extracting salient patches from a plurality of unannotated images using a multi-layer network;
clustering activations of the multi-layer network into key attributes; and
displaying the key attributes to a user on a display and prompting the user to annotate the key attributes with class label;
generating an attribute database based on user generated annotations of the key attributes;
extracting salient patches from a test image, the test image having at least one object associated with the salient patches;
mapping the salient patches onto a binary attribute vector, the binary attribute vector indicating an existence or absence of key attributes in the test image;
classifying at least one object in the test image by identifying an object class in the attribute database corresponding to the binary attribute vector; and
controlling a device based on the classification of the at least one object in the test image.

### Claim 7 (depends on 6)

The computer program product as set forth in claim 6, wherein the at least one object is classified based on an optimization routine that minimizes a distance between the binary attribute vector of textual attributes of a class and the binary attribute vector of the key attributes of the test image.

### Claim 8 (depends on 6)

The computer program product as set forth in claim 6, wherein controlling the device includes causing the device to maneuver based on the classification of the at least one object in the test image.

### Claim 9 (depends on 6)

The computer program product as set forth in claim 6, wherein the attribute database includes a library of object classes and corresponding textual attributes describing each object.

### Claim 10 (depends on 6)

The computer program product as set forth in claim 6, wherein the multi-layer network is a neural network and salient patches are extracted based on the neural network's activation for each image pixel or group of pixels in an image.

### Claim 11

A computer implemented method for classifying novel objects in imagery, the method comprising an act of:
causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
extracting salient patches from a plurality of unannotated images using a multi-layer network;
clustering activations of the multi-layer network into key attributes; and
displaying the key attributes to a user on a display and prompting the user to annotate the key attributes with class label;
generating an attribute database based on user generated annotations of the key attributes;
extracting salient patches from a test image, the test image having at least one object associated with the salient patches;
mapping the salient patches onto a binary attribute vector, the binary attribute vector indicating an existence or absence of key attributes in the test image;
classifying at least one object in the test image by identifying an object class in the attribute database corresponding to the binary attribute vector; and
controlling a device based on the classification of the at least one object in the test image.

### Claim 12 (depends on 11)

The method as set forth in claim 11, wherein the at least one object is classified based on an optimization routine that minimizes a distance between the binary attribute vector of textual attributes of a class and the binary attribute vector of the key attributes of the test image.

### Claim 13 (depends on 11)

The method as set forth in claim 11, wherein controlling the device includes causing the device to maneuver based on the classification of the at least one object in the test image.

### Claim 14 (depends on 11)

The method as set forth in claim 11, wherein the attribute database includes a library of object classes and corresponding textual attributes describing each object.

### Claim 15 (depends on 11)

The method as set forth in claim 11, wherein the multi-layer network is a neural network and salient patches are extracted based on the neural network's activation for each image pixel or group of pixels in an image.
