# US 2018/0322373 A1 · Zero shot machine vision system via joint sparse representations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322373 A1
- Document ID: US20180322373A1
- Status: Published application
- Kind: A1
- Application number: 15/949,896
- Filed: 2018-04-10
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: HRL Laboratories, LLC
- Applicants: HRL Laboratories, LLC
- Inventors: Soheil Kolouri; Shankar R. Rao; Kyungnam Kim
- Web page: https://patentgrep.com/patent/US20180322373A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/096: Transfer learning
- G06F 18/2136: based on sparsity criteria, e.g. with an overcomplete basis
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/24: Classification techniques
- G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/09: Supervised learning
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- B60W 10/18: including control of braking systems
- B60W 10/20: including control of steering systems
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/582: {of traffic signs}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT

## Abstract

Described is a system that can recognize novel objects that the system has never before seen. The system uses a training image set to learn a model that maps visual features from known images to semantic attributes. The learned model is used to map visual features of an unseen input image to semantic attributes. The unseen input image is classified as belonging to an image class with a class label. A device is controlled based on the class label.

## Claims

### Claim 1

A system for object recognition, the system comprising:
one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of:
using a training image set, learning a model that maps visual features from known images to semantic attributes;
using the learned model to map visual features of an unseen input image to semantic attributes;
classifying, with a class label, the unseen input image as belonging to an image class; and
controlling a device based on the class label.

### Claim 2 (depends on 1)

The system as set forth in claim 1, wherein the device is a vehicle component, and wherein the one or more processors further perform an operation of controlling the vehicle component to cause a vehicle operation to be performed, wherein the vehicle operation is at least one of braking and turning.

### Claim 3 (depends on 1)

The system as set forth in claim 1, wherein the one or more processors further perform an operation of generating a training image set comprising images of objects from a plurality of image classes, wherein each image in the training image set has been annotated with a class label and semantic attributes.

### Claim 4 (depends on 1)

The system as set forth in claim 1, wherein a convolutional neural network (CNN) extracts visual features from known images.

### Claim 5 (depends on 1)

The system as set forth in claim 1, wherein the learned model is a joint-sparse dictionary model which provides a joint representation for visual features and semantic attributes.

### Claim 6 (depends on 1)

The system as set forth in claim 1, wherein the one or more processors further perform operations of:
extracting visual features from the unseen input image;
for each extracted image feature, determining a sparse representation of the unseen input image; and
determining at least one image attribute using the sparse representation of the unseen input image; and
determining the class label for the unseen input image by mapping the at least one image attribute to a label space.

### Claim 7 (depends on 6)

The system as set forth in claim 6, wherein a Lasso problem is solved to determine the sparse representation of the unseen input image.

### Claim 8 (depends on 1)

The system as set forth in claim 1, wherein the system is a zero-shot learning machine-vision system.

### Claim 9

A computer implemented method for object recognition, the method comprising an act of:
causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of:
using a training image set, learning a model that maps visual features from known images to semantic attributes;
using the learned model to map visual features of an unseen input image to semantic attributes;
classifying, with a class label, the unseen input image as belonging to an image class; and
controlling a device based on the class label.

### Claim 10 (depends on 9)

The method as set forth in claim 9, wherein the one or more processors further perform an operation of generating a training image set comprising images of objects from a plurality of image classes, wherein each image in the training image set has been annotated with a class label and semantic attributes.

### Claim 11 (depends on 9)

The method as set forth in claim 9, wherein a convolutional neural network (CNN) extracts visual features from known images.

### Claim 12 (depends on 9)

The method as set forth in claim 9, wherein the learned model is a joint-sparse dictionary model which provides a joint representation for visual features and semantic attributes.

### Claim 13 (depends on 9)

The method as set forth in claim 9, wherein the one or more processors further perform operations of:
extracting visual features from the unseen input image;
for each extracted image feature, determining a sparse representation of the unseen input image; and
determining at least one image attribute using the sparse representation of the unseen input image; and
determining the class label for the unseen input image by mapping the at least one image attribute to a label space.

### Claim 14 (depends on 13)

The method as set forth in claim 13, wherein a Lasso problem is solved to determine the sparse representation of the unseen input image.

### Claim 15

A computer program product for object recognition, the computer program product comprising:
computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors for causing the processor to perform operations of:
using a training image set, learning a model that maps visual features from known images to semantic attributes;
using the learned model to map visual features of an unseen input image to semantic attributes;
classifying, with a class label, the unseen input image as belonging to an image class; and
controlling a device based on the class label.

### Claim 16 (depends on 15)

The computer program product as set forth in claim 15, wherein the device is a vehicle component, and wherein the one or more processors further perform an operation of controlling the vehicle component to cause a vehicle operation to be performed, wherein the vehicle operation is at least one of braking and turning.

### Claim 17 (depends on 15)

The computer program product as set forth in claim 15, wherein a convolutional neural network (CNN) extracts visual features from known images.

### Claim 18 (depends on 15)

The computer program product as set forth in claim 15, wherein the learned model is a joint-sparse dictionary model which provides a joint representation for visual features and semantic attributes.

### Claim 19 (depends on 15)

The computer program product as set forth in claim 15, further comprising instructions for causing the one or more processors to further perform operations of:
extracting visual features from the unseen input image;
for each extracted image feature, determining a sparse representation of the unseen input image; and
determining at least one image attribute using the sparse representation of the unseen input image; and
determining the class label for the unseen input image by mapping the at least one image attribute to a label space.

### Claim 20 (depends on 19)

The computer program product as set forth in claim 19, wherein a Lasso problem is solved to determine the sparse representation of the unseen input image.
