# US 2020/0394441 A1 · Image classification system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394441 A1
- Document ID: US20200394441A1
- Status: Published application
- Kind: A1
- Application number: 16/440,859
- Filed: 2019-06-13
- Published: 2020-12-17
- Earliest claimed: 2019-06-13
- Assignees: Expedia, Inc.
- Applicants: Expedia, Inc.
- Inventors: Li Wen; Zhanpeng Huo; Jingya Jiang
- Web page: https://patentgrep.com/patent/US20200394441A1

## CPC

- G06N 3/08: Learning methods
- G06F 16/51: Indexing; Data structures therefor; Storage structures
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/2431: Multiple classes
- G06F 18/24317: {Piecewise classification, i.e. whereby each classification requires several discriminant rules}
- G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image classification system is provided for determining a likely classification of an image using multiple machine learning models that share a base machine learning model. The image classification system may be a browser-based system on a user computing device that obtains multiple machine learning models over a network from a remote system once, stores the models locally in the image classification system, and uses the models multiple times without needing to subsequently request the machine learning models again from the remote system. The image classification system may therefore determine likely a classification associated with an image by running the machine learning models on a user computing device.

## Claims

### Claim 1

A system for classifying images, the system comprising:
a data store comprising a plurality of machine learning models, the plurality of machine learning models comprising a base model and a plurality of work models; and
one or more processors configured by computer-executable instructions to at least:
receive an image selected by a user for classification as one of a plurality of item labels;
generate a base model output vector using the base model and the image, wherein the base model output vector represents a plurality of features of the image;
determine a first item label associated with the image using the base model output vector and a first work model of the plurality of work models;
determine a category group label associated with the image using the base model output vector and a second work model of the plurality of work models;
receive a first user input indicating selection of the category group label;
determine a category label associated with image using the base model output vector and a third work model of the plurality of work models, wherein the third work model is associated with the category group label;
receive a second user input indicating selection of the category label;
determine a second item label associated with image using the base model output and a fourth work model of the plurality of work models, wherein the fourth work model is associated with the category label;
receive a third user input indicating selection of the second item label; and
transmit the image and classification data associated with the image,
wherein the classification data represents the second item label.

### Claim 2 (depends on 1)

The system of claim 1, wherein the base model is used only once to generate output using the image.

### Claim 3 (depends on 1)

The system of claim 1, wherein the base model and the plurality of work models are each neural network based models.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first work model generates a first confidence value associated with the first item label, and wherein the first item label is displayed based at least partly on the first confidence value exceeding a threshold value.

### Claim 5

A computer-implemented method comprising:
as implemented by a computing system comprising one or more computer processors configured to execute specific instructions:
receiving an image to be classified;
generating, using a base model network and the image, a base model output associated with the image;
generating, using a plurality work model networks and the base model output, a plurality of work model outputs;
displaying an image classification interface configured to accept user input regarding classification of the image;
receiving user interaction data representing a user input to the image classification interface; and
determining a label associated with the image, wherein the label is determined based at least partly on the plurality of work model outputs and the user input.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein each of the plurality of work model outputs is generated using the same base model output.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, further comprising storing the base model output in a cache.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising loading the base model output from the cache, wherein at least one of the plurality of work model outputs is generated after the base model output is loaded from the cache.

### Claim 9 (depends on 5)

The computer-implemented method of claim 5, further comprising:
receiving the base model network and the plurality of work model networks from a remote system via a network; and
storing the base model network and the plurality of work model networks in a local data store of the computing system.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising loading the base model network and the plurality of work model networks from the local data store, wherein the local data store comprises a browser-based data store.

### Claim 11 (depends on 5)

The computer-implemented method of claim 5, wherein generating the plurality of work model outputs comprises generating a first work model output using a first work model network in parallel with generating a second work model output using a second work model network.

### Claim 12 (depends on 5)

The computer-implemented method of claim 5, wherein generating the plurality of work model outputs comprises generating a first work model output using a first work model network prior to generating a second work model output using a second work model network

### Claim 13 (depends on 5)

The computer-implemented method of claim 5, wherein a first work model output comprises a first confidence score associated with the label, and a second confidence score associated with a second label.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, further comprising determining to display the label based on the first confidence score satisfying a threshold.

### Claim 15 (depends on 13)

The computer-implemented method of claim 13, further comprising determining to display the label based on a difference between the first confidence score and the second confidence score.

### Claim 16 (depends on 5)

The computer-implemented method of claim 5, further comprising:
receiving a second image selected by the user for classification;
generating, using the base model network and the second image, a second base model output associated with the second image;
generating, using the plurality of work model networks and the base model output, a second plurality of work model outputs; and
determining a second label associated with the second image, wherein the second label is determined based at least partly on the second plurality of work model outputs, and wherein the second label is different than the label.

### Claim 17

A system comprising computer-readable memory and one or more processors configured by computer-executable instructions to at least:
receive, from a user computing device, a request for a network resource configured to:
generate, using a base model, a base model output associated with an image;
generate, using one or more work models, one or more work model outputs based at least partly on the base model output; and
determine a label associated with the image, wherein the label is based at least partly on the one or more work model outputs;
transmit, to the user computing device, the network resource in response to the request; and
receive, from the user computing device, the image and image classification data representing the label associated with the image.

### Claim 18 (depends on 16)

The system of claim 16, wherein the one or more processors are further configured by the computer-executable instructions to transmit the base model and the one or more work models to the user computing device.

### Claim 19 (depends on 16)

The system of claim 16, wherein the label is associated with a confidence score, and wherein the label is displayed based at least partly on the confidence score satisfying a threshold.

### Claim 20 (depends on 16)

The system of claim 16, wherein the base model and the one or more work models comprise neural network based models executed on the user computing device.
