# US 2024/0004889 A1 · Task agnostic open-set prototypes for few-shot open-set recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004889 A1
- Document ID: US20240004889A1
- Status: Published application
- Kind: A1
- Application number: 18/153,899
- Filed: 2023-01-12
- Published: 2024-01-04
- Earliest claimed: 2022-05-18
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Byeonggeun KIM; Juntae LEE; Simyung CHANG
- Web page: https://patentgrep.com/patent/US20240004889A1

## CPC

- G06F 16/2462: {Approximate or statistical queries}
- G06N 3/09: Supervised learning
- G06F 16/285: {Clustering or classification}
- G06N 3/096: Transfer learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and techniques are provided for processing one or more data samples. For example, a neural network classifier can be trained to perform few-shot open-set recognition (FSOSR) based on a task-agnostic open-set prototype. A process can include determining one or more prototype representations for each class included in a plurality of support samples. A task-agnostic open-set prototype representation can be determined, in a same learned metric space as the one or more prototype representations. One or more distance metrics can be determined for each query sample of one or more query samples, based on the one or more prototype representations and the task-agnostic open-set prototype representation. Based on the one or more distance metrics, each query sample can be classified into one of classes associated with the one or more prototype representations or an open-set class associated with the task-agnostic open-set prototype representation.

## Claims

### Claim 1

A processor-implemented method for processing one or more data samples, comprising:
determining one or more prototype representations based on a plurality of support samples associated with one or more classes of data samples, wherein each prototype representation is associated with one of the one or more classes;
determining a task-agnostic open-set prototype representation, wherein the one or more prototype representations and the task-agnostic open-set prototype representation are determined in a same learned metric space;
determining one or more distance metrics for each query sample of one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open-set prototype representation; and
classifying each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open-set class associated with the task-agnostic open-set prototype representation.

### Claim 2 (depends on 1)

The processor-implemented method of claim 1, wherein determining the one or more distance metrics for each query sample further comprises:
determining a Euclidean distance metric between a given query sample and each prototype representation of the one or more prototype representations associated with the one or more classes; and
determining a Euclidean distance metric between the given query sample and the task-agnostic open-set prototype representation.

### Claim 3 (depends on 2)

The processor-implemented method of claim 2, further comprising:
scaling the Euclidean distance metric between the given query sample and the task-agnostic open-set prototype representation using one or more learned scaling factors.

### Claim 4 (depends on 3)

The processor-implemented method of claim 3, further comprising:
scaling the Euclidean distance metric between the given query sample and each prototype representation of the one or more prototype representations using the one or more learned scaling factors.

### Claim 5 (depends on 3)

The processor-implemented method of claim 3, wherein:
the one or more learned scaling factors are determined as a first scalar value and a second scalar value; and
the first scalar value and the second scalar value are learned based on a loss function that enforces the task-agnostic open-set prototype representation as a task-agnostic global-second-best classification for each query sample of the one or more query samples.

### Claim 6 (depends on 5)

The processor-implemented method of claim 5, wherein:
the one or more learned scaling factors are task-agnostic scaling factors; and
the task-agnostic open-set prototype representation is a global-second best classification for a plurality of few-shot open-shot recognition (FSOSR) episodes performed over the data samples.

### Claim 7 (depends on 1)

The processor-implemented method of claim 1, wherein classifying each query sample based on the one or more distance metrics comprises:
determining a probability distribution over the one or more classes and the open-set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric determined between each query sample and a respective prototype associated with each class of the one or more classes and a Euclidean distance metric determined between each query sample and the task-agnostic open-set prototype representation; and
classifying, based on the probability distribution, each query sample into one of the one or more classes or into the open-set class.

### Claim 8 (depends on 7)

The processor-implemented method of claim 7, further comprising:
performing open-set rejection (OSR) based on a set of classified query samples classified into the open-set class associated with the task-agnostic open-set prototype representation.

### Claim 9 (depends on 7)

The processor-implemented method of claim 7, wherein classifying each query sample based on the one or more distance metrics further comprises:
determining a probability that each query sample is included in the open-set class associated with the task-agnostic open-set prototype representation, based on the probability distribution; and
comparing the determined probability to a pre-determined threshold.

### Claim 10 (depends on 9)

The processor-implemented method of claim 9, further comprising:
classifying a given query sample as being included in the open-set class based on a determination that the probability the given query sample is included in the open-set class is greater than the pre-determined threshold; and
classifying the given query sample as being included in a closed-set class based on a determination that the probability the given query sample is included in the open-set class is not greater than the pre-determined threshold.

### Claim 11 (depends on 10)

The processor-implemented method of claim 10, further comprising:
classifying each query sample classified as being included in the closed-set class into a respective class of the one or more classes, wherein each query sample is classified based on maximizing a respective probability determined between each query sample and each respective class of the one or more classes.

### Claim 12 (depends on 11)

The processor-implemented method of claim 11, wherein the probability is an argmax probability.

### Claim 13 (depends on 1)

The processor-implemented method of claim 1, wherein classifying each query sample comprises:
providing each query sample to a trained few-shot open-shot recognition (FSOSR) neural network classifier, wherein the trained FSOSR neural network classifier includes at least the task-agnostic open-set prototype representation and one or more distance scaling factors as learnable components.

### Claim 14 (depends on 13)

The processor-implemented method of claim 13, wherein the trained FSOSR neural network classifier further includes one or more feature embedding networks as a learnable component.

### Claim 15 (depends on 14)

The processor-implemented method of claim 14, wherein determining the one or more distance metrics for each query sample further comprises:
determining an embedding for each query sample of the one or more query samples using the one or more feature embedding networks;
determining the one or more prototype representations as an average embedding of a set of embeddings determined for the plurality of support samples associated with each class of the one or more classes; and
determining each distance metric of the one or more distance metrics based on determining a Euclidean distance metric between the embedding determined for a given query sample and the embedding determined for each prototype representation of the one or more prototype representations associated with the one or more classes.

### Claim 16

An apparatus for processing one or more data samples, comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
determine one or more prototype representations based on a plurality of support samples associated with one or more classes of data samples, wherein each prototype representation is associated with one of the one or more classes;
determine a task-agnostic open-set prototype representation, wherein the one or more prototype representations and the task-agnostic open-set prototype representation are determined in a same learned metric space;
determine one or more distance metrics for each query sample of one or more query samples, wherein the one or more distance metrics are determined based on the one or more prototype representations and the task-agnostic open-set prototype representation; and
classify each query sample based on the one or more distance metrics, wherein each query sample is classified into one of the one or more classes associated with the one or more prototype representations or an open-set class associated with the task-agnostic open-set prototype representation.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein, to determine the one or more distance metrics for each query sample, the at least one processor is further configured to:
determine a Euclidean distance metric between a given query sample and each prototype representation of the one or more prototype representations associated with the one or more classes; and
determine a Euclidean distance metric between the given query sample and the task-agnostic open-set prototype representation.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the at least one processor is further configured to:
scale the Euclidean distance metric between the given query sample and the task-agnostic open-set prototype representation using one or more learned scaling factors.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the at least one processor is further configured to:
scale the Euclidean distance metric between the given query sample and each prototype representation of the one or more prototype representations using the one or more learned scaling factors.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein:
the one or more learned scaling factors are determined as a first scalar value and a second scalar value; and
the first scalar value and the second scalar value are learned based on a loss function that enforces the task-agnostic open-set prototype representation as a task-agnostic global-second-best classification for each query sample of the one or more query samples.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein:
the one or more learned scaling factors are task-agnostic scaling factors; and
the task-agnostic open-set prototype representation is a global-second best classification for a plurality of few-shot open-shot recognition (FSOSR) episodes performed over the data samples.

### Claim 22 (depends on 16)

The apparatus of claim 16, wherein, to classify each query sample based on the one or more distance metrics, the at least one processor is configured to:
determine a probability distribution over the one or more classes and the open-set class, wherein the probability distribution is determined based at least in part on a Euclidean distance metric determined between each query sample and a respective prototype associated with each class of the one or more classes and a Euclidean distance metric determined between each query sample and the task-agnostic open-set prototype representation; and
classify, based on the probability distribution, each query sample into one of the one or more classes or into the open-set class.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the at least one processor is further configured to:
perform open-set rejection (OSR) based on a set of classified query samples classified into the open-set class associated with the task-agnostic open-set prototype representation.

### Claim 24 (depends on 22)

The apparatus of claim 22, wherein, to classify each query sample based on the one or more distance metrics, the at least one processor is further configured to:
determine a probability that each query sample is included in the open-set class associated with the task-agnostic open-set prototype representation, based on the probability distribution; and
compare the determined probability to a pre-determined threshold.

### Claim 25 (depends on 24)

The apparatus of claim 24, wherein the at least one processor is further configured to:
classify a given query sample as being included in the open-set class based on a determination that the probability the given query sample is included in the open-set class is greater than the pre-determined threshold; and
classify the given query sample as being included in a closed-set class based on a determination that the probability the given query sample is included in the open-set class is not greater than the pre-determined threshold.

### Claim 26 (depends on 25)

The apparatus of claim 25, wherein the at least one processor is further configured to:
classify each query sample classified as being included in the closed-set class into a respective class of the one or more classes, wherein each query sample is classified based on maximizing a respective probability determined between each query sample and each respective class of the one or more classes.

### Claim 27 (depends on 26)

The apparatus of claim 26, wherein the probability is an argmax probability.

### Claim 28 (depends on 16)

The apparatus of claim 16, wherein, to classify each query sample, the at least one processor is configured to:
provide each query sample to a trained few-shot open-shot recognition (FSOSR) neural network classifier, wherein the trained FSOSR neural network classifier includes at least the task-agnostic open-set prototype representation and one or more distance scaling factors as learnable components.

### Claim 29 (depends on 28)

The apparatus of claim 28, wherein the trained FSOSR neural network classifier further includes one or more feature embedding networks as a learnable component.

### Claim 30 (depends on 29)

The apparatus of claim 29, wherein, to determine the one or more distance metrics for each query sample, the at least one processor is configured to:
determine an embedding for each query sample of the one or more query samples using the one or more feature embedding networks;
determine the one or more prototype representations as an average embedding of a set of embeddings determined for the plurality of support samples associated with each class of the one or more classes; and
determine each distance metric of the one or more distance metrics based on determining a Euclidean distance metric between the embedding determined for a given query sample and the embedding determined for each prototype representation of the one or more prototype representations associated with the one or more classes.
