# US 11,907,339 B1 · Re-identification of agents using image analysis and machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,339 B1
- Document ID: US11907339B1
- Status: Granted
- Kind: B1
- Application number: 17/811,450
- Filed: 2022-07-08
- Granted: 2024-02-20
- Earliest claimed: 2018-12-13
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Behjat Siddiquie; Tian Lan; Jayakrishnan Eledath; Hoi Cheung Pang
- Web page: https://patentgrep.com/patent/US11907339B1

## CPC

- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification}
- G06N 20/00: Machine learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

As agents move about a materials handling facility, tracklets representative of the position of each agent are maintained along with a confidence score indicating a confidence that the position of the agent is known. If the confidence score falls below a threshold level, image data of the agent associated with the low confidence score is obtained and processed to generate one or more embedding vectors representative of the agent at a current position. Those embedding vectors are then compared with embedding vectors of other candidate agents to determine a set of embedding vectors having a highest similarity. The candidate agent represented by the set of embedding vectors having the highest similarity score is determined to be the agent and the position of that candidate agent is updated to the current position, thereby re-identifying the agent.

## Claims

### Claim 1

A computer-implemented method, comprising:
processing a first image of a first field of view of a scene to distinguish a first foreground object from a plurality of background objects;
processing at least a first portion of the first foreground object to determine that the first foreground object includes a first representation of an agent;
determining a first plurality of pixels of the first image representative of the agent;
processing the first plurality of pixels to generate a first embedding vector of the agent;
processing a second image of a second field of view of the scene to determine a second foreground object, wherein the second field of view is different than the first field of view;
processing at least a second portion of the second foreground object to determine that the second foreground object includes a second representation of the agent;
determining a second plurality of pixels of the second image representative of the agent;
processing the second plurality of pixels to generate a second embedding vector of the agent; and
including the first embedding vector and the second embedding vector in a feature set corresponding to the agent.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating the first image at a first time with a first imaging device; and
generating the second image at a second time with a second imaging device, wherein the first time is different than the second time and the second imaging device is different than the first imaging device.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
comparing the second embedding vector with the first embedding vector to determine that a difference between the second embedding vector and the first embedding vector satisfies an amount; and
in response to determining that the difference satisfies the amount, including the second embedding vector in the feature set.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
processing a third image of a third field of view of the scene to generate a third embedding vector of the agent;
comparing the third embedding vector with at least one of the first embedding vector and the second embedding vector to determine that a difference between the third embedding vector and the at least one of the first embedding vector and the second embedding vector does not satisfy an amount; and
in response to determining that the difference does not satisfy the amount, discarding the third embedding vector without including the third embedding vector in the feature set.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
comparing a third embedding vector of a third image of an unknown agent to the feature set to determine that the unknown agent is the agent.

### Claim 6

A system, comprising:
one or more processors; and
a memory storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least:
process a first plurality of pixels of a first image of a first field of view of a scene to generate a first embedding vector of an agent represented in the first image;
include the first embedding vector in a feature set of the agent;
process a second plurality of pixels of a second image of a second field of view of the scene to generate a second embedding vector of the agent;
determine that a difference between the second embedding vector and at least the first embedding vector satisfies an amount; and
in response to determination that the difference satisfies the amount, include the second embedding vector in the feature set of the agent.

### Claim 7 (depends on 6)

The system of claim 6, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
process the first image to distinguish a foreground object from a plurality of background objects;
process at least a portion of the foreground object to determine that the foreground object includes a representation of the agent; and
determine the first plurality of pixels corresponding to at least a portion of the agent.

### Claim 8 (depends on 6)

The system of claim 6, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
assign a first weight to the first embedding vector of the feature set; and
assign a second weight to the second embedding vector of the feature set, wherein the second weight is different than the first weight.

### Claim 9 (depends on 8)

The system of claim 8, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
receive the first image from a first imaging device, wherein the first image was generated at a first time; and
receive the second image from a second imaging device, wherein the second image was generated at a second time; and
wherein the first weight and the second weight are based at least in part on the first time and the second time.

### Claim 10 (depends on 6)

The system of claim 6, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
compare a third embedding vector of a third image with at least one of the first embedding vector and the second embedding vector to determine that a second difference between the third embedding vector and the at least one of the first embedding vector and the second embedding vector does not satisfy the amount; and
in response to determination that the second difference does not satisfy the amount, discard the third embedding vector without including the third embedding vector in the feature set.

### Claim 11 (depends on 6)

The system of claim 6, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
compare a third embedding vector of a third image with at least one of the first embedding vector and the second embedding vector to determine that a second difference between the third embedding vector and the at least one of the first embedding vector and the second embedding vector satisfies the amount; and
in response to determination that the second difference satisfies the amount, include the third embedding vector in the feature set.

### Claim 12 (depends on 6)

The system of claim 6, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
compare a third embedding vector of a third image of an unknown agent to the feature set to determine that the unknown agent is the agent.

### Claim 13 (depends on 12)

The system of claim 12, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
receive the third image from an imaging element, wherein the third image is of a second scene that is different than the scene.

### Claim 14 (depends on 6)

The system of claim 6, wherein the program instructions that, when executed by the one or more processors, further cause the one or more processors to at least:
define the first embedding vector as an anchor training input;
define the second embedding vector as a positive training input;
define a third embedding vector that does not correspond to the agent as a negative training input; and
use the anchor training input, the positive training input, and the negative training input to train a machine learning system to distinguish between different embedding vectors representative of the agent or a second agent that is different than the agent.

### Claim 15 (depends on 6)

The system of claim 6, wherein the feature set includes a plurality of embedding vectors of the agent, wherein each of the plurality of embedding vectors are representative of the agent from different fields of view.

### Claim 16

A method, comprising:
generating, based at least in part on a first image of an agent from a first field of view, a first embedding vector of the agent;
determining, based at least in part on a first comparison of the first embedding vector with a plurality of embedding vectors included in a feature set associated with the agent that the first embedding vector is to be included in the feature set;
in response to determining that the first embedding vector is to be included in the feature set, including the first embedding vector in the feature set;
generating, based at least in part on a second image of the agent from a second field of view, a second embedding vector of the agent;
determining, based at least in part on a second comparison of the second embedding vector with the plurality of embedding vectors included in the feature set that the second embedding vector is not to be included in the feature set; and
in response to determining that the second embedding vector is not to be included in the feature set, discarding the second embedding vector without including the second embedding vector in the feature set.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
assigning a weight to the first embedding vector, where in the weight is different than at least one other weight assigned to at least one of the plurality of embedding vectors included in the feature set.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
in response to including the first embedding vector in the feature set, discarding a third embedding vector of the plurality of embedding vectors from the feature set.

### Claim 19 (depends on 18)

The method of claim 18, wherein the third embedding vector is an oldest embedding vector of the plurality of embedding vectors.

### Claim 20 (depends on 16)

The method of claim 16, further comprising:
comparing a third embedding vector of an unknown agent to the feature set to determine that the unknown agent is the agent.