# US 2026/0087073 A1 · Similar play retrieval system for video enhancements

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087073 A1
- Document ID: US20260087073A1
- Status: Published application
- Kind: A1
- Application number: 18/896,558
- Filed: 2024-09-25
- Published: 2026-03-26
- Earliest claimed: 2024-09-25
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Yossi Biton; Ziv Rader; Bar Segev; Avi Avraham Ben-Cohen; Ido Yerushalmy; Sam Schwartzstein; Yochai Zvik; Ishay Be'ery; Eliran Nachum
- Web page: https://patentgrep.com/patent/US20260087073A1

## CPC

- G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings}
- G06V 20/42: {of sport video content}
- G06F 16/24578: {using ranking}
- G06F 16/738: Presentation of query results
- G06F 16/75: Clustering; Classification
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and techniques are described for similar play retrieval for video enhancements. In various examples, first tracking data representing first respective locations of a first plurality of players at a first time may be received. First embedding data representing a formation of the first plurality of players at the first time may be generated based at least in part on the first tracking data. Second embedding data may be determined by searching a first data store using the first embedding data. The first data store may include a plurality of historical embeddings representing past plays. A first historical play associated with the second embedding data may be determined.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving a first frame of tracking data indicating a respective location of each player of a first plurality of players on a field plane at a first time, wherein the tracking data is generated by a first plurality of sensors;
generating first embedding data comprising a first vector, wherein a first element of the first vector represents a first player of the first plurality of players, wherein a value of the first element of the first vector represents a first x, y coordinate representing a normalized location of the first player indicated by the first frame of the tracking data, and wherein a second element of the first vector represents a second player of the first plurality of players, wherein a value of the second element of the first vector represents a second x, y coordinate representing a normalized location of the second player indicated by the first frame of the tracking data;
determining first game state data temporally associated with the first frame of the tracking data, wherein the first game state data describes at least one of a current ball location, a current down, or a distance until a first down;
determining a subset of historical embeddings representing historical plays by filtering a set of historical plays using the first game state data;
determining, by searching historical embeddings of the subset of historical embeddings using the first embedding data and a distance metric, second embedding data;
determining a first historical play by ranking historical plays associated with the second embedding data based at least in part on the first game state data; and
sending video data portraying the first historical play to a first computing device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining, using an unsupervised machine learning model, a first plurality of clusters of the subset of historical embeddings;
determining, for the first embedding data, a first cluster of the first plurality of clusters based on a distance between the first embedding data and an aggregated embedding for the first cluster; and
determining, for the first cluster, a first set of nearest neighbors in an embedding space for the first embedding data, wherein the first set of nearest neighbors comprises the second embedding data.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a second frame of the tracking data indicating a respective location of each of the first plurality of players on the field plane at a second time;
generating second embedding data comprising a second vector, wherein a first element of the second vector represents the first player of the first plurality of players, wherein a value of the first element of the second vector represents a third x, y coordinate representing the normalized location of the first player indicated by the second frame of the tracking data, and wherein a second element of the second vector represents the second player of the first plurality of players, wherein a value of the second element of the second vector represents a fourth x, y coordinate representing the normalized location of the second player indicated by the second frame of the tracking data; and
generating aggregated embedding data based on an aggregation of the first vector and the second vector, wherein the aggregated embedding data represents locations of the first plurality of players over the first time and the second time.

### Claim 4

A computer-implemented method comprising:
receiving first tracking data representing first respective locations of a first plurality of players at a first time;
generating first embedding data representing a formation of the first plurality of players at the first time based at least in part on the first tracking data;
determining second embedding data by searching historical embeddings of a first data store using the first embedding data;
determining a first historical play by ranking historical plays associated with the second embedding data based at least in part on game state data; and
retrieving at least one of historical tracking data or historical video data associated with the first historical play.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising receiving the first tracking data from a first plurality of sensors, wherein each sensor of the first plurality of sensors is associated with a respective player of the first plurality of players.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, further comprising:
generating a first vector, wherein a first element of the first vector represents a first coordinate for a first location of a first player of the first plurality of players, and wherein a second element of the first vector represents a second coordinate for a second location of a second player of the first plurality of players, wherein the first embedding data comprises the first vector.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving, from a first metadata service, first game state data associated with the first tracking data; and
determining a subset of historical plays by filtering a set of historical plays using the first game state data, wherein determining the first historical play comprises searching the subset of historical plays using the first embedding data.

### Claim 8 (depends on 4)

The computer-implemented method of claim 4, further comprising generating the first embedding data by inputting first data representing the first tracking data into a graph neural network, wherein the graph neural network is trained to generate embeddings representing formations of players as graph data, wherein a first node of the graph data represents a first player, a second node of the graph data represents a second player, and an edge of the graph data connecting the first node and the second node represents a spacing between the first player and the second player.

### Claim 9 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining a ranked list of embedding data by searching the first data store using the first embedding data, wherein the ranked list of embedding data is ranked based at least in part on a degree of similarity to the first embedding data, wherein the degree of similarity is determined using a first distance metric; and
generating a ranked list of historical plays using the ranked list of embedding data, wherein the ranked list of historical plays comprises the first historical play.

### Claim 10 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving second tracking data representing second respective locations of the first plurality of players at a second time different from the first time;
generating third embedding data representing a formation of the first plurality of players at the second time based at least in part on the second tracking data; and
generating fourth embedding data based at least in part on aggregating the first embedding data and the third embedding data, wherein the second embedding data is determined based at least in part on the fourth embedding data.

### Claim 11 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining a set of historical plays associated with the second embedding data, the set of historical plays comprising the first historical play;
determining a percentage of the set of historical plays that resulted in a successful outcome as determined using a first success metric; and
causing first graphical data indicating the percentage to be rendered on a display.

### Claim 12 (depends on 4)

The computer-implemented method of claim 4, further comprising:
determining, using the historical tracking data associated with the first historical play, a first trajectory associated with a first player;
generating a graphical overlay representing, at least in part, the first trajectory; and
causing the graphical overlay to be rendered on a display over a video feed of the first plurality of players.

### Claim 13 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving a live video feed of the first plurality of players; and
causing the historical video data to be rendered on a display during the live video feed.

### Claim 14

A system comprising:
at least one processor; and
non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to cause the at least one processor to:
receive first tracking data representing first respective locations of a first plurality of players at a first time;
generate first embedding data representing a formation of the first plurality of players at the first time based at least in part on the first tracking data;
determine second embedding data by searching historical embeddings of a first data store using the first embedding data;
determine a first historical play by ranking historical plays associated with the second embedding data based at least in part on game state data; and
retrieve at least one of historical tracking data or historical video data associated with the first historical play.

### Claim 15 (depends on 14)

The system of claim 14, wherein the non-transitory computer-readable memory stores further instructions that, when executed by the at least one processor, are further effective to:
receive the first tracking data from a first plurality of sensors, wherein each sensor of the first plurality of sensors is associated with a respective player of the first plurality of players.

### Claim 16 (depends on 14)

The system of claim 14, wherein the non-transitory computer-readable memory stores further instructions that, when executed by the at least one processor, are further effective to:
generate a first vector, wherein a first element of the first vector represents a first coordinate for a first location of a first player of the first plurality of players, and wherein a second element of the first vector represents a second coordinate for a second location of a second player of the first plurality of players, wherein the first embedding data comprises the first vector.

### Claim 17 (depends on 14)

The system of claim 14, wherein the non-transitory computer-readable memory stores further instructions that, when executed by the at least one processor, are further effective to:
receive, from a first metadata service, first game state data associated with the first tracking data; and
determine a subset of historical plays by filtering a set of historical plays using the first game state data, wherein determining the first historical play comprises searching the subset of historical plays using the first embedding data.

### Claim 18 (depends on 14)

The system of claim 14, wherein the non-transitory computer-readable memory stores further instructions that, when executed by the at least one processor, are further effective to:
generate the first embedding data by inputting first data representing the first tracking data into a graph neural network, wherein the graph neural network is trained to generate embeddings representing formations of players as graph data, wherein a first node of the graph data represents a first player, a second node of the graph data represents a second player, and an edge of the graph data connecting the first node and the second node represents a spacing between the first player and the second player.

### Claim 19 (depends on 14)

The system of claim 14, wherein the non-transitory computer-readable memory stores further instructions that, when executed by the at least one processor, are further effective to:
determine a ranked list of embedding data by searching the first data store using the first embedding data, wherein the ranked list of embedding data is ranked based at least in part on a degree of similarity to the first embedding data, wherein the degree of similarity is determined using a first distance metric; and
generate a ranked list of historical plays using the ranked list of embedding data, wherein the ranked list of historical plays comprises the first historical play.

### Claim 20 (depends on 14)

The system of claim 14, wherein the non-transitory computer-readable memory stores further instructions that, when executed by the at least one processor, are further effective to:
receive second tracking data representing second respective locations of the first plurality of players at a second time different from the first time;
generate third embedding data representing a formation of the first plurality of players at the second time based at least in part on the second tracking data; and
generate fourth embedding data based at least in part on aggregating the first embedding data and the third embedding data, wherein the second embedding data is determined based at least in part on the fourth embedding data.
