# US 2020/0346093 A1 · Reducing human interactions in game annotation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0346093 A1
- Document ID: US20200346093A1
- Status: Published application
- Kind: A1
- Application number: 16/865,230
- Filed: 2020-05-01
- Published: 2020-11-05
- Earliest claimed: 2019-05-03
- Assignees: New York University
- Applicants: New York University
- Inventors: Jorge Piazentin Ono; Arvi Gjoka; Justin Jonathan Salamon; Carlos Augusto Dietrich; Claudio T Silva
- Web page: https://patentgrep.com/patent/US20200346093A1

## CPC

- A63B 71/06: Indicating or scoring devices for games or players {, or for other sports activities}
- G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings}
- G06V 20/42: {of sport video content}
- A63B 2071/0694: {Visual indication, e.g. Indicia}
- A63B 2102/18: Baseball, rounders or similar games
- G09B 19/0038: {Sports}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The sport data tracking systems available today are based on specialized hardware to detect and track targets on the field. While effective, implementing and maintaining these systems pose a number of challenges, including high cost and need for close human monitoring. On the other hand, the sports analytics community has been exploring human computation and crowdsourcing in order to produce tracking data that is trustworthy, cheaper and more accessible. However, state-of-the-art methods require a large number of users to perform the annotation, or put too much burden into a single user. Example methods, systems and user interfaces that facilitate the creation of tracking data sequences of events (e.g., plays of baseball games) by warm-starting a manual annotation process using a vast collection of historical data are described.

## Claims

### Claim 1

A computer-implemented method for helping a user to annotate plays of a sporting game, the computer-implemented method comprising:
a) selecting video and/or audio of a sequence of events to be manually annotated by a user;
b) receiving information about at least one event of the sequence of events from the user;
c) retrieving, using the received information about at least one event of the sequence of events, a set of at least one candidate sequence from a corpus dataset;
d) selecting one of the at least one candidate sequence of the retrieved set to the user as a representative sequence; and
e) using the representative sequence to prepopulate a manual annotation of the sequence by the user.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
f) receiving manual user input to edit the prepopulated manual annotation of the sequence where it does not match the video of the play.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 further comprising:
f) receiving manual user input to revise information about at least one event of the sequence of events from the user;
g) retrieving, using the received revised information about at least one event of the sequence of events, a new set of at least one candidate sequence from a corpus dataset;
h) selecting one of the at least one candidate sequence of the retrieved new set to the user as a new representative sequence; and
i) using the new representative sequence to re-prepopulate a manual annotation of the sequence by the user.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 further comprising:
presenting a set of questions about the sequence to the user,
wherein the act of receiving information defining at least one event of the sequence from the user is performed based on answers provided by the user responsive to the presenting the set of questions about the sequence to the user.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4 wherein the set of questions presented to the user are ordered by their overall impact on narrowing the set of at least one candidate sequence retrieved from the corpus dataset.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5 wherein the sequence is a baseball play, and
wherein the ordered set of questions includes at least two of (1) who ran, (2) who are stealing bases, (3) what are end bases of runners, (4) who caught the batted ball in flight, (5) who threw the ball, and (6) what is the hit type.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5 wherein the sequence is a sports play, and
wherein the set of questions is ordered such that questions directly related to an outcome of the sports play are asked before questions about details of the sports play.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1 wherein each event of the sequence of events is defined by at least one {action, actor} pair.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8 wherein the actor is one of (A) a sports player position, (B) a sports player name, (C) a sports player type, (D) a ball, (E) a puck, and (F) a projectile.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1 wherein each event of the sequence of events has a time stamp measuring a time relative to a starting point.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1 wherein the representative one of the at least one candidate sequences of the retrieved set presented to the user includes an events chart including (1) frames of video and (2) at least one event representation, each associated with at least one of the frames of video.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11 wherein the a temporal sequence of event representations of the events chart are aligned using a start marker, wherein the start marker is determined from at least one of (A) a predetermined distinctive sound in the video and/or audio of the sequence of events, (B) a predetermined distinctive sound sequence in the video and/or audio of the sequence of events, (C) a predetermined distinctive image in the video and/or audio of the sequence of events, (D) a predetermined distinctive image sequence in the video and/or audio of the sequence of events, and (E) a manually entered demarcation and/or audio of the sequence of events.

### Claim 13 (depends on 11)

The computer-implemented method of claim 11 further comprising:
f) receiving a user input manipulating an event representation included in the events chart to change a frame of video of the events chart with which the event representation is associated;
g) performing a temporal query to retrieve, using the received user input for manipulating the event presentation, a new set of at least one candidate sequence;
h) selecting one of the at least one candidate sequence of the retrieved new set to the user as a new representative sequence; and
i) using the new representative sequence to re-prepopulate a manual annotation of the sequence by the user.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1 wherein each sequence is represented as a bit sequence indexing different events, and
wherein the set of at least one candidate play belongs to a cluster with the largest number of bits of the bit sequence in common with the query.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1 wherein the events of the sequence are weighted by the user in order to allow the user to increase or decrease the importance of certain events used to retrieve, using the received information defining the at least one event of the sequence, a set of at least one candidate sequence from the corpus dataset.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1 wherein the representation of a selected one of the at least one candidate sequence of the retrieved set presented to the user includes a timeline of the selected sequence.

### Claim 17 (depends on 1)

The computer-implemented method of claim 1 wherein the sequence is a sports play, and
wherein the representation of a selected one of the at least one candidate sequence of the retrieved set presented to the user includes a plan view of a field of play, the plan view including trajectories of events associated with the selected play.

### Claim 18

Apparatus comprising:
a) at least one processor; and
b) a non-transitory computer readable medium storing instructions which, when executed by the at least one processor, cause the at least one processor to perform a method including
1) selecting video and/or audio of a sequence of events to be manually annotated by a user,
2) receiving information about at least one event of the sequence of events from the user,
3) retrieving, using the received information about at least one event of the sequence of events, a set of at least one candidate sequence from a corpus dataset,
4) selecting one of the at least one candidate sequence of the retrieved set to the user as a representative sequence, and
5) using the representative sequence to prepopulate a manual annotation of the sequence by the user.

### Claim 19

A non-transitory computer readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform any of a method comprising:
a) selecting video and/or audio of a sequence of events to be manually annotated by a user;
b) receiving information about at least one event of the sequence of events from the user;
c) retrieving, using the received information about at least one event of the sequence of events, a set of at least one candidate sequence from a corpus dataset;
d) selecting one of the at least one candidate sequence of the retrieved set to the user as a representative sequence; and
e) using the representative sequence to prepopulate a manual annotation of the sequence by the user.
