# US 2021/0240992 A1 · Methods and systems of spatiotemporal pattern recognition for video content development

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240992 A1
- Document ID: US20210240992A1
- Status: Published application
- Kind: A1
- Application number: 17/238,847
- Filed: 2021-04-23
- Published: 2021-08-05
- Earliest claimed: 2014-02-28
- Assignees: Second Spectrum, Inc.
- Applicants: Second Spectrum, Inc.
- Inventors: Edward Shek Chan; Yu-Han Chang; Rajiv Tharmeswaran Maheswaran; Jeffrey Wayne Su
- Web page: https://patentgrep.com/patent/US20210240992A1

## CPC

- G06V 20/42: {of sport video content}
- A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06N 20/00: Machine learning
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- H04N 5/2224: {related to virtual studio applications}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4345: {Extraction or processing of SI, e.g. extracting service information from an MPEG stream}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- H04N 21/4662: {characterized by learning algorithms}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06T 2207/20081: Training; Learning
- G06T 2207/30221: Sports video; Sports image
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A location in a video may be specified and content related to the location may be accessed. A method for accessing the related content may include receiving a reference to a pixel location in a frame of a video feed of a filmed occurrence and accessing a spatio-temporal index corresponding to the filmed occurrence. The spatio-temporal index may index information relating to events or objects of the filmed occurrence and corresponding pixel locations at which the events or the objects are detected in the video feed. The method may further include querying the spatio-temporal index using the pixel location to determine particular information of an indexed event or an indexed object and receiving the particular information wherein the particular information indicates at least one of spatial and temporal alignment parameters for aligning the indexed event with a corresponding event in at least one other video feed of the filmed occurrence.

## Claims

### Claim 1

A method for accessing content related to location in a video, the method comprising:
receiving a reference to a pixel location in a frame of a video feed of a filmed occurrence;
accessing a spatio-temporal index corresponding to the filmed occurrence, wherein the spatio-temporal index indexes information relating to events or objects of the filmed occurrence and corresponding pixel locations at which the events or the objects are detected in the video feed;
querying the spatio-temporal index using the pixel location to determine particular information of an indexed event or an indexed object; and
receiving the particular information wherein the particular information indicates at least one of spatial and temporal alignment parameters for aligning the indexed event with a corresponding event in at least one other video feed of the filmed occurrence.

### Claim 2 (depends on 1)

The method of claim 1, further comprising displaying the aligned event and corresponding event based on the particular information on a user interface.

### Claim 3 (depends on 1)

The method of claim 1, wherein receiving the particular information further comprises receiving statistics for a participant in the event, wherein event in the filmed occurrence is of a sporting event associated with the pixel location.

### Claim 4 (depends on 1)

The method of claim 1, wherein receiving the particular information further comprises receiving statistics relating to a matchup of a sporting event associated with the pixel location.

### Claim 5 (depends on 1)

The method of claim 1, further comprising querying the spatio-temporal index using an event identifier in the particular information of the indexed event relevant to the pixel location.

### Claim 6 (depends on 1)

The method of claim 1, wherein the video feed of the filmed occurrence is a tracking video feed.

### Claim 7

A method for accessing content related to location in a video, the method comprising:
receiving a reference to a pixel location in a frame of a video feed of a filmed occurrence;
accessing a spatio-temporal index corresponding to the filmed occurrence, wherein the spatio-temporal index indexes information relating to events or objects of the filmed occurrence and corresponding pixel locations at which the events or the objects are detected in the video feed;
querying the spatio-temporal index using the pixel location to determine particular information of an indexed event or an indexed object; and
receiving the particular information, wherein the indexed object is an advertisement overlay object, and the particular information relates to placement of an advertisement as an overlay with respect to the video feed.

### Claim 8 (depends on 7)

The method of claim 7, further comprising displaying the advertisement as an overlay of the video feed based on the particular information on a user interface.

### Claim 9 (depends on 7)

The method of claim 7, wherein the particular information is indexed in the spatio-temporal index for a plurality of pixel locations and a plurality of respective video frames.

### Claim 10 (depends on 7)

The method of claim 7, wherein the particular information further indicates a playing surface depicted in the video feed.

### Claim 11 (depends on 10)

The method of claim 10, wherein the video feed corresponds to a filmed occurrence and wherein the filmed occurrence is a sporting event taking place on the playing surface.

### Claim 12 (depends on 7)

The method of claim 7, wherein the particular information further indicates one or more participants in the indexed event that is relative to the pixel location.

### Claim 13 (depends on 7)

The method of claim 7, wherein the particular information is textual.

### Claim 14 (depends on 7)

The method of claim 7, wherein the particular information is graphical.

### Claim 15 (depends on 7)

The method of claim 7, wherein the particular information indicates if the pixel location corresponds to a location where at least one of information, augmentations, graphics, animations, or advertising may be displayed over a content frame.

### Claim 16 (depends on 7)

The method of claim 7, further comprising providing a smart pipe related to the pixel location, wherein the smart pipe comprises multiple aligned content channels associated with the pixel.

### Claim 17 (depends on 7)

The method of claim 7, wherein the indexed object or the indexed event relates to at least a group of pixels in each frame.

### Claim 18

A method for accessing content related to location in a video, the method comprising:
receiving a reference to a pixel location in a frame of a video feed of a filmed occurrence;
accessing a spatio-temporal index corresponding to the filmed occurrence,
querying the spatio-temporal index using the pixel location to determine particular information stored in the spatio-temporal index for an indexed object or an indexed event that is associated with the pixel location; and
receiving the particular information from the spatio-temporal index;
wherein the spatio-temporal index relates respective second pixel locations of the indexed object or an indexed event a second video feed with respective pixel locations of the indexed object or an indexed event in the video feed and respective spatial locations of the indexed object of the indexed event as determined from a tracking video feed.

### Claim 19 (depends on 18)

The method of claim 18, wherein the indexed object that is associated with the location of the pixel is a participant of a sporting event and the particular information includes statistics relating to the participant.

### Claim 20 (depends on 18)

The method of claim 18, wherein the indexed event that is associated with the location of the pixel is a matchup of a sporting event and the particular information includes statistics relating to the matchup.
