# US 2017/0255827 A1 · Methods and systems of spatiotemporal pattern recognition for video content development

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0255827 A1
- Document ID: US20170255827A1
- Status: Published application
- Kind: A1
- Application number: 15/600,379
- Filed: 2017-05-19
- Published: 2017-09-07
- Earliest claimed: 2014-02-28
- Assignees: Second Spectrum, Inc.
- Applicants: Second Spectrum, Inc.
- Inventors: Yu-Han Chang; Rajiv Maheswaran; Jeffrey Wayne Su; Noel Grant Hollingsworth
- Web page: https://patentgrep.com/patent/US20170255827A1

## CPC

- G06V 20/42: {of sport video content}
- A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06N 20/00: Machine learning
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- H04N 5/2224: {related to virtual studio applications}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4345: {Extraction or processing of SI, e.g. extracting service information from an MPEG stream}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- H04N 21/4662: {characterized by learning algorithms}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06T 2207/20081: Training; Learning
- G06T 2207/30221: Sports video; Sports image
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Producing an event related video content data structure includes processing a video feed through a spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the video feed. Developing the understanding includes identifying context information relating to the event and identifying an entry in a relationship library at least detailing a relationship between two visible features of the video feed. Content of the video feed that displays the event is automatically extracted by a computer and associated with the context information. A video content data structure that includes the context information is produced.

## Claims

### Claim 1

A method, comprising:
taking a video feed;
processing the video feed through a spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the video feed, wherein the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the video feed;
automatically, under computer control, extracting content displaying the event and associating the extracted content with the context information relating to the event; and
producing a video content data structure that includes the context information relating to the event.

### Claim 2 (depends on 1)

The method of claim 1, further comprising determining a plurality of semantic categories for the context information relating to the event, and filtering a plurality of video content data structures based on the plurality of semantic categories, wherein each of the video content data structures of the plurality of video content data structures includes context information relating to the event.

### Claim 3 (depends on 1)

The method of claim 1, further comprising matching events that occur in a first video feed to events that occur in a second video feed, the second video feed separate from the first video feed, and using a semantic understanding captured in the first video feed to at least one of filter and cut a separate second video feed based on the matching events that occur in the first video feed and the second video feed.

### Claim 4 (depends on 1)

The method of claim 1, further comprising determining a pattern relating to a plurality of events and providing a content data structure based on the pattern.

### Claim 5 (depends on 4)

The method of claim 4, wherein the pattern comprises a plurality of important plays in a sports event that are identified based on comparison to similar plays from previous sports events.

### Claim 6 (depends on 4)

The method of claim 4, wherein the pattern comprises a plurality of plays in a sports event that is determined to be unusual based on comparison to video feeds from other sports events.

### Claim 7 (depends on 1)

The method of claim 1, further comprising extracting semantic events over time to draw a comparison of at least one of a player and a team over time.

### Claim 8 (depends on 7)

The method of claim 7, further comprising superimposing video of events extracted from video feeds from at least two different time periods to illustrate the comparison.

### Claim 9 (depends on 1)

The method of claim 1, further comprising allowing a user to interact with the video content data structure to produce an edited video data stream that includes the video data structure.

### Claim 10 (depends on 9)

The method of claim 9, wherein the interaction includes at least one of editing, mixing, cutting, and sharing a video clip that includes the video content data structure.

### Claim 11 (depends on 9)

The method of claim 9, further comprising enabling users to interact with the video content data structure through a user interface to enhance the video content data structure with at least one graphic element selected from a menu of options.

### Claim 12 (depends on 11)

The method of claim 11, further comprising enabling a user to share the enhanced video content.

### Claim 13 (depends on 1)

The method of claim 1, further comprising enabling a user to find similar video clips based on a semantic context identified in the video clips.

### Claim 14 (depends on 1)

The method of claim 1, further comprising using the video data structure and the context information relating to the event to construct modified video content for a second screen that includes the video content data structure.

### Claim 15 (depends on 14)

The method of claim 14, wherein the modified video content for the second screen correlates to the timing of an event displayed on a first screen.

### Claim 16 (depends on 14)

The method of claim 14, wherein the modified video content for the second screen includes a metric determined based on the machine understanding, wherein the metric is selected based on the context information relating to the event.

### Claim 17 (depends on 1)

The method of claim 1, wherein using the machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the video feed and a chip-based player tracking system, and wherein the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

### Claim 18 (depends on 1)

The method of claim 1, wherein using the machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

### Claim 19 (depends on 18)

The method of claim 18, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

### Claim 20 (depends on 18)

The method of claim 18, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

### Claim 21 (depends on 1)

The method of claim 1, further comprising at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

### Claim 22 (depends on 1)

The method of claim 1, further comprising automatically developing a semantic index of the video feed based on the understanding developed with the machine learning of the event in the video feed to indicate a time of the event in the video feed and a location of a display of the event in the video feed.

### Claim 23 (depends on 22)

The method of claim 22, wherein the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.

### Claim 24 (depends on 22)

The method of claim 22, further comprising providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.

### Claim 25 (depends on 24)

The method of claim 24, wherein the augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.

### Claim 26 (depends on 1)

The method of claim 1, wherein automatically, under computer control, extracting content displaying the event includes automatically extracting a cut from the video feed using a combination of the understanding of events developed from the machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

### Claim 27 (depends on 26)

The method of claim 26, wherein the understanding of the other input feed developed by the machine learning includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

### Claim 28

A system, comprising:
an ingestion facility for ingesting a plurality of video feeds;
a machine learning system for processing the video feed through a spatiotemporal pattern recognition algorithm that applies machine learning on a series of events in the plurality of video feeds in order to develop an understanding of the series of events within the plurality of video feeds, wherein the understanding includes identifying context information relating to the series of events and an entry in a relationship library at least detailing a relationship between two visible features of the plurality of video feeds;
an extraction facility for automatically, under computer control, extracting content displaying the series of events and associating the extracted content with the context information; and
a video publishing facility for producing a video content data structure that includes the context information.

### Claim 29 (depends on 28)

The system of claim 28, further comprising an analytic facility for determining a plurality of semantic categories for the context information, and filtering the plurality of video content data structures based on the semantic categories.

### Claim 30 (depends on 28)

The system of claim 28, further comprising a matching engine for matching a series of events that occur in a first video feed to those that occur in a second video feed, the first video feed being separate from the second video feed, and using a semantic understanding captured in the first video feed to at least one of filter and cut the second video feed based on the matching of the series of events that occur in the first video feed with those that occur in the second video feed.

### Claim 31 (depends on 28)

The system of claim 28, further comprising a pattern recognition facility that determines a pattern relating to the series of events and provides a content data structure based on the pattern.

### Claim 32 (depends on 28)

The system of claim 28, wherein using machine learning to develop the understanding of the series of events further comprises using events in position tracking data over time obtained from the plurality of video feeds and a chip-based player tracking system, and wherein the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

### Claim 33 (depends on 28)

The system of claim 28, wherein using machine learning to develop the understanding of the series of events further comprises aligning multiple unsynchronized input feeds related to the series of events using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

### Claim 34 (depends on 33)

The system of claim 33, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

### Claim 35 (depends on 33)

The system of claim 33, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds, wherein at least one algorithm in the hierarchy for validation of the alignment of the unsynchronized input feeds is based on a nature of the input feeds.

### Claim 36 (depends on 28)

The system of claim 28, further comprising at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds, wherein the one or more algorithms is based on the nature of the input feed.

### Claim 37 (depends on 28)

The system of claim 28, further comprising automatically developing a semantic index of a video feed from the plurality of video feeds based on the understanding developed by the machine learning of at least one event from the series of events in the video feed, wherein the semantic index of the video feed indicates a time of the event in the video feed and a location of a display of the event in the video feed.

### Claim 38 (depends on 37)

The system of claim 37, wherein the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.

### Claim 39 (depends on 37)

The system of claim 37, further comprising providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.

### Claim 40 (depends on 39)

The system of claim 39, wherein augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
