# US 2017/0255826 A1 · Methods and systems of spatiotemporal pattern recognition for video content development

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0255826 A1
- Document ID: US20170255826A1
- Status: Published application
- Kind: A1
- Application number: 15/600,355
- Filed: 2017-05-19
- Published: 2017-09-07
- Earliest claimed: 2014-02-28
- Assignees: Second Spectrum, Inc.
- Applicants: Second Spectrum, Inc.
- Inventors: Yu-Han Chang; Rajiv Maheswaran; Jeffrey Wayne Su; Noel Grant Hollingsworth
- Web page: https://patentgrep.com/patent/US20170255826A1

## CPC

- G06V 20/42: {of sport video content}
- A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06N 20/00: Machine learning
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- H04N 5/2224: {related to virtual studio applications}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4345: {Extraction or processing of SI, e.g. extracting service information from an MPEG stream}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- H04N 21/4662: {characterized by learning algorithms}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06T 2207/20081: Training; Learning
- G06T 2207/30221: Sports video; Sports image
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Presenting event-specific video content that conforms to a user selection of an event type includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed, extracting the video content displaying the at least one event and associating the understanding with the video content in a video content data structure. A user interface is configured to permit a user to indicate a preference for at least one event type that is used to retrieve and provide corresponding extracted video content with the data structure in a new video feed.

## Claims

### Claim 1

A method for enabling a user to express preferences relating to display of video content, comprising:
processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;
automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in a video content data structure;
providing a user interface configured to permit a user to indicate a preference for at least one event type;
upon receiving an indication of the preference by the user, retrieving the at least one video content data structure that was determined by the machine learning to be associated with the at least one event type indicated by the user; and
providing the user with a video feed containing the video content and including the at least one video content data structure.

### Claim 2 (depends on 1)

The method of claim 1, wherein using machine learning to develop the understanding of the at least one event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

### Claim 3 (depends on 1)

The method of claim 1, wherein using machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the at least one event, one or more feeds of tracking video for the at least one event, and one or more play-by-play data feeds of the at least one event.

### Claim 4 (depends on 3)

The method of claim 3, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the at least one event.

### Claim 5 (depends on 3)

The method of claim 3, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds.

### Claim 6 (depends on 1)

The method of claim 1, further comprising at least one of validating the understanding and modifying the understanding developed by the machine learning using a hierarchy involving at least two of at least one algorithm, at least one human operator, and at least one input feed where at least one item in the hierarchy used for validating the understanding is based on a nature of the one or more input feeds.

### Claim 7 (depends on 1)

The method of claim 1, wherein the user interface is of at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, a touch screen device, a virtual reality or augmented reality headset, and a smart phone.

### Claim 8 (depends on 1)

The method of claim 1, wherein the user interface further comprises an element configured to permit a user to indicate a preference as to how content will be presented to the user.

### Claim 9 (depends on 1)

The method of claim 1, wherein the understanding developed by the machine learning includes a context for the at least one of event and the context is stored with the at least one video content data structure.

### Claim 10 (depends on 9)

The method of claim 9, wherein the user interface further comprises an element configured to permit a user to indicate a preference for the context.

### Claim 11 (depends on 10)

The method of claim 10, further comprising retrieving portion of the video content corresponding to the context and displaying the portion of the video content corresponding to the context to the user upon receiving an indication of the preference for the context.

### Claim 12 (depends on 11)

The method of claim 11, wherein the context comprises at least one of a presence of a preferred player in the at least one video feed, a preferred matchup of players in the at least one video feed, a preferred team in the at least one video feed, and a preferred matchup of teams in the at least one video feed.

### Claim 13 (depends on 1)

The method of claim 1, wherein the user interface is configured to permit a user to select at least one of a metric and a graphic element to be displayed on the video feed, wherein the at least one metric is based at least in part on the understanding developed by the machine learning.

### Claim 14 (depends on 1)

The method of claim 1, wherein extracting the content displaying the at least one event includes automatically extracting a cut from the video feed using a combination of the understanding of events developed with the machine learning and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

### Claim 15 (depends on 14)

The method of claim 14, wherein the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

### Claim 16

A method for enabling a mobile application that allows user interaction with video content, the method comprising:
taking a video feed;
processing the video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the video feed, wherein the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the video feed;
automatically, under computer control, extracting content displaying the event and associating the extracted content with the context information;
producing a video content data structure that includes the context information; and
automatically, under computer control, producing a story that includes the video content data structure, wherein a portion of the story is based on a user preference, the context information, and the video content data structure.

### Claim 17 (depends on 16)

The method of claim 16, wherein extracting the content displaying the event includes automatically extracting a cut from the video feed using a combination of the understanding of the event developed with the machine learning and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

### Claim 18 (depends on 17)

The method of claim 17, wherein the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

### Claim 19 (depends on 16)

The method of claim 16, further comprising using a combination of the understanding developed with the machine learning of the event from the video feed and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed to at least one of edit the video cut and combine the video cut with other content.

### Claim 20 (depends on 19)

The method of claim 19, wherein the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

### Claim 21 (depends on 16)

The method of claim 16, further comprising automatically developing a semantic index of a video feed based on the understanding developed with the machine learning of at least one event in the video feed indicative of a time of the event in the video feed and a location of a display of the event in the video feed.

### Claim 22 (depends on 21)

The method of claim 21, wherein the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.

### Claim 23 (depends on 21)

The method of claim 21, further comprising providing the semantic index of the video feed with the video feed in combination to enable augmentation of the video feed.

### Claim 24 (depends on 23)

The method of claim 23, wherein the augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.

### Claim 25 (depends on 16)

The method of claim 16, wherein using machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the video feed and a chip-based player tracking system, and wherein the understanding developed with machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

### Claim 26 (depends on 16)

The method of claim 16, wherein using machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

### Claim 27 (depends on 26)

The method of claim 26, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

### Claim 28 (depends on 26)

The method of claim 26, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on a nature of the input feeds.

### Claim 29 (depends on 16)

The method of claim 16, further comprising at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.

### Claim 30 (depends on 16)

The method of claim 16, wherein the user preference for a type of content is based on at least one of a user expressed preference and a preference that is inferred based on user interaction with an item of content.

### Claim 31

A system for enabling a user to express preferences relating to display of video content, comprising:
a machine learning facility that uses at least one spatiotemporal pattern recognition algorithm for developing an understanding of at least one event within at least one video feed to determine at least one event type, wherein the understanding includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;
a video production facility for automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in at least one video content data structure; and
a server for serving data to a user interface that is configured to permit a user to indicate a preference for at least one event type, that retrieves the at least one video content data structure that was determined by the machine learning to have a event type preferred by the user, and that provides the user with a video feed containing the event type preferred by the user.

### Claim 32 (depends on 31)

The system of claim 31, wherein the user interface that is configured to permit the user to indicate the preference for at least one event type is at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, and a smart phone.

### Claim 33 (depends on 31)

The system of claim 31, wherein the user interface that is configured to permit the user to indicate the preference for at least one event type further comprises an element configured to permit a user to indicate a preference as to how content will be presented to the user.

### Claim 34 (depends on 31)

The system of claim 31, wherein the understanding developed by the machine learning facility includes a context for the at least one event and the context is stored with the at least one video content data structure.

### Claim 35 (depends on 34)

The system of claim 34, wherein the user interface further comprises an element configured to permit a user to indicate a preference for at least one context.

### Claim 36 (depends on 35)

The system of claim 35, wherein the server for serving data to the user interface retrieves video content corresponding to the preference for at least one context and displays the video content to the user upon receiving an indication of the preference for the at least one context.

### Claim 37 (depends on 36)

The system of claim 36, wherein the at least one context comprises at least one of a presence of a preferred player in the video feed, a preferred matchup of players in the video feed, a preferred team in the video feed, and a preferred matchup of teams in the video feed.

### Claim 38 (depends on 31)

The system of claim 31, wherein the user interface is configured to permit a user to select at least one of a metric and a graphic element to be displayed on the video feed, wherein the metric is based at least in part on the understanding developed with the machine learning facility.

### Claim 39 (depends on 31)

The system of claim 31, wherein the machine learning facility to develop the understanding of the at least one event uses events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

### Claim 40 (depends on 31)

The system of claim 31, wherein the machine learning facility to develop the understanding of the at least one event aligns multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

### Claim 41 (depends on 40)

The system of claim 40, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the at least one event.

### Claim 42 (depends on 40)

The system of claim 40, wherein the video production facility at least one of validates and modifies the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

### Claim 43 (depends on 31)

The system of claim 31, wherein the video production facility at least one of validates the understanding and modifies the understanding developed by the machine learning facility using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.

### Claim 44 (depends on 31)

The system of claim 31, wherein the video production facility for automatically, under computer control, extracting the video content displaying the at least one event includes automatically extracting a cut from the at least one video feed using a combination of the understanding of the at least one event developed with the machine learning facility and an understanding developed with the machine learning facility of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

### Claim 45 (depends on 44)

The system of claim 44, wherein the understanding developed with the machine learning facility of the another input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the at least one video feed.

### Claim 46

A method for delivering personalized video content, comprising:
processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;
automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in a video content data structure;
developing a personal profile for a user based on at least one of expressed preferences of the user, information about the user, and information collected about actions taken by the user with respect to at least one event type; and
upon receiving an indication of the user profile, retrieving at least one video content data structure that was determined by the machine learning to have a event type likely to be preferred by the user based on the personal profile for the user.

### Claim 47 (depends on 46)

The method of claim 46, wherein using the machine learning to develop the understanding of the at least one event further comprises using the at least one event in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding developed by the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

### Claim 48 (depends on 46)

The method of claim 46, wherein using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the at least one event.

### Claim 49 (depends on 48)

The method of claim 48, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

### Claim 50 (depends on 48)

The method of claim 48, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

### Claim 51 (depends on 46)

The method of claim 46, further comprising at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

### Claim 52 (depends on 51)

The method of claim 51, wherein extracting the video content displaying the at least one event includes automatically extracting a cut from the at least one video feed using a combination of the understanding of the at least one event developed with the machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

### Claim 53 (depends on 52)

The method of claim 52, wherein the understanding of the other input feed developed with the machine learning includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

### Claim 54

A method for delivering personalized video content, comprising:
processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed of a professional game, wherein the understanding developed by the machine learning includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed
develop an understanding using the machine learning of at least one event within a data feed relating to motion of a non-professional player; and
automatically, under computer control, providing an enhanced video feed that represents the non-professional player playing within a context of the professional game based on the understanding of the at least one event within the at least one video feed of the professional game and a data feed relating to the motion of the non-professional player.

### Claim 55 (depends on 54)

The method of claim 54, further comprising providing a facility having cameras for capturing 3D motion data, and capturing video of a non-professional player to provide the data feed for the non-professional player.

### Claim 56 (depends on 55)

The method of claim 55, wherein the non-professional player is represented by mixing video of the non-professional player with at least one video of the professional game.

### Claim 57 (depends on 56)

The method of claim 56, wherein the non-professional player is represented as an animation having attributes based on the data feed relating to motion of the non-professional player.

### Claim 58 (depends on 54)

The method of claim 54, wherein using the machine learning to develop the understanding of the at least one event further comprises using the at least one event in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

### Claim 59 (depends on 54)

The method of claim 54, wherein using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the at least one event.

### Claim 60 (depends on 59)

The method of claim 59, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

### Claim 61 (depends on 59)

The method of claim 59, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

### Claim 62 (depends on 54)

The method of claim 54, further comprising at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
