# US 2020/0401809 A1 · Methods and systems of combining video content with one or more augmentations to produce augmented video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401809 A1
- Document ID: US20200401809A1
- Status: Published application
- Kind: A1
- Application number: 17/006,962
- Filed: 2020-08-31
- Published: 2020-12-24
- Earliest claimed: 2014-02-28
- Assignees: Second Spectrum, Inc.
- Applicants: Second Spectrum, Inc.
- Inventors: Yu-Han Chang; Rajiv Tharmeswaran Maheswaran; Jeffrey Wayne Su; Emil Dotchevski; Jason Kent Simon
- Web page: https://patentgrep.com/patent/US20200401809A1

## CPC

- G06V 20/42: {of sport video content}
- A63F 13/60: Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/11: by using information not detectable on the record carrier
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- H04N 5/2224: {related to virtual studio applications}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/26603: {for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4345: {Extraction or processing of SI, e.g. extracting service information from an MPEG stream}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- H04N 21/4662: {characterized by learning algorithms}
- H04N 21/8455: {involving pointers to the content, e.g. pointers to the I-frames of the video stream}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06T 2207/20081: Training; Learning
- G06T 2207/30221: Sports video; Sports image
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Data processing systems and methods are disclosed for combining video content with one or more augmentations to produce augmented video. Objects within video content may have associated bounding boxes that may each be associated with respective RGB values. Upon user selection of a pixel, the RGBA value of the pixel may be used to determine a bounding box associated with the RGBA value. The client may transmit an indicator of the determined bounding box to an augmentation system to request augmentation data for the object associated with the bounding box. The system then uses the indicator to determine the augmentation data and transmits the augmentation data to the client device.

## Claims

### Claim 1

A computer-implemented data processing method for displaying augmented content on a client device, the method comprising:
receiving, from an external server by one or more processors, video data corresponding to an event, the video data comprising video content and a plurality of definitions of a plurality of bounding areas;
presenting, by one or more processors on a graphical user interface, the video content;
detecting, by one or more processors, a user selection of a portion of the graphical user interface;
determining, by one or more processors, a value associated with one or more pixels associated with the user selection of the portion of the graphical user interface;
determining, by one or more processors, a bounding area that corresponds to the value associated with the one or more pixels associated with the user selection of the portion of the graphical user interface, wherein the bounding area is associated with a particular bounding area from among the plurality of bounding areas;
transmitting, by one or more processors, an indication of the particular bounding area to a renderer;
receiving, by one or more processors from the renderer, augmentation data associated with the bounding area that corresponds to the value associated with the one or more pixels associated with the user selection of the portion of the graphical user interface;
generating, by one or more processors, augmented video content based on the video data and the augmentation data associated with the bounding area that corresponds to the value associated with the one or more pixels associated with the user selection of the portion of the graphical user interface; and
presenting, by one or more processors on the graphical user interface, the augmented video content.

### Claim 2 (depends on 1)

The computer-implemented data processing method of claim 1, wherein each definition of the plurality of definitions of the plurality of bounding areas is associated with a respective object represented in the video content.

### Claim 3 (depends on 2)

The computer-implemented data processing method of claim 2, wherein the augmentation data associated with the bounding area that corresponds to the value associated with the one or more pixels associated with the user selection of the portion of the graphical user interface comprises an augmentation image and video frame data and auxiliary data associated with the augmentation image.

### Claim 4 (depends on 3)

The computer-implemented data processing method of claim 3, wherein transmitting the indication of the particular bounding area to the renderer causes the renderer to select the augmentation image from among a plurality of augmentation images associated with the respective object.

### Claim 5 (depends on 1)

The computer-implemented data processing method of claim 1, wherein the renderer comprises one of an external rendering engine and a client-side rendering engine.

### Claim 6 (depends on 1)

The computer-implemented data processing method of claim 1, wherein detecting the user selection of the portion of the graphical user interface comprises detecting a user tap or a user click of a display on the client device displaying the graphical user interface.

### Claim 7

A video content augmentation system configured on a client device, the video content augmentation system comprising:
one or more computer processors;
memory storing computer-executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
receiving, from an external server, video data corresponding to an event, the video data comprising video content and a plurality of transparent bounding areas;
presenting, on a graphical user interface, the video content;
detecting a user selection of a portion of the graphical user interface;
determining a color value associated with the user selection of the portion of the graphical user interface;
selecting a particular transparent bounding area from among the plurality of transparent bounding areas by determining that a color value associated with the particular transparent bounding area corresponds to the color value associated with the user selection of the portion of the graphical user interface;
transmitting an indicator of the particular transparent bounding area to a renderer;
receiving, from the renderer, one or more augmentation images associated with the particular transparent bounding area;
receiving, from the renderer, video frame data and location data associated with the one or more augmentation images; and
presenting the one or more augmentation images on the graphical user interface based on the video frame data and the location data.

### Claim 8 (depends on 7)

The video content augmentation system of claim 7, wherein each of the one or more augmentation images is a Portable Network Graphics (PNG) image.

### Claim 9 (depends on 7)

The video content augmentation system of claim 7, wherein the video data further comprises a respective pre-defined color value for each transparent bounding area of the plurality of transparent bounding areas.

### Claim 10 (depends on 7)

The video content augmentation system of claim 7, wherein each transparent bounding area of the plurality of transparent bounding areas is associated with a respective object represented in the video content.

### Claim 11 (depends on 10)

The video content augmentation system of claim 10, wherein the event is a sporting event, and
wherein a respective object associated with at least one transparent bounding area of the plurality of transparent bounding areas corresponds to a player in the sporting event.

### Claim 12 (depends on 10)

The video content augmentation system of claim 10, wherein the event is a sporting event, and
wherein a respective object associated with at least one transparent bounding area of the plurality of transparent bounding areas corresponds to a non-player object in the sporting event.

### Claim 13 (depends on 7)

The video content augmentation system of claim 7, wherein presenting the one or more augmentation images on the graphical user interface based on the video frame data and the location data comprises presenting the one or more augmentation images on the graphical user interface in conjunction with the video content so that, when the one or more augmentation images are presented, the one or more augmentation images remain in a substantially fixed orientation relative to an object associated with the one or more augmentation images as the video content is presented on the graphical user interface.

### Claim 14 (depends on 7)

The video content augmentation system of claim 7, wherein the color value associated with the user selection of the portion of the graphical user interface comprises a red, green, blue, alpha (RGBA) value.

### Claim 15

A non-transitory computer-readable medium storing computer-executable instructions for:
determining a plurality of objects within each frame of a plurality of video frames of video content;
assigning a plurality of bounding areas to the plurality of objects, wherein each bounding area is associated with a respective object represented in one or more frames of the plurality of video frames;
assigning a respective color value to each bounding area of the plurality of bounding areas;
transmitting, to a client device, video data corresponding to an event, the video data comprising the video content, the plurality of bounding areas, and the assigned color values;
receiving, from the client device, an indicator of a particular bounding area identified from among the plurality of bounding areas, wherein the particular bounding area is identified based on the assigned color values of the plurality of bounding areas;
determining a particular object within a frame of the plurality of video frames of the video content based on the indicator of the particular bounding area;
determining a current augmentation state for the particular object;
selecting an augmentation image from among a plurality of augmentation images associated with the particular object based on the current augmentation state for the particular object;
determining video frame data and location data associated with the particular object; and
transmitting the augmentation image, the video frame data, and the location data to the client device.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein each of the assigned color values comprises an opacity value.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, further comprising computer-executable instructions for:
assigning an opacity value of 0 to each bounding area of the plurality of bounding areas.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein determining a current augmentation state for the particular object comprises determining that there is no current augmentation image associated with the particular object.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein selecting the augmentation image from among the plurality of augmentation images associated with the particular object based on the current augmentation state for the particular object comprises:
selecting a next sequential augmentation image from among a sequence of augmentation images, wherein the current augmentation state is associated with a current augmentation image, and wherein the next sequential augmentation image follows the current augmentation image in the sequence of augmentation images.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein each of the assigned color values comprises a red, green, blue, alpha (RGBA) value.