# US 2019/0356948 A1 · Machine learning for recognizing and interpreting embedded information card content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0356948 A1
- Document ID: US20190356948A1
- Status: Published application
- Kind: A1
- Application number: 16/411,710
- Filed: 2019-05-14
- Published: 2019-11-21
- Earliest claimed: 2018-05-18
- Assignees: Thuuz, Inc.
- Applicants: Thuuz, Inc.
- Inventors: Mihailo Stojancic; Warren Packard
- Web page: https://patentgrep.com/patent/US20190356948A1

## CPC

- G06V 20/42: {of sport video content}
- H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)}
- G06F 16/908: using metadata automatically derived from the content
- G06T 7/12: Edge-based segmentation
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/48: {Matching video sequences}
- G06V 20/635: {Overlay text, e.g. embedded captions in a TV programme}
- G06V 30/153: {using recognition of characters or words}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/234336: {by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text}
- H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)}
- H04N 21/435: Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/458: Scheduling content for creating a personalised stream, e.g. by combining a locally stored advertisement with an incoming stream; Updating operations, e.g. for OS modules {; time-related management operations (arrangements for replacing or switching information during the broadcast or during the distribution H04H20/10)}
- H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme}
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- G06T 2207/20132: Image cropping
- G06V 20/44: {Event detection}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Metadata for highlights of a video stream is extracted from card images embedded in the video stream. The highlights may be segments of a video stream, such as a broadcast of a sporting event, that are of particular interest to one or more users. Card images embedded in video frames of the video stream are identified and processed to extract text. The text characters may be recognized by applying a machine-learned model trained with a set of characters extracted from card images embedded in sports television programming contents. The training set of character vectors may be pre-processed to maximize metric distance between the training set members. The text may be interpreted to obtain the metadata. The metadata may be stored in association with the portion of the video stream. The metadata may provide information regarding the highlights, and may be presented concurrently with playback of the highlights.

## Claims

### Claim 1

A method for extracting metadata from a video stream, the method comprising:
at a processor, receiving at least a portion of the video stream;
at the processor, identifying one or more card images embedded in one or more video frames of the portion of the video stream;
at the processor, processing the one or more card images to extract text;
at the processor, interpreting the text to obtain metadata; and
at a data store, storing the metadata in association with the portion of the video stream.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
at the data store, storing the received portion of the video stream.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the video stream comprises a television broadcast of a sporting event;
the portion of the video stream comprises a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of the highlight.

### Claim 4 (depends on 3)

The method of claim 3, further comprising, at an output device, outputting the video stream concurrently with at least one of identifying the one or more card images, processing the one or more card images, and interpreting the text.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
at an output device, outputting the highlight; and
concurrently with outputting the highlight, outputting the metadata;
wherein the metadata comprises at least one selected from the group consisting of:
real-time information related to the highlight; and
a timeline of the card images from which the metadata have been obtained.

### Claim 6 (depends on 1)

The method of claim 1, wherein extracting the text comprises:
identifying one or more character strings within the one or more card images; and
recording a location and/or a size of a character image of a card image of the one or more card images that corresponds to each character of the one or more character strings.

### Claim 7 (depends on 6)

The method of claim 6, wherein extracting the text further comprises:
disambiguating character boundaries of characters of the one or more character strings by performing multiple comparisons of detected character boundaries; and
purging any character boundaries that appear too close to each other.

### Claim 8 (depends on 6)

The method of claim 6, wherein extracting the text further comprises performing image validation for characters of the one or more character strings by establishing a contrast ratio between low and high intensity pixel counts.

### Claim 9 (depends on 1)

The method of claim 1, wherein interpreting the text comprises:
generating queries based on the text;
generating a plurality of n-dimensional query feature vectors;
projecting the n-dimensional query feature vectors onto a training set orthogonal basis;
applying the projected n-dimensional query feature vectors to a classification model to produce at least one predicted query; and
extracting meaning of the text from the at least one predicted query.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
generating a plurality of training set feature vectors; and
using the training set feature vectors to derive the training set orthogonal basis.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
generating a plurality of training set feature vectors; and
using the training set feature vectors to generate the classification model.

### Claim 12 (depends on 9)

The method of claim 9, wherein interpreting the text further comprises using at least two selections from the group consisting of:
a string length of one or more character strings within the text;
a position of character boundaries and/or characters within the text; and
a horizontal position of character boundaries and/or characters within the text.

### Claim 13 (depends on 9)

The method of claim 9, wherein storing the metadata in association with the portion of the video stream comprises storing a video frame number of the one or more video frames, associated with queries.

### Claim 14 (depends on 1)

The method of claim 1, wherein interpreting the text comprises:
ascertaining field positions of characters of one or more character strings of the text;
ascertaining alphanumeric values of the characters; and
using the field positions and alphanumeric values to sequentially interpret the one or more character strings.

### Claim 15 (depends on 14)

The method of claim 14, wherein interpreting the text further comprises:
obtaining positional and other information regarding one or more card fields of each of the card images; and
using the positional and other information to compensate for one or more possible missing front characters of the one or more character strings.

### Claim 16

A method for generating a classification model for extracting metadata from a video stream, the method comprising:
at a processor, receiving at least a portion of the video stream;
at a processor, identifying one or more card images embedded in one or more video frames of the portion of the video stream;
at the processor, processing the one or more card images to extract a plurality of character images, each if which contains a character;
at the processor, generating training feature vectors associated with the character images;
at the processor, processing the training feature vectors in a manner that:
increases uniqueness of the training feature vectors;
increases mutual numeric distance of the training feature vectors; and/or
reduces dimensionality of an overall vector space containing the training feature vectors;
at the processor, using at least some of the training feature vectors to train a classification model; and
at a data store, storing the classification model.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
at the data store, storing the received portion of the video stream.

### Claim 18 (depends on 16)

The method of claim 16, further comprising, at the processor, prior to generating the training feature vectors, normalizing the character images to a standard size and/or a standard illumination.

### Claim 19 (depends on 16)

The method of claim 16, wherein generating the training feature vectors comprises formatting a set of n pixels extracted from the character images into n-dimensional vectors.

### Claim 20 (depends on 16)

The method of claim 16, further comprising, at the processor, performing a principal component analysis on the training feature vectors;
wherein using at least some of the training feature vectors to train the classification model comprises:
selecting a subset of the training feature vectors that are orthogonal basis vectors; and
using the orthogonal basis vectors to train the classification model.

### Claim 21 (depends on 20)

The method of claim 20, wherein:
the orthogonal basis vectors span the overall vector space;
reducing the dimensionality of the overall vector space comprises selecting a limited number of the orthogonal basis vectors;
reducing the dimensionality of the overall vector space further comprises selecting only orthogonal basis vectors that correspond to a largest set of singular values derived from a matrix of the orthogonal basis vectors;
storing the classification model comprises storing a limited number of the orthogonal basis vectors for subsequent use in classification model generation and/or query processing; and/or
generating the classification model comprises using a limited number of the orthogonal basis vectors in conjunction with a machine learning algorithm selected from the group consisting of SVM and CNN.

### Claim 22 (depends on 16)

The method of claim 16, further comprising:
at the processor, processing the one or more card images to extract text;
at the processor, interpreting the text to obtain metadata;
at the data store, storing the metadata in association with the portion of the video stream;
at an output device, outputting the portion of the video stream; and
at the output device, concurrently with outputting the portion of the video stream, outputting the metadata;
wherein:
the video stream comprises a broadcast of a sporting event;
the portion of the video stream comprises a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of the highlight.

### Claim 23 (depends on 22)

The method of claim 22, wherein extracting the text comprises extracting text strings of the text as queries.

### Claim 24 (depends on 22)

The method of claim 22, wherein extracting the text comprises extracting at least one of:
of a current time within the sporting event;
a current phase of the sporting event;
a game clock pertaining to the sporting event; and
a game score pertaining to the sporting event.

### Claim 25

A non-transitory computer-readable medium for extracting metadata from a video stream, comprising instructions stored thereon, that when executed by a processor, perform the steps of:
receiving at least a portion of the video stream;
identifying one or more card images embedded in one or more video frames of the portion of the video stream;
processing the one or more card images to extract text;
interpreting the text to obtain metadata; and
causing a data store to store the metadata in association with the portion of the video stream.

### Claim 26 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein:
the video stream comprises a television broadcast of a sporting event;
the portion of the video stream comprises a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of the highlight.

### Claim 27 (depends on 26)

The non-transitory computer-readable medium of claim 26, further comprising instructions stored thereon, that when executed by a processor, cause an output device to output the video stream concurrently with at least one of identifying the one or more card images, processing the one or more card images, and interpreting the text.

### Claim 28 (depends on 26)

The non-transitory computer-readable medium of claim 26, further comprising instructions stored thereon, that when executed by a processor:
cause an output device to output the highlight; and
concurrently with outputting the highlight, output the metadata;
wherein the metadata comprises at least one selected from the group consisting of:
real-time information related to the highlight; and
a timeline of the card images from which the metadata have been obtained.

### Claim 29 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein extracting the text comprises:
identifying one or more character strings within the one or more card images; and
recording a location and/or a size of a character image of a card image of the one or more card images that corresponds to each character of the one or more character strings.

### Claim 30 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein interpreting the text comprises:
generating queries based on the text;
generating a plurality of n-dimensional query feature vectors;
projecting the n-dimensional query feature vectors onto a training set orthogonal basis;
applying the projected n-dimensional query feature vectors to a classification model to produce at least one predicted query; and
extracting meaning of the text from the at least one predicted query.

### Claim 31 (depends on 30)

The non-transitory computer-readable medium of claim 30, further comprising instructions stored thereon, that when executed by a processor:
generate a plurality of training set feature vectors; and
use the training set feature vectors to derive the training set orthogonal basis and/or generate the classification model.

### Claim 32 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein interpreting the text comprises:
ascertaining field positions of characters of one or more character strings of the text;
ascertaining alphanumeric values of the characters; and
using the field positions and alphanumeric values to sequentially interpret the one or more character strings.

### Claim 33

A non-transitory computer-readable medium for generating a classification model for extracting metadata from a video stream, comprising instructions stored thereon, that when executed by a processor, perform the steps of:
receiving at least a portion of the video stream;
identifying one or more card images embedded in one or more video frames of the portion of the video stream;
processing the one or more card images to extract a plurality of character images, each if which contains a character;
generating training feature vectors associated with the character images;
processing the training feature vectors in a manner that:
increases uniqueness of the training feature vectors;
increases mutual numeric distance of the training feature vectors; and/or
reduces dimensionality of an overall vector space containing the training feature vectors;
using at least some of the training feature vectors to train a classification model; and
causing a data store to store the classification model.

### Claim 34 (depends on 33)

The non-transitory computer-readable medium of claim 33, further comprising instructions stored thereon, that when executed by a processor, perform a principal component analysis on the training feature vectors;
wherein using at least some of the training feature vectors to train the classification model comprises:
selecting a subset of the training feature vectors that are orthogonal basis vectors; and
using the orthogonal basis vectors to train the classification model.

### Claim 35 (depends on 34)

The non-transitory computer-readable medium of claim 34, wherein:
the orthogonal basis vectors span the overall vector space;
reducing the dimensionality of the overall vector space comprises selecting a limited number of the orthogonal basis vectors;
reducing the dimensionality of the overall vector space further comprises selecting only orthogonal basis vectors that correspond to a largest set of singular values derived from a matrix of the orthogonal basis vectors;
storing the classification model comprises storing a limited number of the orthogonal basis vectors for subsequent use in classification model generation and/or query processing; and/or
generating the classification model comprises using a limited number of the orthogonal basis vectors in conjunction with a machine learning algorithm selected from the group consisting of SVM and CNN.

### Claim 36 (depends on 33)

The non-transitory computer-readable medium of claim 33, further comprising instructions stored thereon, that when executed by a processor, perform the steps of:
processing the one or more card images to extract text;
interpreting the text to obtain metadata;
causing the data store to store the metadata in association with the portion of the video stream;
causing an output device to output the portion of the video stream; and
causing the output device, concurrently with outputting the portion of the video stream, to output the metadata;
wherein:
the video stream comprises a broadcast of a sporting event;
the portion of the video stream comprises a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of the highlight.

### Claim 37

A system for extracting metadata from a video stream, the system comprising:
a processor configured to:
receive at least a portion of the video stream;
identify one or more card images embedded in one or more video frames of the portion of the video stream;
process the one or more card images to extract text; and
interpret the text to obtain metadata; and
a data store configured to store the metadata in association with the portion of the video stream.

### Claim 38 (depends on 37)

The system of claim 37, wherein:
the video stream comprises a television broadcast of a sporting event;
the portion of the video stream comprises a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of the highlight.

### Claim 39 (depends on 38)

The system of claim 38, further comprising an output device configured to output the video stream concurrently with at least one of identifying the one or more card images, processing the one or more card images, and interpreting the text.

### Claim 40 (depends on 38)

The system of claim 38, further comprising an output device configured to output the highlight;
wherein:
the processor is further configured to, concurrently with outputting the highlight, output the metadata; and
the metadata comprises at least one selected from the group consisting of:
real-time information related to the highlight; and
a timeline of the card images from which the metadata have been obtained.

### Claim 41 (depends on 37)

The system of claim 37, wherein the processor is further configured to extract the text by:
identifying one or more character strings within the one or more card images; and
recording a location and/or a size of a character image of a card image of the one or more card images that corresponds to each character of the one or more character strings.

### Claim 42 (depends on 37)

The system of claim 37, wherein the processor is further configured to interpret the text by:
generating queries based on the text;
generating a plurality of n-dimensional query feature vectors;
projecting the n-dimensional query feature vectors onto a training set orthogonal basis;
applying the projected n-dimensional query feature vectors to a classification model to produce at least one predicted query; and
extracting meaning of the text from the at least one predicted query.

### Claim 43 (depends on 42)

The system of claim 42, wherein the processor is further configured to:
generate a plurality of training set feature vectors; and
use the training set feature vectors to derive the training set orthogonal basis and/or generate the classification model.

### Claim 44 (depends on 37)

The system of claim 37, wherein the processor is further configured to interpret the text by:
ascertaining field positions of characters of one or more character strings of the text;
ascertaining alphanumeric values of the characters; and
using the field positions and alphanumeric values to sequentially interpret the one or more character strings.

### Claim 45

A system for generating a classification model for extracting metadata from a video stream, the system comprising:
a processor configured to:
receive at least a portion of the video stream;
identify one or more card images embedded in one or more video frames of the portion of the video stream;
process the one or more card images to extract a plurality of character images, each if which contains a character;
generate training feature vectors associated with the character images;
process the training feature vectors in a manner that:
increases uniqueness of the training feature vectors;
increases mutual numeric distance of the training feature vectors; and/or
reduces dimensionality of an overall vector space containing the training feature vectors; and
use at least some of the training feature vectors to train a classification model; and
a data store configured to store the classification model.

### Claim 46 (depends on 45)

The system of claim 45, wherein the processor is further configured to:
perform a principal component analysis on the training feature vectors; and
use at least some of the training feature vectors to train the classification model by:
selecting a subset of the training feature vectors that are orthogonal basis vectors; and
using the orthogonal basis vectors to train the classification model.

### Claim 47 (depends on 46)

The system of claim 46, wherein:
the orthogonal basis vectors span the overall vector space; and
the processor is further configured to reduce the dimensionality of the overall vector space by selecting a limited number of the orthogonal basis vectors;
the processor is further configured to reduce the dimensionality of the overall vector space by selecting only orthogonal basis vectors that correspond to a largest set of singular values derived from a matrix of the orthogonal basis vectors;
the data store is further configured to store the classification model by storing a limited number of the orthogonal basis vectors for subsequent use in classification model generation and/or query processing; and/or
the processor is further configured to generate the classification model by using a limited number of the orthogonal basis vectors in conjunction with a machine learning algorithm selected from the group consisting of SVM and CNN.

### Claim 48 (depends on 45)

The system of claim 45, wherein:
the processor is further configured to:
process the one or more card images to extract text; and
interpret the text to obtain metadata;
the data store is further configured to store the metadata in association with the portion of the video stream;
the system further comprises an output device configured to:
output the portion of the video stream; and
concurrently with outputting the portion of the video stream, output the metadata;
the video stream comprises a broadcast of a sporting event;
the portion of the video stream comprises a highlight deemed to be of particular interest to one or more users; and
the metadata is descriptive of the highlight.
