# US 2022/0030325 A1 · Methods and apparatus for efficient media search and analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0030325 A1
- Document ID: US20220030325A1
- Status: Published application
- Kind: A1
- Application number: 17/468,603
- Filed: 2021-09-07
- Published: 2022-01-27
- Earliest claimed: 2018-01-31
- Assignees: WowYow, Inc.
- Applicants: WowYow, Inc.
- Inventors: Adam Boskovich
- Web page: https://patentgrep.com/patent/US20220030325A1

## CPC

- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme}
- G06F 16/433: {using audio data}
- G06F 16/7837: {using objects detected or recognised in the video content}
- G06V 20/635: {Overlay text, e.g. embedded captions in a TV programme}
- G06V 20/64: Three-dimensional [3D] objects
- H04N 19/114: Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames (H04N19/107 takes precedence)
- H04N 21/2668: Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles {(information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063)}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- H04N 21/858: Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
- G06V 20/20: in augmented reality scenes
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/61: in combination with predictive coding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and apparatus for providing attribute-based search of media assets such as video and audio assets, and associated augmented reality functions including dynamic provision of relevant secondary content relating to the identified attribute(s). In one embodiment, media or content assets are ingested into a processing system and processed according to one or more attribute detection, identification, and characterization algorithms. Attributes detected may include for example the presence of tangible items such as clothing or chattels, particular persons such as celebrities, and/or certain contexts such as sporting activities and musical performances, as rendered within the media asset. In one implementation, the characterized assets are provided a unique ID, and stored so as to permit cross-correlation based on, e.g., the identified and characterized attributes. Secondary content (e.g., advertising or promotional content) is correlated to each identified asset, and dynamically served upon a user providing some threshold level of interaction with the attribute.

## Claims

### Claim 1

A computerized method for automatically providing secondary digital content relating to a user-cognizable digital element present in a digital content asset, the method comprising:
identifying a plurality of digital content assets;
processing, using a plurality of computer programs, the plurality of digital content assets, the processing comprising:
identifying at least one user-cognizable digital element within each of the plurality of identified digital content assets;
characterizing the identified at least one user-cognizable digital element in each of the plurality of identified digital content assets to generate characterizing data;
assigning data indicative of a unique identifier to each of the plurality of identified digital content assets;
storing the generated characterizing data along with at least a portion of the data indicative of the unique identifier in a database; and
forming an association between (i) at least a portion of the stored generated characterizing data of the identified at least one of the user-cognizable digital element in each of the plurality of identified digital content assets, and (ii) secondary digital content;
receiving data indicative of interest by a user in the at least one identified user-cognizable digital element within at least one of the plurality of identified digital content assets; and
based at least on the received data indicative of interest, causing provision of the secondary digital content to a computerized device associated with the user.

### Claim 2 (depends on 1)

The computerized method of claim 1, wherein:
the plurality of identified digital content assets comprise digitally encoded video data having a plurality of I-Frames, P-Frames, and B-Frames; and
the identifying the at least one user-cognizable digital element within each of the plurality of identified digital content assets comprises utilizing at least one shape recognition algorithm to analyze at least one I-Frame associated with the digitally encoded video data of each of the plurality of identified digital content assets.

### Claim 3 (depends on 2)

The computerized method of claim 2, wherein the utilizing the at least one shape recognition algorithm to analyze at least one I-Frame associated with the digitally encoded video data of each of the plurality of identified digital content assets comprises:
decoding the digital video data;
identifying the at least one I-frame within the decoded digital video data; and
utilizing the at least one polygonal shape recognition algorithm on data of the identified at least one I-frame to identify at least one polygon therein; and
utilizing the identified at least one polygon to identify an attribute.

### Claim 4 (depends on 3)

The computerized method of claim 3, wherein the utilizing the identified at least one polygon to identify an attribute comprises using one or more geometric relationships associated with the polygon to search a database of known geometric relationships associated with one or more tangible items.

### Claim 5 (depends on 1)

The computerized method of claim 1, wherein the processing the plurality of digital content assets using the plurality of computer programs at least in part comprises pre-processing the plurality of digital content assets during a first prescribed period of time and prior to a second, later prescribed period of time during which the pre-processed plurality of digital content assets are made available to one or more computerized user devices, the computerized device associated with the user comprising one of the computerized user devices.

### Claim 6 (depends on 1)

The computerized method of claim 1, wherein the receiving data indicative of interest by the user in the at least one identified user-cognizable digital element within at least one of the plurality of identified digital content assets comprises receiving data generated by a JavaScript element operatively associated with an Internet website, the data generated by the JavaScript element received as part of an HTTPS session established between the computerized device associated with the user and at least one server apparatus.

### Claim 7 (depends on 1)

The computerized method of claim 1, wherein the receiving data indicative of interest by the user in the at least one identified user-cognizable element within at least one of the plurality of identified digital content assets comprises receiving data generated by a JavaScript element operatively associated with an Internet website, the data generated based at least in part on a user performing at least one of the following via an input device of the computerized device associated with the user: (i) causing, via the input device, a cursor to be located within a boundary of the at least one identified user-cognizable digital element that is rendered on a display device of the computerized device associated with the user; and/or (ii) using the input device to pause rendering of the at least one identified digital content asset by the computerized device associated with the user.

### Claim 8 (depends on 1)

The computerized method of claim 1, wherein the forming the association between (i) the at least portion of the stored generated characterizing data of the at least one of the identified user-cognizable digital elements in each of the identified digital content assets, and (ii) secondary digital content comprises:
receiving data indicative of an identity of a tangible item;
searching the generated characterizing data of the database based at least in part on the data indicative of the identity of the tangible item to identify one or more instances of the tangible item within the generated characterizing data; and
associating the secondary digital content to the identified one or more instances, the secondary digital content comprising digitally rendered content relating to the tangible item.

### Claim 9 (depends on 1)

The computerized method of claim 1, wherein:
the identifying at least one user-cognizable digital element within each of the plurality of identified digital content assets comprises:
decoding an audio stream of each of the plurality of identified digital content assets; and
processing the decoded audio stream of each of the plurality of identified digital content assets to identify at least one user-cognizable audio element within each; and
the characterizing of the at least one identified user-cognizable digital element in each of the plurality of identified digital content assets to generate characterizing data comprises characterizing the at least one identified user-cognizable audio element in each of the plurality of identified digital content assets based on a plurality of tones or sounds therein, the plurality of tones or sounds corresponding to at least one of a (i) trademark or service mark, and/or (ii) a trade dress element, associated with a particular good or service.

### Claim 10

A computerized system for media processing, analysis, and selective provision of secondary content, the computerized system comprising:
a computerized server apparatus;
a database in data communication with the computerized server apparatus; and
a customer portal environment operative to execute on one or more computerized customer devices associated with an operator of the system.

### Claim 11 (depends on 10)

The computerized system of claim 10, wherein the computerized server apparatus comprises an application programming interface (API) configured to enable each of the one or more computerized customer devices to, via a lightweight script operative to execute on a network site accessible to each of the one or more computerized customer devices, transparently access augmented reality (AR) information having secondary content associated with one or more identifiable elements within data frames or containers of media assets accessed on the network site and renderable by the one or more computerized customer devices.

### Claim 12 (depends on 11)

The computerized system of claim 11, wherein the server apparatus comprises a plurality of servers configured to operate in logical and data communication with one another so as to enable (i) substantially real time (near-real time) processing of one or more user-selected media assets to identify and characterize the identifiable elements, the one or more user-selected media assets selected by at least one of the one or more computerized customer devices, and (ii) dynamic return of related secondary content to the at least one of the one or more computerized customer devices while the rendering of the one or more user-selected media asset occurs.

### Claim 13

A computerized apparatus configured for automatic provision of secondary digital content relating to a user-cognizable digital element present in a digital content asset, the computerized apparatus comprising:
data interface apparatus configured for data communication with a plurality of computerized user devices and a database;
data processor apparatus in data communication with the data interface apparatus; and
data storage apparatus in data communication with the data processor apparatus, the data storage apparatus having at least one computer program stored thereon, the at least one computer program comprising a plurality of instructions, the plurality of instructions configured to, when executed on the data processor apparatus, cause the computerized apparatus to:
identify a plurality of digital content assets; and
perform processing of the plurality of digital content assets, the processing comprising:
identification a plurality of user-cognizable digital elements, one or more of the plurality of user-cognizable digital elements being within each of the plurality of identified digital content assets;
generation of characterizing data related to individual ones of the plurality of user-cognizable digital elements;
generation of data indicative of a plurality of unique identifiers, each of the plurality of unique identifiers corresponding ton one of the plurality of identified digital content assets;
storage, in the database, of the generated characterizing data related to individual ones of the plurality of user-cognizable digital elements associated with corresponding ones of the plurality of unique identifiers; and
generation of data indicative of an association between (i) at least a portion of the stored characterizing data, and (ii) secondary digital content.

### Claim 14 (depends on 13)

The computerized apparatus of claim 13, wherein the plurality of instructions are further configured to, when executed on the data processor apparatus, cause the computerized apparatus to receive data indicative of interest by a user in at least one of the plurality of user-cognizable digital elements within at least one of the plurality of identified digital content assets, the data indicative of interest by the user comprising data indicative of a user action performed via an input device of at least one of the plurality of computerized user devices which is associated with the user, the data indicative of the user action comprising data indicative of one or more of (i) the user causing, via the input device, a cursor to be located within a boundary of the at least one identified user-cognizable digital element, or (ii) the user using the input device to pause rendering of the at least one identified digital content asset by the computerized device associated with the user.

### Claim 15 (depends on 14)

The computerized apparatus of claim 14, wherein the plurality of instructions are further configured to, when executed on the data processor apparatus, cause the computerized apparatus to, based at least on the received data indicative of interest by the user, cause provision of at least one element of the secondary digital content to the at least one of the plurality of computerized user devices which is associated with the user.
