# US 12,003,821 B2 · Techniques for enhanced media experience > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,821 B2 - Document ID: US12003821B2 - Status: Granted - Kind: B2 - Application number: 16/853,451 - Filed: 2020-04-20 - Published: 2021-10-21 - Granted: 2024-06-04 - Earliest claimed: 2020-04-20 - Assignees: Disney Enterprises, Inc. - Applicants: Disney Enterprises, Inc. - Inventors: Douglas A. Fidaleo; Michael P. Goslin - Web page: https://patentgrep.com/patent/US12003821B2 ## CPC - H04N 21/4722: for requesting additional data associated with the content - H04N 21/478: Supplemental services, e.g. displaying phone caller identification, shopping application - H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window} - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - H04N 21/4666: {using neural networks, e.g. processing the feedback provided by the user} - H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme} - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present disclosure sets forth a technique for providing supplemental content along with source media. The technique includes receiving source media and content metadata associated with the source media, generating supplemental content based on the content metadata and preferences of one or more media consumers, and delivering the supplement content along with the source media to the one or more media consumers. ## Claims ### Claim 1 A computer-implemented method for providing supplemental content along with source media, the method comprising: receiving source media and content metadata associated with the source media; generating supplemental content based on a subset of the content metadata that is associated with a first portion of the source media and preferences of one or more media consumers, wherein the generated supplemental content does not include the first portion of the source media; delivering the generated supplemental content along with the first portion of the source media to the one or more media consumers; receiving video sensor data captured by one or more video sensors that indicates a first reaction of the one or more media consumers to at least one of the first portion of the source media or the supplemental content; determining at least one of a first emotional state of the one or more media consumers, a first sentiment state of the one or more media consumers, or a first level of engagement of the one or more media consumers based on the video sensor data indicating the first reaction; and generating additional supplemental content based on the at least one of the first emotional state, the first sentiment state, or the first level of engagement. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the source media comprises audio media or audio-visual media. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: modifying the preferences of the one or more media consumers based on the at least one of the first emotional state, the first sentiment state, or the first level of engagement; generating the additional supplemental content based on the content metadata and the modified preferences; and delivering the additional supplemental content to the one or more media consumers along with the source media. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the generated supplemental content comprises one or more audio elements, video elements, or holographic elements. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the generated supplemental content is customized based on an identity of at least one of the one or more media consumers. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the generated supplemental content comprises a virtual companion that consumes the source media along with the one or more media consumers. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the virtual companion reacts to one or more of the source media, another virtual companion, or the one or more media consumers. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, wherein the virtual companion interacts with one or more of the media consumers or another virtual companion. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein delivering the generated supplemental content along with the first portion of the source media comprises ducking or pausing the first portion of the source media when the generated supplemental content is delivered. ### Claim 10 One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to provide supplemental content along with source media by performing steps comprising: receiving source media and content metadata associated with the source media; generating supplemental content based on a subset of the content metadata that is associated with a first portion of the source media and preferences of one or more media consumers, wherein the generated supplemental content does not include the first portion of the source media; delivering the generated supplemental content along with the first portion of the source media to the one or more media consumers; receiving video sensor data captured by one or more video sensors that indicates a first reaction of the one or more media consumers to at least one of the first portion of the source media or the supplemental content; determining at least one of a first emotional state of the one or more media consumers, a first sentiment state of the one or more media consumers, or a first level of engagement of the one or more media consumers based on the video sensor data indicating the first reaction; and generating additional supplemental content based on the at least one of the first emotional state, the first sentiment state, or the first level of engagement. ### Claim 11 (depends on 10) The one or more non-transitory computer-readable storage media of claim 10, wherein the generated supplemental content comprises a virtual companion that consumes the first portion of the source media along with the one or more media consumers. ### Claim 12 (depends on 10) The one or more non-transitory computer-readable storage media of claim 10, wherein the content metadata includes at least one of global content metadata or time-aligned content metadata. ### Claim 13 (depends on 10) The one or more non-transitory computer-readable storage media of claim 10, wherein the preferences of the one or more media consumers include at least one of historical preferences or transient preferences of the one or more media consumers. ### Claim 14 (depends on 13) The one or more non-transitory computer-readable storage media of claim 13, wherein generating the supplemental content comprises giving greater weight to the transient preferences than to the historical preferences. ### Claim 15 (depends on 13) The one or more non-transitory computer-readable storage media of claim 13, further comprising determining the transient preferences based on one or more of a world state associated with a location where the source media is being consumed, local metadata determined from the one or more video sensors at the location, a sentiment of the one or more media consumers, an engagement of the one or more media consumers, or an emotion of the one or more media consumers. ### Claim 16 (depends on 10) The one or more non-transitory computer-readable storage media of claim 10, wherein the steps further comprise: analyzing the source media; and determining the content metadata based on the analyzing. ### Claim 17 (depends on 16) The one or more non-transitory computer-readable storage media of claim 16, wherein determining the content metadata is further based on at least one of metadata retrieved from one or more data sources or input from one or more test media consumers. ### Claim 18 A computing device, comprising: a memory; and one or more processors coupled to the memory; wherein the one or more processors are configured to: receive source media and content metadata associated with the source media; generate a virtual companion based on the content metadata and preferences of one or more media consumers; generate supplemental content to be delivered by the virtual companion, wherein the supplemental content is generated based on a subset of the content metadata that is associated with a first portion of the source media and does not include the first portion of the source media; have the virtual companion deliver the generated supplemental content to the one or media consumers along with the first portion of the source media; receive video sensor data captured by one or more video sensors that indicates a first reaction of the one or more media consumers to at least one of the first portion of the source media or the generated supplemental content; determine at least one of a first emotional state of the one or more media consumers, a first sentiment state of the one or more media consumers, or a first level of engagement of the one or more media consumers based on the video sensor data indicating the first reaction, wherein the first level of engagement comprises a value on an engagement spectrum; and generate additional supplemental content based on the at least one of the first emotional state, the first sentiment state, or the value on the engagement spectrum that is determined based on the video sensor data indicating the first reaction. ### Claim 19 (depends on 18) The computing device of claim 18, wherein to generate the supplemental content, the one or more processors are configured to select a first supplemental content element from labeled supplemental content or customize a second supplemental content element from the labeled supplemental content.