# US 11,115,722 B2 · Crowdsourcing supplemental content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,722 B2 - Document ID: US11115722B2 - Status: Granted - Kind: B2 - Application number: 13/671,626 - Filed: 2012-11-08 - Published: 2014-05-08 - Granted: 2021-09-07 - Earliest claimed: 2012-11-08 - Assignees: Comcast Cable Communications, LLC - Applicants: Comcast Cable Communications, LLC - Inventors: Derek Johnson - Web page: https://patentgrep.com/patent/US11115722B2 ## CPC - H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)} - G06F 16/40: of multimedia data, e.g. slideshows comprising image and additional audio data (retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70) - H04N 21/41265: {having a remote control device for bidirectional communication between the remote control device and client device (unidirectional remote control device H04N21/42204)} - H04N 21/47815: {Electronic shopping (payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12)} - H04N 21/4784: receiving rewards - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - H04N 21/8547: involving timestamps for synchronizing content - G06F 16/60: of audio data - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and systems for sourcing supplemental content are disclosed. Secondary devices may be used to identify content streaming on first screen devices and to generate supplemental data for the content. In this manner, users may be leveraged to create various data for a variety of content. The data may be collected and organized so that users watching content at a later time may have access to the data. Methods and systems for using second screen devices to access metadata created by the crowd are also disclosed. ## Claims ### Claim 1 A method comprising: receiving, by a computing system and from a user device: a sample of audio indicating a first time point associated with a portion of primary content that is being presented; user-provided data; and a request to associate the user-provided data with the first time point; determining, by the computing system and based on the sample of audio, a first image from the primary content and associated with the first time point; sending, by the computing system and to the user device, the first image and a request for confirmation that the user-provided data is to be associated with the first time point; receiving, by the computing system, from the user device, and after sending the request for confirmation, a request to associate the user-provided data with a modified time point, offset from and instead of the first time point; and generating, by the computing system and based on the request to associate the user-provided data with the modified time point, metadata that associates the user-provided data with the modified time point. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining, by the computing system, an audio clip from the primary content and associated with the first time point; sending, by the computing system and to the user device, the audio clip with the request for confirmation; receiving, by the computing system and from the user device, a user comment associated with the sample of audio; and associating the user comment with the user-provided data. ### Claim 3 (depends on 2) The method of claim 2, further comprising: comparing textual data of the user comment with a predetermined word; discarding a portion of the textual data that corresponds to the predetermined word; and causing one or more other user devices to display a remaining portion of the textual data and the user-provided data with a portion of the primary content associated with the modified time point. ### Claim 4 (depends on 3) The method of claim 3, further comprising ranking the user comment based on an identity of a user submitting the user-provided data. ### Claim 5 (depends on 1) The method of claim 1, further comprising: receiving, by the computing system and from the user device, genre information for the user-provided data, wherein the genre information is specified by a user of the user device; extracting, based on the genre information and based on the request to associate the user-provided data with the modified time point, user-provided data associated with the genre information and associated with the modified time point; and adding the user-provided data to the extracted user-provided data. ### Claim 6 (depends on 1) The method of claim 1, wherein the generating metadata comprises adding the user-provided data to a plurality of additional user-provided data items previously associated with the modified time point. ### Claim 7 (depends on 1) The method of claim 1, further comprising: associating a plurality of user-provided links with an object presented in the first image; storing the plurality of user-provided links associated with the object presented in the first image; and sending, by the computing system and to one or more user devices, the plurality of user-provided links along with the portion of the primary content. ### Claim 8 (depends on 1) The method of claim 1, further comprising: associating the user-provided data with an object presented in a second image associated with the modified time point; and causing the user device to display the user-provided data with the object presented in the second image. ### Claim 9 (depends on 8) The method of claim 8, further comprising ranking a data tag of the associated user-provided data based on feedback from a user associated with the user device or based on an indication that a person had a favorable reaction to the data tag. ### Claim 10 (depends on 1) The method of claim 1, further comprising: determining, by the computing system and based on one or more filter criteria, whether to reject the request to associate the user-provided data with the modified time point. ### Claim 11 (depends on 10) The method of claim 10, wherein the determination of whether to reject the request to associate the user-provided data with the modified time point further comprises determining whether the user-provided data is duplicative of one or more other user-provided data associated with the modified time point. ### Claim 12 A method comprising: detecting, by a user device, a sample of audio associated with video content that is being presented; receiving, by the user device, user-provided data and a request to associate the user-provided data with the video content; sending, from the user device and to a computing system: the sample of audio, wherein the sample of audio indicates a first time point associated with a portion of the video content; the user-provided data; and a request to associate the user-provided data with the first time point; receiving, by the user device, from the computing system, and after sending the request to associate the user-provided data with the first time point, an image from the video content; causing, by the user device, display of: the image; and a request to confirm that the image correctly indicates a time point, in the video content, that is to be associated with the user-provided data; receiving, by the user device and after causing the display of the request to confirm, a user input indicating that the user-provided data is to be associated with a modified time point, offset from and instead of the first time point; and sending, by the user device, to the computing system, and based on the user input, a request to associate the user-provided data with the modified time point. ### Claim 13 (depends on 12) The method of claim 12, further comprising receiving, from a user of the user device and before sending the sample of audio, a user indication that the sample of audio is to be associated with the user-provided data. ### Claim 14 (depends on 12) The method of claim 12, further comprising generating combined data comprising the sample of audio and the user-provided data. ### Claim 15 (depends on 12) The method of claim 12, further comprising: receiving, by the user device, from the computing system, and after sending the request to associate the user-provided data with the first time point, an audio clip; and causing output of the audio clip with the display of the request to confirm that the image correctly indicates the time point. ### Claim 16 (depends on 12) The method of claim 12, further comprising: receiving, by the user device, genre information for the user-provided data, wherein the genre information is specified by a user of the user device; and sending, to the computing system, the genre information with the request to associate the user-provided data with the first time point. ### Claim 17 A method comprising: receiving, by a computing system and from a user device: a sample of audio indicating a first time point associated with a portion of media content that is being presented; user-provided data; and a request to associate the user-provided data with the first time point; determining, by the computing system and based on the sample of audio, an audio clip from the media content and associated with the first time point; sending, by the computing system and to the user device, the audio clip and a request for confirmation that the user-provided data is to be associated with the first time point; receiving, by the computing system, from the user device, and after sending the request for confirmation, a request to associate the user-provided data with a modified time point, offset from and instead of the first time point; and generating, by the computing system and based on the request to associate the user-provided data with the modified time point, metadata that associates the user-provided data with the modified time point. ### Claim 18 (depends on 17) The method of claim 17, further comprising: determining, by the computing system and based on the first time point, a video frame from the media content and associated with the first time point; and sending, by the computing system and to the user device, the video frame with the request for confirmation. ### Claim 19 (depends on 17) The method of claim 17, further comprising: receiving, by the computing system and from the user device, genre information for the user-provided data, wherein the genre information is specified by a user of the user device, wherein the generating metadata comprises generating, based on the genre information, metadata that associates the user-provided data with one or more data tags, wherein the one or more data tags are associated with the modified time point and with the genre information. ### Claim 20 (depends on 17) The method of claim 17, wherein the generating metadata comprises adding the user-provided data to a plurality of additional user-provided data items previously associated with the modified time point. ### Claim 21 (depends on 17) The method of claim 17, wherein the generating metadata comprises: editing, by the computing system, first metadata associated with a plurality of additional user-provided data by adding a hyperlink to the user-provided data to the first metadata, wherein the plurality of additional user-provided data are previously associated with the modified time point. ### Claim 22 (depends on 17) The method of claim 17, wherein the generating metadata comprises: assigning, by the computing system and to the user-provided data, a time value corresponding to the modified time point.