# US 11,908,491 B2 · Selecting supplemental audio segments based on video analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,491 B2 - Document ID: US11908491B2 - Status: Granted - Kind: B2 - Application number: 17/804,547 - Filed: 2022-05-27 - Published: 2022-12-01 - Granted: 2024-02-20 - Earliest claimed: 2021-05-27 - Assignees: Xdmind Inc. - Applicants: Xdmind Inc. - Inventors: Linda Bernardi; Ara Bernardi - Web page: https://patentgrep.com/patent/US11908491B2 ## CPC - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - A63F 13/35: Details of game servers - A63F 13/54: involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/47: {Detecting features for summarising video content} - G11B 27/005: {Reproducing at a different information rate from the information rate of recording (for television signals H04N5/783)} - G11B 27/11: by using information not detectable on the record carrier - G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/2368: Multiplexing of audio and video streams - H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)} - H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window} - H04N 21/8113: {comprising music, e.g. song in MP3 format} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - H04N 21/816: {involving special video data, e.g 3D video} - A63F 13/61: using advertising information - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Aspects of the present application correspond to generation of supplemental content based on processing information associated with content to be rendered. More specifically, aspects of the present application correspond to the generation of audio track information, such as music tracks, that are created for playback during the presentation of video content. Illustratively, one or more frames of the video content are processed by machine learned algorithm(s) to generate processing results indicative of one or more attributes characterizing individual frames of video content. A selection system can then identify potential music track or other audio data in view of the processing results. ## Claims ### Claim 1 A method for generating supplemental content utilizing supplemental audio content generation application decoupled from one or more content generating application, the method comprising: obtaining, by the supplemental audio content generation application, one or more video frames generated by the one or more content generating applications for rendering on a client device, wherein the supplemental audio content generation application and the one or more content generation applications are independently executed on the client device and wherein the supplemental audio content generation application and the one or more content generating applications are decoupled and independently executable; processing, by the supplemental audio content generation application, content depicted in the obtained one or more video frames to generate an attribute vector characterizing content depicted in the one or more video frames without receiving additional information from the one or more content generating applications; processing, by the supplemental audio content generation application, the generated attribute vector to identify at least one audio segment based on providing the generated attributed vector as input to a machine-learned algorithm; and causing, by the supplemental audio content generation application, playback of the identified at least one audio segment in correlation with the independent rendering of the one or more video frames generated by the one or more content generation applications. ### Claim 2 (depends on 1) The method of claim 1, wherein one or more content generating applications include at least one social media application generating video frame data. ### Claim 3 (depends on 1) The method of claim 1, wherein one or more content generating applications include at least one video gaming application generating video frame data. ### Claim 4 (depends on 1) The method of claim 1, wherein obtaining, by the supplemental audio content generation application, one or more video frames generated by the one or more content generating applications includes obtaining individual frames of video data based on a capture frequency. ### Claim 5 (depends on 4) The method of claim 4, wherein the capture frequency is dynamically modified based on attributes of the video frame data. ### Claim 6 (depends on 1) The method of claim 1, wherein the attribute vector characterizing content depicted in the one or more video frames includes one or more attributes corresponding to a characterization of a theme of objects depicted in the one or more video frames. ### Claim 7 (depends on 1) The method of claim 1, wherein the attribute vector characterizing content depicted in the one or more video frames includes one or more attributes corresponding to characterization of a measured progress of objects depicted in the one or more video frames. ### Claim 8 (depends on 1) The method of claim 1, wherein the attribute vector characterizing content depicted in the one or more video frames includes one or more attributes corresponding to the characterization of mood of objects depicted in the one or more video frames. ### Claim 9 (depends on 1) The method of claim 1, wherein the attribute vector characterizing content depicted in the one or more video frames includes one or more attributes corresponding to characterization of a pace of objects depicted in the one or more video frames. ### Claim 10 (depends on 1) The method of claim 1, wherein the attribute vector characterizing content depicted in the one or more video frames includes one or more attributes corresponding to characterization of at least one of a style of play or intent of play in video gaming. ### Claim 11 (depends on 1) The method of claim 1 wherein processing, by the supplemental audio content generation application, the generated attribute vector to identify at least one audio segment based on a machine-learned algorithm includes identifying a segment of an audio track correlated to the generated attribute vector. ### Claim 12 (depends on 1) The method of claim 1, wherein processing, by the supplemental audio content generation application, the generated attribute vector to identify at least one audio segment based on a machine-learned algorithm includes adapting at least one target segment based on the generated attribute vector. ### Claim 13 (depends on 1) The method of claim 1, wherein adapting at least one target segment based on the generated attribute vector includes adapting a speed of play. ### Claim 14 (depends on 1) The method of claim 1, wherein adapting at least one target segment based on the generated attribute vector includes adapting fade information. ### Claim 15 (depends on 1) The method of claim 1, wherein adapting at least one target segment based on the generated attribute vector includes combining two or more audio segments. ### Claim 16 (depends on 1) The method of claim 1, wherein adapting at least one target segment based on the generated attribute vector includes generating new audio tracks based on the generated attribute vector and preference information. ### Claim 17 A system to supplemental audio content comprising: one or more computing processors and memories associated with a supplemental audio content generation application, wherein the supplemental audio content generation application and is executed on a client device and wherein the supplemental audio content generation application, and one or more content generating applications are independently executable, wherein the supplemental audio content generation application is configured to: obtain one or more video frames generated by the one or more content generating applications for rendering on a client device, wherein the supplemental audio content generation application and the one or more content generation applications are independently executed on a client device and wherein the supplemental audio content generation application and the one or more content generating applications are decoupled and independently executable; process content depicted in the obtained one or more video frames to generate an attribute vector characterizing content depicted in the one or more video frames without receiving additional information from the one or more content generating applications; process the generated attribute vector to identify at least one audio segment based on providing the generated attributed vector as input to a machine-learned algorithm, wherein the identified at least one audio segment is adapted based, at least in part, on the generated attribute vector; and cause an independent playback of the identified at least one audio segment in correlation with rendering of the one or more content generation applications. ### Claim 18 (depends on 17) The system of claim 17, wherein the supplemental audio content generation application obtains individual frames of video data based on a capture frequency. ### Claim 19 (depends on 17) The system of claim 17, wherein the attribute vector characterizing content depicted in the one or more video frames includes one or more attributes corresponding to a characterization of at least one of a theme of objects depicted in the one or more video frames; a measured progress of objects depicted in the one or more video frames; mood of objects depicted in the video frame data; pace of objects depicted in the one or more video frames; or a style of play or intent of play in video gaming. ### Claim 20 (depends on 17) The system of claim 17, wherein the supplemental content application adapts at least one target segment based on the generated attribute vector based on at least one of adapting a speed of play; adapting fade information; combining two or more audio segments; or generating new audio tracks based on the generated attribute vector and preference information. ### Claim 21 A method for generating supplemental content utilizing supplemental audio content generation application decoupled from one or more content generating application, the method comprising: obtaining, by the supplemental audio content generation application, an individual video frame generated by the one or more content generating applications, wherein the supplemental audio content generation application and the one or more content generation applications are independently executed on a client device and wherein the supplemental audio content generation application and the one or more content generating applications are decoupled and independently executable; processing, by the supplemental audio content generation application, content depicted in the obtained one or more video frames to generate an attribute vector characterizing content depicted in the one or more video frames without receiving additional information from the one or more content generating applications; processing, by the supplemental audio content generation application, the generated attribute vector to identify at least one audio segment based on a machine-learned algorithm, wherein the identified at least one audio segment is adapted based, at least in part, on the generated attribute vector; and causing, by the supplemental audio content generation application, playback of the identified at least one audio segment. ### Claim 22 (depends on 21) The method of claim 21, wherein obtaining, by the supplemental audio content generation application, one or more video frames generated by the one or more content generating application includes obtaining individual frames of video data based on a capture frequency. ### Claim 23 (depends on 21) The method of claim 21, wherein the attribute vector characterizing content depicted in the one or more video frames includes one or more attributes corresponding to a characterization of at least one of a theme of objects depicted in the individual video frame; a measured progress of objects depicted in the individual video frame; mood of objects depicted in the video frame data; pace of objects depicted in the individual video frame; or a style of play or intent of play in video gaming. ### Claim 24 (depends on 21) The method of claim 21, wherein the supplemental content application adapts at least one target segment based on the generated attribute vector based on at least one of adapting a speed of play; adapting fade information; combining two or more audio segments; or generating new audio tracks based on the generated attribute vector and preference information.