# US 12,003,831 B2 · Automated content segmentation and identification of fungible content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,831 B2 - Document ID: US12003831B2 - Status: Granted - Kind: B2 - Application number: 17/366,675 - Filed: 2021-07-02 - Published: 2023-01-05 - Granted: 2024-06-04 - Earliest claimed: 2021-07-02 - Assignees: Disney Enterprises, Inc. - Applicants: Disney Enterprises, Inc. - Inventors: Miquel Angel Farre Guiu; Marc Junyent Martin; Pablo Pernias - Web page: https://patentgrep.com/patent/US12003831B2 ## CPC - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - G06N 20/00: Machine learning - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)} - H04N 21/4662: {characterized by learning algorithms} - H04N 21/8352: involving content or source identification data, e.g. Unique Material Identifier [UMID] - G06V 20/48: {Matching video sequences} - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - H04N 21/26603: {for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A content segmentation system includes a computing platform having processing hardware and a system memory storing a software code and a trained machine learning model. The processing hardware is configured to execute the software code to receive content, the content including multiple sections each having multiple content blocks in sequence, to select one of the sections for segmentation, and to identify, for each of the content blocks of the selected section, at least one respective representative unit of content. The software code is further executed to generate, using the at least one respective representative unit of content, a respective embedding vector for each of the content blocks of the selected section to provide a multiple embedding vectors, and to predict, using the trained machine learning model and the embedding vectors, subsections of the selected section, at least some of the subsections including more than one of the content blocks. ## Claims ### Claim 1 A system comprising: a computing platform including processing hardware and a system memory; a software code stored in the system memory; and a machine learning model trained to predict scene boundaries; the processing hardware configured to execute the software code to: receive content, the content including a plurality of acts each having a plurality of shots in sequence; select one of the plurality of acts for segmentation; identify, for each of the plurality of shots of the selected act, at least one respective representative unit of content, the at least one respective unit of content comprising at least one video frame, an audio sample, or a combination thereof; generate, using the at least one respective representative unit of content, a respective embedding vector for each of the plurality of shots of the selected act to provide a plurality of embedding vectors; and predict, using the machine learning model and the plurality of embedding vectors, respective scene boundaries of each of a plurality of scenes of the selected act, at least one of the plurality of scenes including more than one of the plurality of shots. ### Claim 2 (depends on 1) The system of claim 1, wherein the processing hardware is configured to execute the software code to use the machine learning model to predict the respective scene boundaries of each of the plurality of scenes of the selected act by comparing pairs of the plurality of embedding vectors corresponding respectively to consecutive pairs of the plurality of shots. ### Claim 3 (depends on 1) The system of claim 1, wherein each of the plurality of scenes of the selected act corresponds to a different respective location depicted by the content. ### Claim 4 (depends on 1) The system of claim 1, wherein the at least one respective representative unit of content consists of a single video frame or an audio spectrogram. ### Claim 5 (depends on 1) The system of claim 1, wherein the at least one respective representative unit of content comprises the audio sample, and the audio sample comprises an audio spectrogram. ### Claim 6 (depends on 1) The system of claim 1, wherein the processing hardware is further configured to execute the software code to: receive, from a user system, a manually segmented content sample prepared by a user; and adjust, based on the manually segmented content sample, one or more hyperparameters of the machine learning model to substantially replicate the manually segmented content sample. ### Claim 7 (depends on 1) The system of claim 1, wherein the respective embedding vector for each one of the plurality of shots is an audio-video embedding vector, and wherein a first audio-video embedding vector corresponding to a first shot of the plurality of shots includes a video embedding vector for the first shot combined with an audio embedding vector for the first shot. ### Claim 8 (depends on 7) The system of claim 7, wherein the first audio-video embedding vector further includes a time duration of the first shot. ### Claim 9 (depends on 1) The system of claim 1, wherein the processing hardware is further configured to execute the software code to: identify, using computer vision, the plurality of acts included in the content. ### Claim 10 (depends on 1) The system of claim 1, wherein the respective scene boundaries of each of the plurality of scenes of the selected act are predicted based on a similarity score between consecutive embedding vectors being less than or equal to a threshold value. ### Claim 11 A computer-readable non-transitory storage medium having stored thereon instructions, which when executed by a processing hardware of a system, instantiate a method comprising: receiving content, the content including a plurality of acts each having a plurality of shots in sequence; selecting one of the plurality of acts for segmentation; identifying, for each of the plurality of shots of the selected act, at least one respective representative unit of content, the at least one respective unit of content comprising at least one video frame, an audio sample, or a combination thereof; generating, using the at least one respective representative unit of content, a respective embedding vector for each of the plurality of shots of the selected act to provide a plurality of embedding vectors; and predicting, using a machine learning model and the plurality of embedding vectors, respective scene boundaries of each of a plurality of scenes of the selected act, at least one of the plurality of scenes including more than one of the plurality of shots. ### Claim 12 (depends on 11) The computer-readable non-transitory storage medium of claim 11, wherein predicting the respective scene boundaries of each of the plurality of scenes of the selected act comprises comparing pairs of the plurality of embedding vectors corresponding respectively to consecutive pairs of the plurality of shots. ### Claim 13 (depends on 11) The computer-readable non-transitory storage medium of claim 11, wherein each of the plurality of scenes of the selected act corresponds to a different respective location depicted by the content. ### Claim 14 (depends on 11) The computer-readable non-transitory storage medium of claim 11, wherein the at least one respective representative unit of content consists of a single video frame or an audio spectrogram. ### Claim 15 (depends on 11) The computer-readable non-transitory storage medium of claim 11, wherein the at least one respective representative unit of content comprises the audio sample, and the audio sample comprises an audio spectrogram. ### Claim 16 (depends on 11) The computer-readable non-transitory storage medium of claim 11, having stored thereon additional instructions, which when executed by the processing hardware of the system, instantiate the method further comprising: receiving, from a user system, a manually segmented content sample prepared by a user; and adjusting, based on the manually segmented content sample, one or more hyperparameters of the machine learning model to substantially replicate the manually segmented content sample. ### Claim 17 A method for use by a system including a computing platform having processing hardware, a system memory storing a software code, and a machine learning model trained to predict scene boundaries, the method comprising: receiving content, by the software code executed by the processing hardware, the content including a plurality of acts each having a plurality of shots in sequence; selecting, by the software code executed by the processing hardware, one of the plurality of acts for segmentation; identifying, by the software code executed by the processing hardware for each of the plurality of shots of the selected act, at least one respective representative unit of content, the at least one respective unit of content comprising at least one video frame, an audio sample, or a combination thereof; generating, by the software code executed by the processing hardware and using the at least one respective representative unit of content, a respective embedding vector for each of the plurality of shots of the selected act to provide a plurality of embedding vectors; and predicting, by the software code executed by the processing hardware using the machine learning model and the plurality of embedding vectors, respective scene boundaries of each of a plurality of scenes of the selected act, at least one of the plurality of scenes including more than one of the plurality of shots. ### Claim 18 (depends on 17) The method of claim 17, wherein predicting the respective scene boundaries of each of the plurality of scenes of the selected act comprises comparing pairs of the plurality of embedding vectors corresponding respectively to consecutive pairs of the plurality of shots. ### Claim 19 (depends on 17) The method of claim 17, wherein each of the plurality of scenes of the selected act corresponds to a different respective location depicted by the content. ### Claim 20 (depends on 17) The method of claim 17, further comprising: receiving from a user system, by the software code executed by the processing hardware, a manually segmented content sample prepared by a user; and adjusting, by the software code executed by the processing hardware based on the manually segmented content sample, one or more hyperparameters of the machine learning model to substantially replicate the manually segmented content sample.