# US 12,014,752 B2 · Fully automated post-production editing for movies, tv shows and multimedia contents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,014,752 B2 - Document ID: US12014752B2 - Status: Granted - Kind: B2 - Application number: 17/728,577 - Filed: 2022-04-25 - Published: 2022-08-11 - Granted: 2024-06-18 - Earliest claimed: 2020-05-08 - Assignees: WeMovie Technologies - Applicants: WeMovie Technologies - Inventors: Xidong Wu; Xiubo Wu - Web page: https://patentgrep.com/patent/US12014752B2 ## CPC - G11B 27/034: on discs (G11B27/036, G11B27/038 take precedence) - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} - H04N 21/43072: {of multiple content streams on the same device} - H04N 21/8113: {comprising music, e.g. song in MP3 format} - H04N 21/8146: {involving graphical data, e.g. 3D object, 2D graphics} - H04N 21/854: Content authoring - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Methods, apparatus and systems related to post-production stage of a multimedia content are described. In one example aspect, a method for performing automated post-production editing includes receiving one or more machine-readable scripts corresponding to one or more scenes of a storyline. The one or more machine-readable scripts include information about multimodal data and editing instructions for each of the one or more scenes. The method also includes identifying, based on the multimodal data for each of the one or more scenes, at least one change in an audio or video feature in the multiple streams of multimedia content corresponding to the one or more scenes, editing the multiple streams of multimedia content based on the editing instructions and selectively based on the identified change, and generating a final stream of multimedia content based on the edited multiple streams. ## Claims ### Claim 1 A post-production editing device for automatically editing a digital multimedia content, comprising at least one processor that is configured to: receive multiple streams of multimedia content data corresponding to one or more scenes based on a storyline for a digital multimedia content; receive one or more machine-readable scripts that comprise information about the one or more scenes based on the storyline, wherein the information comprises editing instructions for an individual scene of the one or more scenes, and wherein the one or more machine-readable scripts further comprise information indicating a location of preloaded reference information related to a set of predefined features; detect an audio or video feature in the multiple streams of the multimedia content data based on one or more machine learning techniques; identify a change of the audio or video feature by comparing the detected audio or video feature with the set of predefined features indicated in the one or more machine-readable scripts, wherein the comparing comprises feature matching between the audio or video feature with the preloaded reference information, wherein the preloaded reference information comprises at least an image of a facial expression of an actor or an actress or an audio sample; and selectively edit the multiple streams of the multimedia content data based on the identified change. ### Claim 2 (depends on 1) The post-production editing device of claim 1, wherein the change of the audio or video feature comprises at least one of: a change in an action performed by the actor or the actress, or a change in an emotion of the actor or the actress. ### Claim 3 (depends on 1) The post-production editing device of claim 1, wherein the at least one processor is configured to receive the multiple streams of the multimedia content data from an external storage device. ### Claim 4 (depends on 1) The post-production editing device of claim 1, wherein the multiple streams of the multimedia content data comprise at least an audio stream and a video stream of the digital multimedia content. ### Claim 5 (depends on 1) The post-production editing device of claim 1, wherein the preloaded reference information comprises multiple images of a face of the actor or the actress captured at different angles. ### Claim 6 (depends on 1) The post-production editing device of claim 1, wherein the preloaded reference information comprises an image of an equipment used during a production of the digital multimedia content. ### Claim 7 A system for producing a multimedia digital content, comprising: a production subsystem configured to: receive one or more machine-readable scripts, wherein the one or more machine-readable scripts comprise information about one or more scenes in a storyline, wherein the information comprises editing instructions for an individual scene of the one or more scenes; produce multiple streams of the multimedia digital content using a set of production devices based on the one or more machine-readable scripts; and a post-production editing subsystem that is synchronized with the set of production devices in the production subsystem, wherein the post-production editing subsystem is configured to: receive the multiple streams of the multimedia digital content and the one or more machine-readable scripts from the production subsystem, detect an audio or video feature in the multiple streams of the multimedia digital content based on one or more machine learning techniques, identify a change of the audio or video feature by comparing the detected audio or video feature with a set of predefined features indicated in the one or more machine-readable scripts, wherein the comparing comprises feature matching between the audio or video feature with preloaded reference information, wherein the preloaded reference information comprises at least an image of a facial expression of an actor or an actress or an audio sample; and selectively edit the multiple streams of the multimedia digital content based on the identified change, wherein the post-production editing subsystem is further configured to: determine, based on the one or more machine-readable scripts, whether the identified change of the audio or video feature is a candidate for one or more editing operations for the multiple streams of the multimedia digital content. ### Claim 8 (depends on 7) The system of claim 7, wherein the multiple streams of the multimedia digital content comprise at least one of: a soundtrack, an image, an embedded commercial content, or a special effect media content. ### Claim 9 (depends on 7) The system of claim 7, wherein the information in the one or more machine-readable scripts further comprises actions for the set of production devices to take for producing the multiple streams, the system further comprising: a device management subsystem in communication with the production subsystem and configured to: control the set of production devices based on the actions in the one or more machine-readable scripts, and provide real-time information about production of the multiple streams to enable the production subsystem to update the one or more machine-readable scripts in real-time during the production according to the real-time information. ### Claim 10 (depends on 7) The system of claim 7, wherein the change of the audio or video feature comprises at least one of: a change in an action performed by the actor or the actress, or a change in an emotion of the actor or the actress. ### Claim 11 (depends on 7) The system of claim 7, wherein the preloaded reference information comprises multiple images of a face of the actor or the actress captured at different angles. ### Claim 12 (depends on 7) The system of claim 7, wherein the preloaded reference information comprises an image of an equipment used during the producing of the digital multimedia content. ### Claim 13 (depends on 7) The system of claim 7, wherein the one or more machine-readable scripts comprise information indicating a location of the preloaded reference information. ### Claim 14 A method for operating a post-production editing service, comprising: receiving, by the post-production editing service, one or more machine-readable scripts that comprise information about one or more scenes in a storyline, wherein the information comprises editing instructions for an individual scene of the one or more scenes, and wherein the one or more machine-readable scripts further comprise information indicating a location of preloaded reference information related to a set of predefined features; receiving, by the post-production editing service, multimedia content data corresponding to the one or more scenes in the storyline from one or more production devices; detecting, by the post-production editing service, an audio or video feature in the multimedia content data based on one or more machine learning techniques; identifying a change of the audio or video feature by comparing the detected audio or video feature with the set of predefined features indicated in the one or more machine-readable scripts, wherein the comparing comprises feature matching between the audio or video feature with the preloaded reference information that comprises at least an image of a facial expression of an actor or an actress or an audio sample; and selectively editing the multimedia content data based on the identified change. ### Claim 15 (depends on 14) The method of claim 14, comprising synchronizing with the one or more production devices. ### Claim 16 (depends on 14) The method of claim 14, comprising: receiving, by the post-production editing service, production information from the one or more production devices. ### Claim 17 (depends on 14) The method of claim 14, comprising: generating a final stream of the multimedia content data based on the selective editing. ### Claim 18 (depends on 14) The method of claim 14, wherein the post-production editing service is deployed in a distributed manner across multiple sites that are interconnected by a communication network. ### Claim 19 (depends on 14) The method of claim 14, comprising: embedding a commercial content into the multimedia content based on the one or more machine-readable scripts. ### Claim 20 (depends on 14) The method of claim 14, wherein the change of the audio or video feature comprises at least one of: a change in an action performed by the actor or the actress, or a change in an emotion of the actor or the actress.