# US 2023/0057566 A1 · Multimedia processing method and apparatus based on artificial intelligence, and electronic device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0057566 A1 - Document ID: US20230057566A1 - Status: Published application - Kind: A1 - Application number: 17/977,590 - Filed: 2022-10-31 - Published: 2023-02-23 - Earliest claimed: 2020-08-19 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Gui HUANG; Shuai TANG - Web page: https://patentgrep.com/patent/US20230057566A1 ## CPC - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract A multimedia processing method includes: obtaining, in response to an editing operation for a multimedia file, a material editing mode and an interaction template corresponding to a type of a material in the multimedia file; identifying the material in the multimedia file according to the material editing mode, and applying the interaction template to the identified material; and displaying the multimedia file after the interaction template is applied. ## Claims ### Claim 1 A multimedia processing method, performed by an electronic device, the method comprising: obtaining, in response to an editing operation for a multimedia file, a material editing mode and an interaction template corresponding to a type of a material in the multimedia file; identifying the material in the multimedia file according to the material editing mode, and applying the interaction template to the identified material; and displaying the multimedia file after the interaction template is applied. ### Claim 2 (depends on 1) The multimedia processing method according to claim 1, wherein the interaction template is configured to form interaction effects that are based on the material, the interaction effects comprise a first effect and a second effect different from the first effect, and the first effect comprises at least one of an augmented reality effect or a virtual reality effect; the applying the interaction template to the identified material comprises: applying the first effect in the interaction template to the identified material when the identified material meets a material interaction condition corresponding to the material editing mode; and the multimedia processing method further comprises: obtaining, when the identified material does not meet the material interaction condition, a set position corresponding to the second effect in the interaction template, and applying the second effect at the set position in the multimedia file. ### Claim 3 (depends on 2) The multimedia processing method according to claim 2, wherein the material interaction condition comprises a first material interaction condition corresponding to a non-real-time scenario and a second material interaction condition corresponding to a real-time scenario; and the identifying the material in the multimedia file according to the material editing mode comprises: when an editing scenario is the non-real-time scenario, identifying the material in the multimedia file according to the material editing mode to obtain a matching degree between the material and the multimedia file, and determining that the identified material meets the first material interaction condition when the matching degree is greater than a matching degree threshold in the first material interaction condition; and when an editing scenario is the real-time scenario, compressing the multimedia file and identifying the material in the compressed multimedia file according to the material editing mode, to obtain a matching degree between the material and the compressed multimedia file, and determining that the identified material meets the second material interaction condition when the matching degree is greater than a matching degree threshold in the second material interaction condition, wherein the matching degree threshold in the first material interaction condition is greater than the matching degree threshold in the second material interaction condition. ### Claim 4 (depends on 1) The multimedia processing method according to claim 1, further comprising: performing one of the following processing when there are a plurality of material editing modes: identifying, for one of the material editing modes, a candidate material in the multimedia file according to the material editing mode, and using the one of the material editing modes as the material editing mode to be applied when the candidate material meets a material interaction condition corresponding to the material editing mode; or displaying a preview identification process of the plurality of material editing modes, and using, in response to a selection operation for the material editing modes, a selected material editing mode as the material editing mode, wherein the material editing mode to be applied is used for identifying the material in the multimedia file. ### Claim 5 (depends on 4) The multimedia processing method according to claim 4, wherein the identifying a candidate material in the multimedia file according to the material editing mode comprises: performing, when the multimedia file is a video, periodic frame extraction on the multimedia file to obtain candidate video frames; and identifying the candidate material in the candidate video frames according to the material editing mode; and the identifying the material in the multimedia file according to the material editing mode comprises: identifying the material in the candidate video frames of the multimedia file according to the material editing mode. ### Claim 6 (depends on 5) The multimedia processing method according to claim 5, wherein before the identifying the candidate material in the candidate video frames according to the material editing mode, the method further comprises: compressing the plurality of candidate video frames when storage space occupied by the plurality of candidate video frames is greater than a storage space threshold; and the identifying the candidate material in the candidate video frames according to the material editing mode comprises: identifying the candidate material in the compressed candidate video frames according to the material editing mode. ### Claim 7 (depends on 4) The multimedia processing method according to claim 4, wherein the displaying a preview identification process of the plurality of material editing modes comprises: performing at least one of the following processing for one of the material editing modes: displaying a preview identification process for the multimedia file according to the material editing mode; or displaying a preview identification process for a sample multimedia file according to the material editing mode. ### Claim 8 (depends on 1) The multimedia processing method according to claim 1, further comprising: performing one of the following processing when there are a plurality of interaction templates: using, when the identified material meets a material interaction condition corresponding to a candidate interaction template, the candidate interaction template corresponding to the met material interaction condition as the interaction template; or displaying a preview application process of the plurality of interaction templates, and using, in response to a selection operation for the interaction templates, a selected interaction template as the interaction template. ### Claim 9 (depends on 8) The multimedia processing method according to claim 8, wherein the displaying a preview application process of the plurality of interaction templates comprises: performing at least one of the following processing for one of the plurality of interaction templates: displaying a preview application process for the multimedia file according to the interaction template; or displaying a preview application process for a sample multimedia file according to the interaction template. ### Claim 10 (depends on 1) The multimedia processing method according to claim 1, further comprising: performing, when the multimedia file is a video, periodic frame extraction on the multimedia file to obtain candidate video frames; and performing material identification processing on the candidate video frames to obtain the type of the material in the multimedia file. ### Claim 11 (depends on 10) The multimedia processing method according to claim 10, wherein before the performing material identification processing on the candidate video frames, the method further comprises: compressing the plurality of candidate video frames when storage space occupied by the plurality of candidate video frames is greater than a storage space threshold; and the performing material identification processing on the candidate video frames to obtain the type of the material in the multimedia file comprises: respectively performing material identification processing on the plurality of compressed candidate video frames, to obtain a type of a material in each of the candidate video frames, and using a type of a material with a largest ratio in the plurality of candidate video frames as the type of the material used for obtaining the corresponding material editing mode, wherein the ratio comprises any one of an area ratio or a quantity ratio of the material. ### Claim 12 (depends on 10) The multimedia processing method according to claim 10, further comprising: obtaining a plurality of sample multimedia files; performing material identification processing on the sample multimedia files by using an artificial intelligence model, to obtain types of materials in the sample multimedia files; and updating a weight parameter of the artificial intelligence model according to differences between the types obtained through the material identification processing and actual types, wherein the updated artificial intelligence model is configured to perform material identification processing on the candidate video frames. ### Claim 13 (depends on 1) The multimedia processing method according to claim 1, wherein the obtaining a material editing mode and an interaction template corresponding to a type of a material in the multimedia file comprises: obtaining, in candidate material editing modes respectively corresponding to a plurality of types, the material editing mode corresponding to the type of the material in the multimedia file, and obtaining at least one interaction template corresponding to the material editing mode. ### Claim 14 (depends on 1) The multimedia processing method according to claim 1, wherein after the identifying the material in the multimedia file according to the material editing mode, the method further comprises: displaying a confirmation prompt corresponding to the identified material, wherein the confirmation prompt comprises at least one of the type of the identified material, location information of the identified material in the multimedia file, or a preview result obtained after the interaction template is applied to the identified material; and the applying the interaction template to the identified material comprises: applying the interaction template to the identified material in response to a confirmation operation for the confirmation prompt. ### Claim 15 (depends on 1) The multimedia processing method according to claim 1, wherein the interaction template comprises an interaction effect and a duration of the interaction effect; and the applying the interaction template to the identified material comprises: applying the interaction effect in the interaction template to the identified material and maintaining the applied interaction effect until the duration is reached. ### Claim 16 (depends on 1) The multimedia processing method according to claim 1, wherein after the identifying the material in the multimedia file according to the material editing mode, the method further comprises: cropping the multimedia file according to location information of the identified material in the multimedia file; and the applying the interaction template to the identified material comprises: applying the interaction template to the identified material in the cropped multimedia file. ### Claim 17 (depends on 16) The multimedia processing method according to claim 16, wherein the cropping the multimedia file according to location information of the identified material in the multimedia file comprises: cropping, when the location information is time location information, the multimedia file based on time according to a first timestamp and a last timestamp in the time location information; and cropping, when the location information is region location information, the multimedia file based on region according to the region location information. ### Claim 18 A multimedia processing apparatus, comprising: a memory, configured to store executable instructions; and a processor, configured, when executing the executable instructions stored in the memory, to implement: obtaining, in response to an editing operation for a multimedia file, a material editing mode and an interaction template corresponding to a type of a material in the multimedia file; identifying the material in the multimedia file according to the material editing mode, and applying the interaction template to the identified material; and displaying the multimedia file after the interaction template is applied. ### Claim 19 (depends on 18) The apparatus according to claim 18, wherein the interaction template is configured to form interaction effects that are based on the material, the interaction effects comprise a first effect and a second effect different from the first effect, and the first effect comprises at least one of an augmented reality effect or a virtual reality effect; the applying the interaction template to the identified material comprises: applying the first effect in the interaction template to the identified material when the identified material meets a material interaction condition corresponding to the material editing mode; and the processor is further configured to implement: obtaining, when the identified material does not meet the material interaction condition, a set position corresponding to the second effect in the interaction template, and applying the second effect at the set position in the multimedia file. ### Claim 20 A non-transitory computer-readable storage medium, storing executable instructions, the executable instructions, when executed by a processor, cause the processor to perform: obtaining, in response to an editing operation for a multimedia file, a material editing mode and an interaction template corresponding to a type of a material in the multimedia file; identifying the material in the multimedia file according to the material editing mode, and applying the interaction template to the identified material; and displaying the multimedia file after the interaction template is applied.