# US 12,192,598 B2 · Method of processing audio or video data, device, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,598 B2 - Document ID: US12192598B2 - Status: Granted - Kind: B2 - Application number: 17/646,122 - Filed: 2021-12-27 - Published: 2022-04-21 - Granted: 2025-01-07 - Earliest claimed: 2021-02-19 - Assignees: Beijing Baidu Netcom Science Technology Co., Ltd. - Applicants: Beijing Baidu Netcom Science Technology Co., Ltd. - Inventors: Mingyue Zhang - Web page: https://patentgrep.com/patent/US12192598B2 ## CPC - H04N 21/8547: involving timestamps for synchronizing content - G06F 16/313: {Selection or weighting of terms for indexing} - G06F 16/345: {Summarisation for human users} - G06F 16/41: Indexing; Data structures therefor; Storage structures - G06F 16/483: using metadata automatically derived from the content - G06F 16/61: Indexing; Data structures therefor; Storage structures - G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)} - G06F 16/71: Indexing; Data structures therefor; Storage structures - G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data} - G06F 40/137: Hierarchical processing, e.g. outlines - G06F 40/258: Heading extraction; Automatic titling; Numbering - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - H04N 21/440236: {by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text} - H04N 21/8549: Creating video summaries, e.g. movie trailer {(retrieval in video databases by using presentations in form of a video summary G06F16/739)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method of processing audio or video data is provided, which relates to a field of a natural language processing technology, and in particular to a semantic understanding of a natural language. The method includes: extracting a text information from the audio or video data; generating a text outline and a plurality of time periods according to the text information, the text outline includes multi-level outline entries, and the plurality of time periods are associated with the multi-level outline entries; generating a display field for the audio or video data according to the text outline and the plurality of time periods; adding the display field to the audio or video data, so as to obtain updated audio or video data. A device, and a storage medium are further provided. ## Claims ### Claim 1 A method of processing audio or video data, comprising: extracting a text information from the audio or video data; generating a text outline and a plurality of time periods according to the text information, wherein the text outline comprises multi-level outline entries used as titles for text paragraphs, and the plurality of time periods are associated with the multi-level outline entries; generating a display field for the audio or video data according to the text outline and the plurality of time periods, wherein the display field stores an association relationship between the outline entries in the text outline and the plurality of time periods; and adding the display field to the audio or video data, so as to obtain updated audio or video data, wherein the generating a text outline and a plurality of time periods according to the text information comprises: dividing the text information into a plurality of first text paragraphs; generating a first text paragraph title for each first text paragraph of the plurality of first text paragraphs as a first level outline entry; constructing a text paragraph set based on the plurality of first text paragraphs; processing the each first text paragraph in the text paragraph set, and in response to a data amount of the each first text paragraph being greater than or equal to a preset threshold, dividing the each first text paragraph into a plurality of second text paragraphs; and generating an outline entry in a next level for each second text paragraph of the plurality of second text paragraphs, until the each first text paragraph and the each second text paragraph in the text paragraph set has a data amount less than the preset threshold, wherein the preset threshold is determined according to a depth or a granularity of the text outline. ### Claim 2 (depends on 1) The method of claim 1, wherein the processing the each first text paragraph in the text paragraph set, and in response to a data amount of the each first text paragraph being greater than or equal to a preset threshold, dividing the each first text paragraph into a plurality of second text paragraphs comprises: comparing the data amount of the each first text paragraph with the preset threshold; and in response to the data amount of the each first text paragraph being greater than or equal to the preset threshold: dividing the each first text paragraph into the plurality of second text paragraphs; wherein the generating an outline entry in a next level for each second text paragraph of the plurality of second text paragraphs, until the each first text paragraph and the each second text paragraph in the text paragraph set has a data amount less than the preset threshold comprises: generating a second text paragraph title for each second text paragraph in the plurality of second text paragraphs, and determining the second text paragraph title as a secondary level outline entry of the first text paragraph title generated for the each first text paragraph; and adding the plurality of second text paragraphs as first text paragraphs to the text paragraph set. ### Claim 3 (depends on 2) The method of claim 2, wherein the dividing the each first text paragraph into a plurality of second text paragraphs comprises: extracting a summary of the each first text paragraph by using a first deep learning model, so as to obtain a text paragraph summary; and dividing the text paragraph summary by using a second deep learning model, so as to obtain the plurality of second text paragraphs. ### Claim 4 (depends on 2) The method of claim 2, wherein the generating a text outline and a plurality of time periods according to the text information further comprises: marking a start time and an end time of each text statement contained in the text information according to the audio or video data; and determining the time period according to the start time and the end time of the each text statement contained in the text information. ### Claim 5 (depends on 4) The method of claim 4, wherein the determining the time period according to the start time and the end time of the each text statement contained in the text information comprises: in response to the first text paragraph title being generated for the each first text paragraph of the plurality of first text paragraphs as the first level outline entry, determining a start time of a first text statement in the each first text paragraph as a start time of a time period associated with the first level outline entry; and determining an end time of a last text statement in the each first text paragraph as an end time of the time period associated with the first level outline entry. ### Claim 6 (depends on 5) The method of claim 5, wherein the determining the time period according to the start time and the end time of the each text statement contained in the text information further comprises: in response to the second text paragraph title being generated for the each second text paragraph of the plurality of second text paragraphs and the second text paragraph title being determined as the secondary level outline entry of the first text paragraph title generated for the each first text paragraph, determining a start time of a first text statement in the each second text paragraph as a start time of a time period associated with the secondary level outline entry; and determining an end time of a last text statement in the each second text paragraph as an end time of the time period associated with the secondary level outline entry. ### Claim 7 (depends on 5) The method of claim 5, wherein the generating a display field for the audio or video data according to the text outline and the plurality of time periods comprises: combining each outline entry of the plurality of outline entries in the text outline and an associated time period in the plurality of time periods into a data pair; and storing the data pair in a string-type variable or a set-type variable as the display field, according to a hierarchical relationship between the multi-level outline entries. ### Claim 8 (depends on 6) The method of claim 6, wherein the generating a display field for the audio or video data according to the text outline and the plurality of time periods comprises: combining each outline entry of the plurality of outline entries in the text outline and an associated time period in the plurality of time periods into a data pair; and storing the data pair in a string-type variable or a set-type variable as the display field, according to a hierarchical relationship between the multi-level outline entries. ### Claim 9 A method of processing audio or video data, comprising: acquiring updated audio or video data; and extracting a display field from the updated audio or video data, and creating a display control according to the display field, wherein the updated audio or video data is generated according to operations of processing audio or video data, comprising: extracting a text information from the audio or video data; generating a text outline and a plurality of time periods according to the text information, wherein the text outline comprises multi-level outline entries used as titles for text paragraphs, and the plurality of time periods are associated with the multi-level outline entries; generating a display field for the audio or video data according to the text outline and the plurality of time periods, wherein the display field stores an association relationship between the outline entries in the text outline and the plurality of time periods; and adding the display field to the audio or video data, so as to obtain updated audio or video data, wherein the generating a text outline and a plurality of time periods according to the text information comprises: dividing the text information into a plurality of first text paragraphs; generating a first text paragraph title for each first text paragraph of the plurality of first text paragraphs as a first level outline entry; constructing a text paragraph set based on the plurality of first text paragraphs; processing the each first text paragraph in the text paragraph set, and in response to a data amount of the each first text paragraph being greater than or equal to a preset threshold, dividing the each first text paragraph into a plurality of second text paragraphs; and generating an outline entry in a next level for each second text paragraph of the plurality of second text paragraphs, until the each first text paragraph and the each second text paragraph in the text paragraph set has a data amount less than the preset threshold, wherein the preset threshold is determined according to a depth or a granularity of the text outline. ### Claim 10 (depends on 9) The method of claim 9, wherein the display control comprises a floating layer control and a plurality of sub-controls, and the creating a display control according to the display field comprises: creating the floating layer control in response to the display field being not empty; parsing the display field so as to obtain multi-level outline entries; and adding the sub-controls level by level to the floating layer control according to the multi-level outline entries. ### Claim 11 (depends on 10) The method of claim 10, further comprising: adding a click event for the display control, wherein the click event is configured to, in response to clicking a sub-control, acquire a time period associated with an outline entry displayed by the sub-control, and call a player to decode the audio or video data within the time period, so as to play the audio or video data within the time period. ### Claim 12 (depends on 9) The method of claim 9, wherein the processing the each first text paragraph in the text paragraph set, and in response to a data amount of the each first text paragraph being greater than or equal to a preset threshold, dividing the each first text paragraph into a plurality of second text paragraphs comprises: comparing the data amount of the each first text paragraph with the preset threshold; and in response to the data amount of the each first text paragraph being greater than or equal to the preset threshold: dividing the each first text paragraph into the plurality of second text paragraphs; wherein the generating an outline entry in a next level for each second text paragraph of the plurality of second text paragraphs, until the each first text paragraph and the each second text paragraph in the text paragraph set has a data amount less than the preset threshold comprises: generating a second text paragraph title for each second text paragraph in the plurality of second text paragraphs, and determining the second text paragraph title as a secondary level outline entry of the first text paragraph title generated for the each first text paragraph; and adding the plurality of second text paragraphs as first text paragraphs to the text paragraph set. ### Claim 13 (depends on 12) The method of claim 12, wherein the dividing the each first text paragraph into a plurality of second text paragraphs comprises: extracting a summary of the each first text paragraph by using a first deep learning model, so as to obtain a text paragraph summary; and dividing the text paragraph summary by using a second deep learning model, so as to obtain the plurality of second text paragraphs. ### Claim 14 (depends on 12) The method of claim 12, wherein the generating a text outline and a plurality of time periods according to the text information further comprises: marking a start time and an end time of each text statement contained in the text information according to the audio or video data; and determining the time period according to the start time and the end time of the each text statement contained in the text information. ### Claim 15 (depends on 14) The method of claim 14, wherein the determining the time period according to the start time and the end time of the each text statement contained in the text information comprises: in response to the first text paragraph title being generated for the each first text paragraph of the plurality of first text paragraphs as the first level outline entry, determining a start time of a first text statement in the each first text paragraph as a start time of a time period associated with the first level outline entry; and determining an end time of a last text statement in the each first text paragraph as an end time of the time period associated with the first level outline entry. ### Claim 16 (depends on 15) The method of claim 15, wherein the determining the time period according to the start time and the end time of the each text statement contained in the text information further comprises: in response to the second text paragraph title being generated for the each second text paragraph of the plurality of second text paragraphs and the second text paragraph title being determined as the secondary level outline entry of the first text paragraph title generated for the each first text paragraph, determining a start time of a first text statement in the each second text paragraph as a start time of a time period associated with the secondary level outline entry; and determining an end time of a last text statement in the each second text paragraph as an end time of the time period associated with the secondary level outline entry. ### Claim 17 (depends on 1) An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to implement the method of claim 1. ### Claim 18 (depends on 9) An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to implement the method of claim 9. ### Claim 19 (depends on 1) A non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions allow a computer to implement the method of claim 1. ### Claim 20 (depends on 9) A non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions allow a computer to implement the method of claim 9.