# US 10,482,168 B2 · Method and apparatus for annotating video content with metadata generated using speech recognition technology > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,168 B2 - Document ID: US10482168B2 - Status: Granted - Kind: B2 - Application number: 15/472,033 - Filed: 2017-03-28 - Published: 2017-07-13 - Granted: 2019-11-19 - Earliest claimed: 2007-05-11 - Assignees: Google Technology Holdings LLC - Applicants: Google Technology Holdings LLC - Inventors: Kevin M. McKoen; Michael A. Grossman - Web page: https://patentgrep.com/patent/US10482168B2 ## CPC - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F 16/70: of video data - G06F 16/78: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/7844: {using original textual content or text extracted from visual content or transcript of audio data} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method and apparatus is provided for annotating video content with metadata generated using speech recognition technology. The method begins by rendering video content on a display device. A segment of speech is received from a user such that the speech segment annotates a portion of the video content currently being rendered. The speech segment is converted to a text-segment and the text-segment is associated with the rendered portion of the video content. The text segment is stored in a selectively retrievable manner so that it is associated with the rendered portion of the video content. ## Claims ### Claim 1 A computer-implemented method comprising: while rendering video content on a display device: receiving, at a processor in communication with the display device, audio data from a user, the audio data annotating a particular portion of the video content; converting, by the processor, the audio data into text; and generating, by the processor, annotation metadata associating the text with the particular portion of the video content; after rendering the video content on the display device, receiving, at the processor, a search query comprising one or more terms present in the text; and providing, by the processor, the particular portion of the video content for output based on the search query and the annotation metadata associating the text with the particular portion of the video content. ### Claim 2 (depends on 1) The method of claim 1, further comprising, prior to receiving the audio data from the user while rendering the video content on the display device: superimposing, by the processor, a user interface over the video content rendered on the display device, the user interface comprising an annotation selectable control to enter an annotation mode; in response to receiving a selection indication indicating selection of the annotation selectable control, entering, by the processor, the annotation mode. ### Claim 3 (depends on 2) The method of claim 2, wherein the user interface further comprises an analyze selectable control to enter a mode to analyze previous annotations and an edit selectable control to enter a mode to edit previous annotations. ### Claim 4 (depends on 1) The method of claim 1, further comprising: while the particular portion of the video content is being output, and while an annotation mode is activated, providing, by the processor, for output, an interface that includes a request for information related to the particular portion of the video content; receiving, at the processor, the information related to the particular portion of the video content; and including, by the processor, the information related to the particular portion of the video content in the text. ### Claim 5 (depends on 1) The method of claim 1, further comprising: while the particular portion of the video content is being output, and while an annotation mode is activated, providing, by the processor, for output, a selectable control to activate a voice recognition mode; and in response to receiving data indicating a selection of the selectable control to enter the annotation mode, entering, by the processor, the voice recognition mode. ### Claim 6 (depends on 1) The method of claim 1, further comprising, after the annotation mode is deactivated, providing, by the processor, for output, the particular portion of the video content overlaid with the text. ### Claim 7 (depends on 6) The method of claim 6, wherein providing, for output, the particular portion of the video content overlaid with the text comprises: providing, for output, a selectable control to edit the text; and the method further comprises: receiving data indicating a selection of the selectable control to edit the text; and providing, for output, a user interface to edit the text. ### Claim 8 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: while rendering video content on a display device: receiving audio data from a user, the audio data annotating a particular portion of the video content; converting the audio data into text; and generating annotation metadata associating the text with the particular portion of the video content; after rendering the video content on the display device, receiving a search query comprising one or more terms present in the text; and providing the particular portion of the video content for output based on the search query and the annotation metadata associating the text with the particular portion of the video content. ### Claim 9 (depends on 8) The system of claim 8, wherein the operations further comprise, prior to receiving the audio data from the user while rendering the video content on the display device: superimposing a user interface over the video content rendered on the display device, the user interface comprising an annotation selectable control to enter an annotation mode; and in response to receiving a selection indication indicating selection of the annotation selectable control, entering the annotation mode. ### Claim 10 (depends on 9) The system of claim 9, wherein the user interface further comprises an analyze selectable control to enter a mode to analyze previous annotations and an edit selectable control to enter a mode to edit previous annotations. ### Claim 11 (depends on 8) The system of claim 8, wherein the operations further comprise: while the particular portion of the video content is being output, and while an annotation mode is activated, providing, for output, an interface that includes a request for information related to the particular portion of the video content; receiving the information related to the particular portion of the video content; and including the information related to the particular portion of the video content in the text. ### Claim 12 (depends on 8) The system of claim 8, wherein the operations further comprise: while the particular portion of the video content is being output, and while an annotation mode is activated, providing, for output, a selectable control to activate a voice recognition mode; and in response to receiving data indicating a selection of the selectable control to enter the annotation mode, entering the voice recognition mode. ### Claim 13 (depends on 8) The system of claim 8, wherein the operations further comprise, after the annotation mode is deactivated, providing, for output, the particular portion of the video content overlaid with the text. ### Claim 14 (depends on 13) The system of claim 13, wherein providing, for output, the particular portion of the video content overlaid with the text further comprises: providing, for output, a selectable control to edit the text; and wherein the operations further comprise: receiving data indicating a selection of the selectable control to edit the text; and providing, for output, a user interface to edit the text. ### Claim 15 A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: while rendering video content on a display device: receiving audio data from a user, the audio data annotating a particular portion of the video content; converting the audio data into text; and generating annotation metadata associating the text with the particular portion of the video content; after rendering the video content on the display device, receiving a search query comprising one or more terms present in the text; and providing the particular portion of the video content for output based on the search query and the annotation metadata associating the text with the particular portion of the video content. ### Claim 16 (depends on 15) The medium of claim 15, wherein the operations further comprise, prior to receiving the audio data from the user while rendering the video content on the display device: superimposing a user interface over the video content rendered on the display device, the user interface comprising an annotation selectable control to enter an annotation mode; and in response to receiving a selection indication indicating selection of the annotation selectable control, entering the annotation mode. ### Claim 17 (depends on 16) The medium of claim 16, wherein the user interface further comprises an analyze selectable control to enter a mode to analyze previous annotations and an edit selectable control to enter a mode to edit previous annotations. ### Claim 18 (depends on 15) The medium of claim 15, wherein the operations further comprise: while the particular portion of the video content is being output, and while the annotation mode is activated, providing, for output, an interface that includes a request for information related to the particular portion of the video content; receiving the information related to the particular portion of the video content; and including the information related to the particular portion of the video content in the text. ### Claim 19 (depends on 15) The medium of claim 15, wherein the operations further comprise: while the particular portion of the video content is being output, and while the annotation mode is activated, providing, for output, a selectable control to activate a voice recognition mode; and in response to receiving data indicating a selection of the selectable control to enter the annotation mode, entering the voice recognition mode. ### Claim 20 (depends on 15) The system of claim 15, wherein the operations further comprise, after the annotation mode is deactivated, providing, for output, the particular portion of the video content overlaid with the text.