# US 11,122,341 B1 · Contextual event summary annotations for video streams > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,122,341 B1 - Document ID: US11122341B1 - Status: Granted - Kind: B1 - Application number: 17/102,701 - Filed: 2020-11-24 - Granted: 2021-09-14 - Earliest claimed: 2020-11-24 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Clement Decrop; Zachary A. Silverstein; Timothy Yep; Jeremy R. Fox - Web page: https://patentgrep.com/patent/US11122341B1 ## CPC - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F 40/30: Semantic analysis - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/42203: {sound input device, e.g. microphone} - H04N 21/43074: {of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV programme} - H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window} - H04N 21/435: Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/4884: {for displaying subtitles} - H04N 21/6581: {Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application} - H04N 21/8405: represented by keywords - G06F 40/216: using statistical methods - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Generating event summary annotation overlays on a video stream is provided. In response to detecting an occurrence of a triggering event in the video stream, a context-appropriate summary annotation of the triggering event occurring in the video stream is generated based on analysis of content and tone of dialog in the video stream within a defined window of time surrounding the triggering event. The context-appropriate summary annotation of the triggering event is displayed in the video stream for a defined interval of time. ## Claims ### Claim 1 A computer-implemented method for generating event summary annotation overlays on a video stream, the computer-implemented method comprising: responsive to a computer detecting that the video stream is muted, performing, by the computer, natural language processing continuously on dialog in the video stream in real time to detect content and tone of the dialog in preparation for occurrence of a triggering event; responsive to the computer detecting an occurrence of a triggering event in the video stream, generating, by the computer, a context-appropriate summary annotation of the triggering event occurring in the video stream based on analysis of the content and the tone of the dialog in the video stream within a defined window of time surrounding the triggering event; and displaying, by the computer, the context-appropriate summary annotation of the triggering event in the video stream for a defined interval of time. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 further comprising: receiving, by the computer, a request for the video stream from a video stream display device of a user via a network; and responsive to the computer receiving the request, transmitting, by the computer, the video stream to the video stream display device of the user via the network. ### Claim 3 (depends on 2) The computer-implemented method of claim 2 further comprising: connecting, by the computer, to a microphone corresponding to the video stream display device of the user via the network; receiving, by the computer, sound waves from the microphone corresponding to the video stream display device of the user via the network; and comparing, by the computer, the sound waves received from the microphone corresponding to the video stream display device of the user to sound waves corresponding to the video stream. ### Claim 4 (depends on 3) The computer-implemented method of claim 3 further comprising: determining, by the computer, whether the sound waves received from the microphone match sound waves corresponding to the video stream within a predefined range based on the comparing; and responsive to the computer determining that the sound waves received from the microphone do not match the sound waves corresponding to the video stream within the predefined range based on the comparing, detecting, by the computer, that audio of the video stream is muted on the video stream display device of the user. ### Claim 5 (depends on 1) The computer-implemented method of claim 1 further comprising: retrieving, by the computer, a set of keywords and special characters that relates to the content and the tone of the dialog in the video stream within the defined window of time surrounding the triggering event from a triggering event keyword database; and displaying, by the computer, the set of keywords and special characters related to the content and the tone of the dialog corresponding to the trigger event for the defined interval of time as the context-appropriate summary annotation of the triggering event in an overlay on the video stream playing on a video stream display device of a user via a network. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein a profile corresponding to the user includes preferences that define how the computer displays the context-appropriate summary annotation in the overlay on the video stream, and wherein the preferences include one or more of where the overlay is displayed on the video stream, how long the overlay is displayed on the video stream, color of text corresponding to the context-appropriate summary annotation, background color of the overlay, and overlay animations for particular triggering events. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, wherein the keywords and the special characters are retrieved from closed captioning of the video stream. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the triggering event is one of a sudden increase in volume of the video stream occurring above a defined volume threshold level, a particular type of predefined sound occurring in the video stream, a particular type of item suddenly appearing in the video stream, particular types of predefined body movements occurring in the video stream, and one or more particular words spoken in the video stream. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the video stream is a live video stream in real time. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the defined window of time surrounding the triggering event comprises a first amount of time prior to the occurrence of the triggering event and a second amount of time after the occurrence of the triggering event. ### Claim 11 A computer system for generating event summary annotation overlays on a video stream, the computer system comprising: a bus system; a storage device connected to the bus system, wherein the storage device stores program instructions; and a processor connected to the bus system, wherein the processor executes the program instructions to: perform natural language processing continuously on dialog in the video stream in real time to detect content and tone of dialog in preparation for occurrence of a triggering event in response to detecting that the video stream is muted; generate a context-appropriate summary annotation of the triggering event occurring in a video stream based on analysis of the content and the tone of the dialog in the video stream within a defined window of time surrounding the triggering event in response to detecting an occurrence of the triggering event in the video stream; and display the context-appropriate summary annotation of the triggering event in the video stream for a defined interval of time. ### Claim 12 (depends on 11) The computer system of claim 11, wherein the processor further executes the program instructions to: receive a request for the video stream from a video stream display device of a user via a network; and transmit the video stream to the video stream display device of the user via the network in response to receiving the request. ### Claim 13 (depends on 12) The computer system of claim 12, wherein the processor further executes the program instructions to: connect to a microphone corresponding to the video stream display device of the user via the network; receive sound waves from the microphone corresponding to the video stream display device of the user via the network; and compare the sound waves received from the microphone corresponding to the video stream display device of the user to sound waves corresponding to the video stream. ### Claim 14 (depends on 13) The computer system of claim 13, wherein the processor further executes the program instructions to: determine whether the sound waves received from the microphone match sound waves corresponding to the video stream within a predefined range based on comparing; and detect that audio of the video stream is muted on the video stream display device of the user in response to determining that the sound waves received from the microphone do not match the sound waves corresponding to the video stream within the predefined range based on the comparing. ### Claim 15 (depends on 11) The computer system of claim 11, wherein the natural language processing is performed by analyzing keywords and special characters retrieved from closed captioning of the video stream. ### Claim 16 A computer program product for generating event summary annotation overlays on a video stream, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method of: responsive to the computer detecting that the video stream is muted, performing, by the computer, natural language processing continuously on dialog in the video stream in real time to detect content and tone of the dialog in preparation for occurrence of a triggering event; responsive to the computer detecting an occurrence of a triggering event in the video stream, generating, by the computer, a context-appropriate summary annotation of the triggering event occurring in the video stream based on analysis of the content and the tone of the dialog in the video stream within a defined window of time surrounding the triggering event; and displaying, by the computer, the context-appropriate summary annotation of the triggering event in the video stream for a defined interval of time. ### Claim 17 (depends on 16) The computer program product of claim 16 further comprising: receiving, by the computer, a request for the video stream from a video stream display device of a user via a network; and responsive to the computer receiving the request, transmitting, by the computer, the video stream to the video stream display device of the user via the network. ### Claim 18 (depends on 17) The computer program product of claim 17 further comprising: connecting, by the computer, to a microphone corresponding to the video stream display device of the user via the network; receiving, by the computer, sound waves from the microphone corresponding to the video stream display device of the user via the network; and comparing, by the computer, the sound waves received from the microphone corresponding to the video stream display device of the user to sound waves corresponding to the video stream. ### Claim 19 (depends on 18) The computer program product of claim 18 further comprising: determining, by the computer, whether the sound waves received from the microphone match sound waves corresponding to the video stream within a predefined range based on the comparing; and responsive to the computer determining that the sound waves received from the microphone do not match the sound waves corresponding to the video stream within the predefined range based on the comparing, detecting, by the computer, that audio of the video stream is muted on the video stream display device of the user. ### Claim 20 (depends on 16) The computer program product of claim 16, wherein the natural language processing is performed by analyzing keywords and special characters retrieved from closed captioning of the video stream.