# US 2024/0205489 A1 · Digital assistant for providing graphical overlays of video events > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0205489 A1 - Document ID: US20240205489A1 - Status: Published application - Kind: A1 - Application number: 18/587,689 - Filed: 2024-02-26 - Published: 2024-06-20 - Earliest claimed: 2021-08-31 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Gavin K. DUFFY; Raymond M. MACHARIA; Jessica J. PECK; Robert M. SCHULMAN - Web page: https://patentgrep.com/patent/US20240205489A1 ## CPC - H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/1431: {using a single graphics controller} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/1822: {Parsing for meaning understanding} - G06V 20/20: in augmented reality scenes - G06V 20/42: {of sport video content} - G06V 20/63: {Scene text, e.g. street names} - G06V 40/18: Eye characteristics, e.g. of the iris - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An example process includes while displaying, on a display, a video event: receiving, by a digital assistant, a natural language speech input corresponding to a participant of the video event; in accordance with receiving the natural language speech input, identifying, by the digital assistant, based on context information associated with the video event, a first location of the participant; and in accordance with identifying the first location of the participant, augmenting, by the digital assistant, the display of the video event with a graphical overlay displayed at a first display location corresponding to the first location of the participant. ## Claims ### Claim 1 A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device having a display, cause the electronic device to: while displaying, on the display, a video event: receive, by a digital assistant operating on the electronic device, a natural language speech input corresponding to a participant of the video event; in accordance with receiving the natural language speech input, identify, by the digital assistant, based on context information associated with the video event, a first location of the participant; and in accordance with identifying the first location of the participant, augment, by the digital assistant, the display of the video event with a graphical overlay displayed at a first display location corresponding to the first location of the participant. ### Claim 2 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: detect a change from the first location of the participant to a second location of the participant; and in response to detecting the change, display, on the display, the graphical overlay at a second display location corresponding to the second location of the participant. ### Claim 3 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein: the video event is displayed via video pass-through depicting a second display of an external electronic device; and the display and the second display concurrently display the video event. ### Claim 4 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein: the natural language speech input includes a request to identify the participant; and the graphical overlay includes information corresponding to the participant. ### Claim 5 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein the information corresponding to the participant indicates at least one of: an identity of the participant; a predetermined highlight corresponding to the participant; and statistics corresponding to the participant. ### Claim 6 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: detect a user gesture input, wherein the context information includes the detected user gesture input, and wherein identifying the first location of the participant includes: determining that the user gesture input corresponds to the first location at a start time of the natural language speech input; and analyzing the display of the video event at the first location using the context information associated with the video event to identify the first participant. ### Claim 7 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: detect user gaze data, wherein the context information includes the detected user gaze data, and wherein identifying the first location of the participant includes: determining, based on the user gaze data, that a user gaze is directed to the first location; and analyzing the display of the video event at the first location using the context information associated with the video event to identify the first participant. ### Claim 8 (depends on 6) The non-transitory computer-readable storage medium of claim 6, wherein the context information further includes information corresponding to opposing parties of the video event. ### Claim 9 (depends on 7) The non-transitory computer-readable storage medium of claim 7, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: determine that the natural language speech input refers to the participant in the present tense, wherein determining that the user gaze is directed to the first location includes determining, in accordance with a determination that the natural language speech input refers to the participant in the present tense, that the user gaze is directed to the first location at a start time of the natural language speech input. ### Claim 10 (depends on 4) The non-transitory computer-readable storage medium of claim 4, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: determine that the natural language speech input refers to the participant in the past tense; and in accordance with a determination that the natural language speech input refers to the participant in the past tense: determine, based on previous user gaze data detected before a start time of the natural language speech input, that a previous user gaze was directed to a previously displayed location; and analyze the display of the video event at the previously displayed location using the context information associated with the video event to identify the participant, wherein: the first location of the participant is a currently displayed location of the participant; and identifying the currently displayed location of the participant is performed in accordance with identifying the participant. ### Claim 11 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein the context information associated with the video event includes at least one of: information corresponding to opposing parties of the video event; an audio stream of the video event; and an annotated event stream of the video event. ### Claim 12 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein: the natural language speech input specifies the participant and includes a request to locate the participant; and the graphical overlay indicates a location of the participant. ### Claim 13 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the natural language speech input includes a request to visually distinguish between a first party of the video event and an opposing second party of the video event, wherein the participant corresponds to the first party, and wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: in accordance with receiving the natural language speech input and a determination that the natural language speech input includes a request to visually distinguish between the first party and the opposing second party: identify, by the digital assistant, based on the context information, a second location of a second participant of the video event, the second participant corresponding to the opposing second party; and in accordance with identifying the second location of the second participant, augment, by the digital assistant, the display of the video event with a second graphical overlay displayed at a second display location corresponding to the second location of the second participant. ### Claim 14 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the graphical overlay and the second graphical overlay are concurrently displayed, and wherein the graphical overlay is displayed in a different manner than the second graphical overlay when the graphical overlay and the second graphical overlay are concurrently displayed. ### Claim 15 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein: the natural language speech input includes a request to analyze an occurrence of the video event; and the graphical overlay indicates a trajectory of the participant, the trajectory corresponding to the occurrence. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: in accordance with a determination that the natural language speech input includes a request to analyze the occurrence of the video event, display, on the display, a replay of the occurrence, wherein: the first location of the participant is a location in the replay of the occurrence; and the first display location corresponds to the replay of the occurrence. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein identifying the first location of the participant includes: identifying, based on the context information associated with the video event, the participant; and in accordance with identifying the participant, identifying, based on the context information associated with the video event, the first location. ### Claim 18 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to: in accordance with a determination that the natural language speech input includes a request to analyze the occurrence of the video event: identify, based on the context information associated with the video event, a second participant corresponding to the occurrence; in accordance with identifying the second participant, identify, based on the context information associated with the video event, a third location of the second participant in the replay; and augment the display of the replay of the occurrence with a second graphical overlay displayed at a third display location corresponding to the third location of the second participant, the second graphical overlay indicating a second trajectory of the second participant. ### Claim 19 (depends on 1) The non-transitory computer-readable storage medium of claim 1, wherein the context information includes at least one of: second information corresponding to opposing parties of the video event; a second audio stream of the video event; and a second annotated event stream of the video event. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the second information corresponding to the opposing parties of the video event indicates: respective jersey numbers of the participants of the opposing parties; respective identities of the participants of the opposing parties; and respective jersey designs of the opposing parties. ### Claim 21 An electronic device comprising: a display; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: while displaying, on the display, a video event: receiving, by a digital assistant operating on the electronic device, a natural language speech input corresponding to a participant of the video event; in accordance with receiving the natural language speech input, identifying, by the digital assistant, based on context information associated with the video event, a first location of the participant; and in accordance with identifying the first location of the participant, augmenting, by the digital assistant, the display of the video event with a graphical overlay displayed at a first display location corresponding to the first location of the participant. ### Claim 22 A method, comprising: at an electronic device having one or more processors, memory, and a display: while displaying, on the display, a video event: receiving, by a digital assistant operating on the electronic device, a natural language speech input corresponding to a participant of the video event; in accordance with receiving the natural language speech input, identifying, by the digital assistant, based on context information associated with the video event, a first location of the participant; and in accordance with identifying the first location of the participant, augmenting, by the digital assistant, the display of the video event with a graphical overlay displayed at a first display location corresponding to the first location of the participant.