# US 12,003,825 B1 · Enhanced control of video subtitles > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,825 B1 - Document ID: US12003825B1 - Status: Granted - Kind: B1 - Application number: 17/949,822 - Filed: 2022-09-21 - Granted: 2024-06-04 - Earliest claimed: 2022-09-21 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Naveen Sudhakaran Nair - Web page: https://patentgrep.com/patent/US12003825B1 ## CPC - H04N 21/4884: {for displaying subtitles} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Devices, systems, and methods are provided for presenting on-screen text during video playback. A method may include detecting a user request to determine when to activate and deactivate presentation of on-screen text during playback of a video; inputting, to a machine learning model, text data of video titles, audio data of the video titles, video frames of the video titles, and user data associated with users of a streaming video application; generating, using the machine learning model, based on the text data, the audio data, the video frames, and the user data, the first times and the second times; sending a bitstream comprising streaming video and indications of the first times and the second times; activating, based on the first times, presentation of the on-screen text during presentation of the streaming video; and deactivating, based on the second times, presentation of the on-screen text during presentation of the streaming video. ## Claims ### Claim 1 A method for presenting on-screen text during video playback, the method comprising: detecting, by at least one processor of a first device, a user request to automatically determine when to activate and deactivate presentation of on-screen text during playback of a video using a streaming video application, the on-screen text comprising subtitles; providing, by at least one processor of a second device, a neural network trained to predict first times when the user is likely to activate the on-screen text and second times when the user is likely to deactivate the on-screen text based on identifying text features, audio features, video frame features, and user features at times when users have activated and deactivated subtitles; identifying, by the at least one processor of the second device, on-screen text preferences of the user; inputting, by the at least one processor of the second device, to the neural network, text data of video titles, audio data of the video titles, video frames of the video titles, and user data associated with users of the streaming video application, the user data comprising the on-screen text preferences; generating, by the at least one processor of the second device, using the neural network, first vector embeddings comprising first values indicative of first text features of the text data, second vector embeddings comprising second values indicative of first audio features of the audio data, third vector embeddings comprising third values indicative of first video features of the video frames, and fourth vector embeddings comprising fourth values indicative of first user features of the user data; generating, by the at least one processor of the second device, using the neural network, based on comparisons of the first vector embeddings, the second vector embeddings, the third vector embeddings, and the fourth vector embeddings to the text features, the audio features, the video frame features, and the user features, predictions for the first times and the second times; sending, by the at least one processor of the second device, to the first device, a bitstream comprising streaming video and indications of the first times and the second times; activating, by the at least one processor of the first device, based on the first times, presentation of the on-screen text during presentation of the streaming video; and deactivating, by the at least one processor of the first device, based on the second times, presentation of the on-screen text during presentation of the streaming video. ### Claim 2 (depends on 1) The method of claim 1, wherein the neural network comprises a text encoder associated with generating the first vector embeddings, an audio encoder associated with generating the second vector embeddings, a video encoder associated with generating the third vector embeddings, and a user encoder associated with generating the fourth vector embeddings. ### Claim 3 (depends on 1) The method of claim 1, further comprising: identifying the first times based on when users have activated the on-screen text during presentation of the streaming video; and identifying the second times based on when the users have deactivated the on-screen text during presentation of the streaming video. ### Claim 4 (depends on 1) The method of claim 1, further comprising: identifying the first times based on when users have activated the on-screen text during presentation of the video titles; and identifying the second times based on when the users have deactivated the on-screen text during presentation of the video titles, wherein the streaming video is absent from the video titles. ### Claim 5 A method for presenting on-screen text during video playback, the method comprising: detecting, by at least one processor of a first device, a user request to automatically determine when to activate and deactivate presentation of on-screen text during playback of a video using a streaming video application; providing, by at least one processor of a second device, a machine learning model trained to predict first times when the user is likely to activate the on-screen text and second times when the user is likely to deactivate the on-screen text based on identifying text features, audio features, video frame features, and user features at times when users have activated and deactivated subtitles; inputting, by the at least one processor of the second device, to the machine learning model, text data of video titles, audio data of the video titles, video frames of the video titles, and user data associated with users of the streaming video application, the user data comprising on-screen text preferences of the user; generating, by the at least one processor of the second device, using the machine learning model, based on comparisons of the text data, the audio data, the video frames, and the user data to the text features, the audio features, the video frame features, and the user features, predictions for the first times and the second times; sending, by the at least one processor of the second device, to the first device, a bitstream comprising streaming video and indications of the first times and the second times; activating, by the at least one processor of the first device, based on the first times, presentation of the on-screen text during presentation of the streaming video; and deactivating, by the at least one processor of the first device, based on the second times, presentation of the on-screen text during presentation of the streaming video. ### Claim 6 (depends on 5) The method of claim 5, wherein the machine learning model comprises a text encoder associated with generating first embeddings based on the text data, an audio encoder associated with generating second embeddings based on the audio data, a video encoder associated with generating third embeddings based on the video frames, and a user encoder associated with generating fourth embeddings based on the user data. ### Claim 7 (depends on 6) The method of claim 6, wherein the first embeddings comprise a vector of first values indicative of text features of subtitles, wherein the second embeddings comprise a vector of second values indicative of audio features of spoken audio and background noise, and wherein the third embeddings comprise a vector of third values indicative of a number of video frames during which the on-screen text is to be presented, and wherein the comparisons comprise comparing the text features, the audio features, and the video frame features to the vector of first values, the vector of second values, and the vector of third values. ### Claim 8 (depends on 7) The method of claim 7, wherein the fourth embeddings comprise a vector of fourth features indicative of when a user from which the user request was received has activated the on-screen text and of when the user has deactivated the on-screen text, and wherein the comparisons further comprise comparing the user features to the vector of fourth values. ### Claim 9 (depends on 7) The method of claim 7, wherein the fourth embeddings comprise a vector of fourth features indicative of when users of the streaming video application have activated the on-screen text and of when the users have deactivated the on-screen text, and wherein the comparisons further comprise comparing the user features to the vector of fourth values. ### Claim 10 (depends on 5) The method of claim 5, further comprising: identifying the first times based on when users have activated the on-screen text during presentation of the streaming video; and identifying the second times based on when the users have deactivated the on-screen text during presentation of the streaming video. ### Claim 11 (depends on 5) The method of claim 5, further comprising: identifying the first times based on when users have activated the on-screen text during presentation of the video titles; and identifying the second times based on when the users have deactivated the on-screen text during presentation of the video titles, wherein the streaming video is absent from the video titles. ### Claim 12 (depends on 5) The method of claim 5, further comprising: detecting, during presentation of the streaming video during one of the second times, a second user selection to activate the on-screen text, that a volume of the first device is below a volume threshold, or that an ambient noise level exceeds an ambient noise threshold; sending, to the second device, an indication that presentation of the on-screen text was activated during the one of the second times based on the second user selection, the volume being below the volume threshold, or the ambient noise level exceeding the ambient noise threshold; and updating the machine learning model based on the indication, wherein activating the presentation of the on-screen text during presentation of the streaming video is further based on the second user selection, the volume being below the volume threshold, or the ambient noise level exceeding the ambient noise threshold. ### Claim 13 (depends on 12) The method of claim 12, wherein the second user selection is based on a first user push of a button unassociated with an on-screen selection, and wherein the presentation of the on-screen text is deactivated based on a second user push of the button or by depressing the button. ### Claim 14 (depends on 12) The method of claim 12, wherein the second user selection is based on a user hovering a selector over a portion of a display of the first device to activate the presentation of the on-screen text, and wherein the presentation of the on-screen text is deactivated based on the user moving the selector off of the portion of the display. ### Claim 15 (depends on 14) The method of claim 14, wherein the second user selection is based on the user selecting an on-screen text option while hovering the selector over the portion of the display. ### Claim 16 (depends on 14) The method of claim 14, wherein the hovering is unassociated with a subsequent user input to activate the presentation of the on-screen text. ### Claim 17 A system for presenting on-screen text during video playback, the system comprising: a first device comprising first memory coupled to at least one processor; and a second device, wherein the at least one processor is configured to: detect a user request to automatically determine when to activate and deactivate presentation of on-screen text during playback of a video using a streaming video application; receive a bitstream from the second device, the bitstream comprising streaming video and indications of first times when the user is likely to activate the on-screen text and second times when the user is likely to deactivate the on-screen text activate, based on the first times, presentation of the on-screen text during presentation of the streaming video; and deactivate, based on the second times, presentation of the on-screen text during presentation of the streaming video, wherein the second device is configured to: provide a machine learning model trained to predict the first times and the second times based on identifying text features, audio features, video frame features, and user features at times when users have activated and deactivated subtitles; input, to the machine learning model, text data of video titles, audio data of the video titles, video frames of the video titles, and user data associated with users of the streaming video application, the user data comprising on-screen text preferences of the user; generate, using the machine learning model, based on comparisons of the text data, the audio data, the video frames, and the user data to the text features, the audio features, the video frame features, and the user features, predictions for the first times and the second times; and send, to the first device, the bitstream. ### Claim 18 (depends on 17) The system of claim 17, wherein the machine learning model comprises a text encoder associated with generating first embeddings based on the text data, an audio encoder associated with generating second embeddings based on the audio data, a video encoder associated with generating third embeddings based on the video frames, and a user encoder associated with generating fourth embeddings based on the user data. ### Claim 19 (depends on 18) The system of claim 18, wherein the first embeddings comprise a vector of first features indicative of text features of subtitles, wherein the second embeddings comprise a vector of second features indicative of audio features of spoken audio and background noise, and wherein the third embeddings comprise a vector of third features indicative of a number of video frames during which the on-screen text is to be presented, and wherein the comparisons comprise comparing the text features, the audio features, and the video frame features to the vector of first values, the vector of second values, and the vector of third values. ### Claim 20 (depends on 19) The system of claim 19, wherein the fourth embeddings comprise a vector of fourth features indicative of when a user from which the user request was received has activated the on-screen text and of when the user has deactivated the on-screen text, and wherein the comparisons further comprise comparing the user features to the vector of fourth values.