# US 12,114,024 B2 · Streaming video with internally-variable frame quality based on intelligent identification of high-relevance regions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,114,024 B2 - Document ID: US12114024B2 - Status: Granted - Kind: B2 - Application number: 17/744,911 - Filed: 2022-05-16 - Published: 2023-11-16 - Granted: 2024-10-08 - Earliest claimed: 2022-05-16 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Mrinal Kumar Sharma - Web page: https://patentgrep.com/patent/US12114024B2 ## CPC - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - H04N 19/167: Position within a video image, e.g. region of interest [ROI] - H04N 19/169: characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding - H04N 19/17: the unit being an image region, e.g. an object - H04N 19/176: the region being a block, e.g. a macroblock - H04N 21/2187: Live feed - H04N 21/23406: {involving management of server-side video buffer} - H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment} - H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)} - H04N 21/472: End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content {(end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for streaming videos with internally-variable frame quality is implemented via a computing system including a processor. The method includes accessing a video including and evenly sampling the video frames. The method includes, for each sampled video frame, analyzing the video frame using a CNN model to determine whether the video frame includes high-relevance region(s). The method also includes, for each sampled video frame including high-relevance region(s), extracting coordinates of the high-relevance region(s) using the CNN model, and for each sampled video frame including high-relevance region(s) and each intervening video frame between the sampled video frame and a next sampled video frame, setting a minimum frame quality for the extracted coordinates. The method further includes streaming the video with internally-variable frame quality by buffering the extracted coordinates within the video frames with the minimum frame quality, while buffering remaining coordinates within the video frames with a lower frame quality. ## Claims ### Claim 1 A method for streaming a video with internally-variable frame quality, wherein the method is implemented via a computing system comprising a processor, and wherein the method comprises: accessing, via the processor, a video comprising a sequence of video frames; evenly sampling at least a first video frame in the sequence of video frames and a second video frame in the sequence of video frames, wherein unsampled intermediate video frame(s) are positioned between the first sampled video frame and the second sampled video frame in the sequence of video frames; analyzing regions of the first sampled video frame using a convolutional neural network (CNN) model to identify a high-relevance region comprising textual information, the high-relevance region being more relevant to a user than low-relevance region(s) within the first sampled video frame; extracting coordinates of the high-relevance region within the first sampled video frame using the CNN model; using the coordinates extracted from the first sampled video frame to identify corresponding high-relevance region(s) within the unsampled intermediate video frame(s), wherein the high-relevance region within the first sampled video frame and the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) have identical coordinates; and streaming the video such that the streamed video has an internally-variable frame quality, wherein streaming the video comprises buffering the high-relevance region of the first sampled video frame and the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) at a higher resolution than the low-relevance region(s) of the first sampled video frame. ### Claim 2 (depends on 1) The method of claim 1, wherein the CNN model comprises a region-based CNN (R-CNN) model. ### Claim 3 (depends on 2) The method of claim 2, wherein analyzing regions of the first sampled video frame using the CNN model to identify the high-relevance region comprises: applying the R-CNN model to the first sampled video frame; generating a layer of features corresponding to the first sampled video frame; identifying objects within the first sampled video frame; generating candidate region proposals for the objects identified within the first sampled video frame; classifying a presence of a high-relevance object based on the generated layer of features and the generated candidate region proposals; and extracting the coordinates of the high-relevance region corresponding to the high-relevance object within the first sampled video frame. ### Claim 4 (depends on 3) The method of claim 3, further comprising automatically applying offset values to the extracted coordinates to increase a precision of the extracted coordinates. ### Claim 5 (depends on 1) The method of claim 1, further comprising automatically updating a sampling frequency for evenly sampling video frames within the sequence of video frames. ### Claim 6 (depends on 1) The method of claim 1, wherein the video comprises a live video, and wherein the method is executed in real-time. ### Claim 7 (depends on 1) The method of claim 1, further comprising storing the video with internally-variable frame quality settings for subsequent streaming. ### Claim 8 (depends on 1) The method of claim 1, further comprising activating an internally-variable frame quality mode of a video streaming application based at least on user input received via a GUI element provided by the video streaming application, wherein activation of the internally-variable frame quality mode causes the video to be streamed with the internally-variable frame quality. ### Claim 9 (depends on 1) The method of claim 1, further comprising training the CNN model prior to accessing the video. ### Claim 10 (depends on 1) The method of claim 1, wherein the extracted coordinates identify the high-relevance region within the first sampled video frame as well as corresponding high-relevance region(s) within the unsampled intermediate video frame(s). ### Claim 11 (depends on 10) The method of claim 10, wherein streaming the video based on the extracted coordinates further comprises buffering the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) at a higher resolution than the low-relevance region(s) of the first sampled video frame. ### Claim 12 (depends on 1) The method of claim 1, wherein the coordinates extracted from the first sampled video frame are used to identify the corresponding high-relevance region(s) within the unsampled intermediate video frame(s) without comparing the unsampled intermediate video frame(s) with the first sampled video frame for purposes of object tracking. ### Claim 13 A computing system, comprising: a processor; and a computer-readable storage medium operatively coupled to the processor, the computer-readable storage medium comprising computer-executable instructions that, when executed by the processor, cause the processor to: access a video comprising a sequence of video frames; evenly sample at least a first video frame in the sequence of video frames and a second video frame in the sequence of video frames, wherein unsampled intermediate video frame(s) are positioned between the first sampled video frame and the second sampled video frame in the sequence of video frames; analyze regions of the first sampled video frame using a convolutional neural network (CNN) model to identify a high-relevance region comprising textual information, the high-relevance region being more relevant to a user than low-relevance region(s) within the first sampled video frame; extract coordinates of the high-relevance region within the first sampled video frame using the CNN model; using the coordinates extracted from the first sampled video frame to identify corresponding high-relevance region(s) within the unsampled intermediate video frame(s), wherein the high-relevance region within the first sampled video frame and the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) have identical coordinates; and stream the video based on the extracted coordinates such that the streamed video has an internally-variable frame quality, wherein the processor streams the video based on the extracted coordinates by buffering the high-relevance region of the first sampled video frame and the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) at a higher resolution than the low-relevance region(s) of the first sampled video frame. ### Claim 14 (depends on 13) The computing system of claim 13, wherein computer-executable instructions further cause the processor to automatically update a sampling frequency for evenly sampling video frames within the sequence of video frames. ### Claim 15 (depends on 13) The computing system of claim 13, wherein computer-executable instructions further cause the processor to store the video with internally-variable frame quality settings for subsequent streaming. ### Claim 16 (depends on 13) The computing system of claim 13, wherein computer-executable instructions further cause the processor to activate an internally-variable frame quality mode of a video streaming application based at least on user input received via a GUI element provided by the video streaming application, wherein activation of the internally-variable frame quality mode causes the video to be streamed with the internally-variable frame quality. ### Claim 17 (depends on 13) The computing system of claim 13, wherein computer-executable instructions further cause the processor to train the CNN model prior to accessing the video. ### Claim 18 (depends on 13) The computing system of claim 13, wherein the extracted coordinates identify the high-relevance region within the first sampled video frame as well as corresponding high-relevance region(s) within the unsampled intermediate video frame(s). ### Claim 19 (depends on 13) The computing system of claim 13, wherein streaming the video based on the extracted coordinates further comprises buffering the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) at a higher resolution than the low-relevance region(s) of the first sampled video frame. ### Claim 20 A computer-readable storage medium comprising computer-executable instructions that, when executed by a processor, cause the processor to: access a video comprising a sequence of video frames; evenly sample at least a first video frame in the sequence of video frames and a second video frame in the sequence of video frames, wherein unsampled intermediate video frame(s), positioned between the first sampled video frame and the second sampled video frame in the sequence of video frames; analyze regions of the first sampled video frame using a convolutional neural network (CNN) model to identify a high-relevance region comprising textual information, the high-relevance region being more relevant to a user than low-relevance region(s) within the first sampled video frame; extract coordinates of the high-relevance region within the first sampled video frame using the CNN model; use the coordinates extracted from the first sampled video frame to identify corresponding high-relevance region(s) within the unsampled intermediate video frame(s), wherein the high-relevance region within the first sampled video frame and the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) have identical coordinates; and stream the video based on the extracted coordinates such that the streamed video has an internally-variable frame quality, wherein the processor streams the video by buffering the high-relevance region of the first sampled video frame and the corresponding high-relevance region(s) of the unsampled intermediate video frame(s) at a higher resolution than the low-relevance region(s) of the first sampled video frame.