# US 12,118,144 B2 · Gaze based video stream processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,144 B2 - Document ID: US12118144B2 - Status: Granted - Kind: B2 - Application number: 18/237,473 - Filed: 2023-08-24 - Published: 2023-12-07 - Granted: 2024-10-15 - Earliest claimed: 2021-04-16 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Brian T. Hawkins; Alexandros Neophytou; Eric Chris Wolfgang Sommerlade - Web page: https://patentgrep.com/patent/US12118144B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - H04N 21/234309: {by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo (conversion of standards in analog television systems H04N7/01)} - G06T 7/73: using feature-based methods - H04N 7/0127: {by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter} - H04N 7/0806: {the signals being two or more video signals (H04N7/0803, H04N7/081 take precedence)} - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - H04N 7/152: {Multipoint control units therefor} - H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment} - H04N 21/234363: {by altering the spatial resolution, e.g. for clients with a lower screen resolution} - H04N 21/234381: {by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping} - H04N 21/4223: Cameras (H04N23/00 takes precedence) - H04N 21/4666: {using neural networks, e.g. processing the feedback provided by the user} - H04N 21/6587: Control parameters, e.g. trick play commands, viewpoint selection - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In various embodiments, a method for processing video streams is described. A plurality of video streams for transmission to a display device are received. The plurality of video streams have respective initial image quality levels. An estimated gaze location of a user of the display device is estimated. At least one video stream of the plurality of video streams is processed to have a modified image quality level based on the estimated gaze location. The modified image quality level is less than a corresponding initial image quality level. The plurality of video streams are transmitted to the display device. ## Claims ### Claim 1 A method for processing video streams, the method comprising: receiving a plurality of video streams for transmission to a plurality of display devices, the plurality of video streams having respective initial image quality levels; estimating respective estimated gaze locations of users of the plurality of display devices based on images from the plurality of video streams using a neural network model; processing some video streams of the plurality of video streams to have a modified image quality level based on the respective estimated gaze locations, the modified image quality level being less than a corresponding initial image quality level; generating a plurality of composite video streams, wherein each composite video stream of the plurality of composite video streams comprises processed video streams having the modified image quality level and remaining video streams of the plurality of video streams, the processed video streams being outside of a corresponding respective estimated gaze location and the remaining video streams being within the corresponding estimated gaze location; and transmitting the plurality of composite video streams to the plurality of display devices. ### Claim 2 (depends on 1) The method of claim 1, wherein estimating the respective estimated gaze locations comprises: extracting facial features from the images; providing the extracted facial features as an input to the neural network model. ### Claim 3 (depends on 2) The method of claim 2, wherein the facial features comprise at least one of a location of eyes, pupils, nose, chin, or ears of the users of the plurality of display devices. ### Claim 4 (depends on 1) The method of claim 1, wherein estimating the respective estimated gaze locations comprises estimating the respective estimated gaze locations in real-time. ### Claim 5 (depends on 1) The method of claim 1, wherein estimating the respective estimated gaze locations comprises executing the neural network model on a neural processing unit. ### Claim 6 (depends on 1) The method of claim 1, wherein the neural network model outputs coordinates of respective estimated gaze locations within respective display devices. ### Claim 7 (depends on 1) The method of claim 1, wherein the modified image quality level has at least one of a reduced pixel count, a reduced frame rate, and an increased compression relative to the corresponding initial image quality level. ### Claim 8 (depends on 1) The method of claim 1, wherein processing the some of the video streams comprises selecting the modified image quality level from a plurality of quality levels based on a distance between the some of the video streams as displayed on the display devices and the respective estimated gaze locations. ### Claim 9 A computing device for processing video streams, the computing device comprising: a gaze detector; and a stream processor; wherein: the computing device is configured to receive a plurality of video streams for display by a display device, the plurality of video streams having respective initial image quality levels; the gaze detector is configured to estimate an estimated gaze location of a user of the display device; the stream processor is configured to process some video streams of the plurality of video streams to have a modified image quality level based on the estimated gaze location, the modified image quality level being higher than a corresponding initial image quality level; and the system is configured to display the processed video streams and remaining video streams of the plurality of video streams via the display device, the remaining video streams having the respective initial image quality levels. ### Claim 10 (depends on 9) The computing device of claim 9, wherein the gaze detector comprises a neural network model that estimates the estimated gaze locations based on the images. ### Claim 11 (depends on 10) The computing device of claim 10, wherein the gaze detector is configured to extract and provide facial features as an input to the neural network model. ### Claim 12 (depends on 11) The computing device of claim 11, wherein the facial features comprise at least one of a location of eyes, pupils, nose, chin, or ears of the users of the plurality of display devices. ### Claim 13 (depends on 10) The computing device of claim 10, wherein the gaze detector is configured to estimate the estimated gaze locations in real time. ### Claim 14 (depends on 10) The computing device of claim 10, wherein the gaze detector comprises a neural processing unit on which the neural network model is executed. ### Claim 15 A computing device for processing video streams, the computing device comprising: a gaze detector; and a stream processor; wherein: the computing device is configured to receive a plurality of video streams for transmission to a plurality of display devices, the plurality of video streams having respective initial image quality levels; the gaze detector is configured to estimate respective estimated gaze locations of users of the plurality of display devices based on images from the plurality of video streams using a neural network model; the stream processor is configured to: process some video streams of the plurality of video streams to have a modified image quality level based on the respective estimated gaze locations, the modified image quality level being less than a corresponding initial image quality level; generate a plurality of composite video streams, wherein each composite video stream of the plurality of composite video streams comprises processed video streams having the modified image quality level and remaining video streams of the plurality of video streams, the processed video streams being outside of a corresponding respective estimated gaze location and the remaining video streams being within the corresponding estimated gaze location; and transmit the plurality of composite video streams to the plurality of display devices. ### Claim 16 (depends on 15) The method of claim 15, wherein estimating the respective estimated gaze locations comprises: extracting facial features from the images; providing the extracted facial features as an input to the neural network model. ### Claim 17 (depends on 16) The method of claim 16, wherein the facial features comprise at least one of a location of eyes, pupils, nose, chin, or ears of the users of the plurality of display devices. ### Claim 18 (depends on 15) The method of claim 15, wherein estimating the respective estimated gaze locations comprises estimating the respective estimated gaze locations in real-time. ### Claim 19 (depends on 15) The method of claim 15, wherein estimating the respective estimated gaze locations comprises executing the neural network model on a neural processing unit. ### Claim 20 (depends on 15) The method of claim 15, wherein the neural network model outputs coordinates of respective estimated gaze locations within respective display devices.