# US 12,015,787 B2 · Predicting and verifying regions of interest selections > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,015,787 B2 - Document ID: US12015787B2 - Status: Granted - Kind: B2 - Application number: 17/198,085 - Filed: 2021-03-10 - Published: 2021-06-24 - Granted: 2024-06-18 - Earliest claimed: 2016-11-17 - Assignees: Dolby Laboratories Licensing Corporation - Applicants: Dolby Laboratories Licensing Corporation - Inventors: Chaitanya Atluru; Ajit Ninan - Web page: https://patentgrep.com/patent/US12015787B2 ## CPC - H04N 19/167: Position within a video image, e.g. region of interest [ROI] - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - H04N 19/162: User input - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 19/182: the unit being a pixel - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Video images are rendered in viewports of content viewers. Each content viewer views video images through a respective viewport in the viewports. Spatial locations in the video images to which foveal visions of the content viewers are directed are determined. ROIs in the video images are identified based on the spatial locations in the video images. ## Claims ### Claim 1 A method for determining regions of interest (ROIs) in video images, comprising: generating user viewing behavior data while rendering a video clip including a video image in a plurality of viewports of a plurality of content viewers, wherein each content viewer in the plurality of content viewers views the video clip including the video image through a respective viewport in the plurality of viewports; determining, from the user viewing behavior data, a plurality of spatial locations as candidate ROIs, in the video image and other video images of the video clip, to which a plurality of foveal visions of the plurality of content viewers is directed; identifying, based on the plurality of spatial locations in the video image and the other images of the video clip, one or more unique spatial paths represented in the video clip, wherein the one or more unique spatial paths are identified by applying one or more computing techniques to the plurality of spatial locations in the video image and the other images of the video clip; using the one or more unique spatial paths represented in the video clip to identify at least one of two or more ROIs in the video image; encoding two or more image portions in the two or more ROIs of the video image and one or more other image portions outside the two or more ROIs of the video image into a video stream; wherein the two or more image portions are non-overlapping with the one or more other image portions; wherein the two or more image portions in the two or more ROIs of the video image are encoded in the video stream at spatiotemporal resolutions higher than other spatiotemporal resolutions at which the one or more other image portions outside the two or more ROIs of the video image are encoded in the video stream; transmitting, to a streaming client device, the video stream. ### Claim 2 (depends on 1) The method of claim 1, further comprising: compressing pixel values, of the video image, in the two or more image portions in the two or more ROIs with first peak-signal-noise-ratios (PSNRs); compressing pixel values, of the video image, in the one or more other image portions outside the two or more ROIs with second PSNRs lower than the first PSNRs. ### Claim 3 (depends on 1) The method of claim 1, further comprising: classifying the plurality of foveal visions of the plurality of content viewers into two or more foveal vision clusters; identifying the two or more ROIs from the two or more foveal vision clusters. ### Claim 4 (depends on 3) The method of claim 3, wherein the two or more ROIs include at least one ROI predicted based on one or more of: artificial intelligence methods, machine learning methods, k-NN classification algorithms, k-means clustering algorithms, linear discriminant analyses (LDAs), nearest centroid classifiers, nearest prototype classifiers, or Rochhio classifiers. ### Claim 5 (depends on 1) The method of claim 1, further comprising: determining one or more spatial locations, represented in the one or more first video images, of sound sources, in spatial audio to be concurrently rendered with the one or more first video images; determining at least one of the one or more ROIs based in part on the one or more spatial locations of the sound sources in the spatial audio. ### Claim 6 (depends on 1) The method of claim 1, wherein the plurality of image locations is identified based on first viewing behavior data that comprise the plurality of foveal visions of the plurality of content viewers; wherein the first viewing behavior data are collected from the plurality of content viewers up to a first time point; further comprising: collecting second viewing behavior data that comprise a second plurality of foveal visions of a second plurality of content viewers, wherein the second viewing behavior data are at least partly collected from the plurality of content viewers after the first time point; determining a second plurality of spatial locations, in the video image, to which the second plurality of foveal visions of the plurality of content viewers is directed; identifying, based on the second plurality of spatial locations in the video image, one or more second ROIs in the video image. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining a set of user perceivable characteristics in connection with the video image; determining a set of second user perceivable characteristics in connection with a second video image; based on the set of user perceivable characteristics and the second set of user perceivable characteristics, predicting a second ROI, in the second video image, that have one or more second perceivable characteristics correlating with one or more user perceivable characteristics of an ROI in the video image. ### Claim 8 (depends on 7) The method of claim 7, wherein the second ROI in the second video image is identified from the one or more user perceivable characteristics before user viewing behavior data have been collected from the second video image. ### Claim 9 (depends on 7) The method of claim 7, wherein the second ROI in the second video image is identified from the one or more user perceivable characteristics after at least a part of user viewing behavior data has been collected from the second video image. ### Claim 10 (depends on 7) The method of claim 7, wherein the one or more user perceivable characteristics comprise at least one of: one or more visual characteristics, one or more audio characteristics, or one or more non-visual non-audio user perceptible characteristics. ### Claim 11 (depends on 7) The method of claim 7, wherein the video image is in a first video clip, and wherein the second video image is in a second different video clip. ### Claim 12 (depends on 7) The method of claim 7, wherein objective metrics of computer vision for the video image is different from those for the second video image. ### Claim 13 (depends on 12) The method of claim 12, wherein the objective metrics of computer vision comprise one or more of: luminance characteristics, luminance distributions, chrominance characteristics, chrominance distributions, or spatial resolutions. ### Claim 14 (depends on 7) The method of claim 7, further comprising: correlating a sound source, in spatial audio for the video image, to a ROI in the two or more ROIs of the video image; determining one or more second sound sources in second spatial audio for the second video image; predicting at least one of the one or more ROIs in the second video image, based at least in part on one or more spatial locations of the one or more second sound sources in the second video image. ### Claim 15 (depends on 1) An apparatus performing the method as recited in claim 1. ### Claim 16 (depends on 1) A system performing the method as recited in claim 1. ### Claim 17 (depends on 1) A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors causes performance of the method recited in claim 1. ### Claim 18 (depends on 1) A computing device comprising one or more processors and one or more non-transitory storage media, storing a set of instructions, which when executed by one or more processors cause performance of the method recited in claim 1.