# US 2025/0155970 A1 · Gaze depth-assisted encoding of region of interest in images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0155970 A1
- Document ID: US20250155970A1
- Status: Published application
- Kind: A1
- Application number: 18/505,680
- Filed: 2023-11-09
- Published: 2025-05-15
- Earliest claimed: 2023-11-09
- Assignees: Varjo Technologies Oy
- Applicants: Varjo Technologies Oy
- Inventors: Mikko Strandborg; Timo Toivanen; Mikko Ollila
- Web page: https://patentgrep.com/patent/US20250155970A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/50: Depth or shape recovery
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06T 15/00: Three-dimensional [3D] image rendering
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- H04N 19/115: Selection of the code volume for a coding unit prior to coding
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/17: the unit being an image region, e.g. an object
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system includes at least one server configured to receive, from a client device, information indicative of gaze directions of a user's eyes; determine a gaze point and a gaze depth of the user's eyes, using said information; identify, in an image, a gaze-contingent region that includes and surrounds the gaze point; identify a region of interest in the gaze-contingent region, wherein the region of interest comprises pixels whose optical depth lies within a first predefined distance (Δd1) from the gaze depth; encode the image by applying a first encoding setting to the pixels belonging to the region of interest; and send the image that is encoded, to the client device.

## Claims

### Claim 1

A system comprising at least one server configured to:
receive, from a client device, information indicative of gaze directions of a user's eyes;
determine a gaze point and a gaze depth of the user's eyes, using said information;
identify, in an image, a gaze-contingent region that includes and surrounds the gaze point;
identify a region of interest in the gaze-contingent region, wherein the region of interest comprises pixels whose optical depth lies within a first predefined distance (Δd1) from the gaze depth;
encode the image by applying a first encoding setting to the pixels belonging to the region of interest; and
send the image that is encoded, to the client device.

### Claim 2 (depends on 1)

The system of claim 1, wherein the region of interest further comprises pixels that lie within a second predefined distance (Δd2) from the gaze point.

### Claim 3 (depends on 1)

The system of claim 1, wherein the first encoding setting comprises:
emphasis values indicative of a relative importance of the pixels belonging to the region of interest with respect to other pixels in the image,
a first compression ratio to be employed while encoding the pixels belonging to the region of interest, when the encoded image is to be transported by utilizing at least a portion of an entire bandwidth of a communication channel.

### Claim 4 (depends on 3)

The system of claim 3, wherein the emphasis values are determined based on an emphasis curve that defines how emphasis values vary with respect to a distance of a given point from a three-dimensional position of the gaze point.

### Claim 5 (depends on 1)

The system of claim 1, wherein the at least one server is further configured to:
obtain a baseline encoding setting map comprising baseline foveated encoding settings for the image, wherein the baseline foveated encoding settings comprise a first baseline encoding setting for pixels belonging to the gaze-contingent region, and a second baseline encoding setting for pixels belonging to a peripheral region of the image; and
determine a modulation that is to be applied to the first baseline encoding setting, wherein the first encoding setting is equal to the modulated first baseline encoding setting.

### Claim 6 (depends on 1)

The system of claim 1, wherein when encoding the image, the at least one server is further configured to:
apply a second encoding setting to remaining pixels in the image, wherein the second encoding setting is different from the first encoding setting; or
apply a second encoding setting to remaining pixels in the gaze-contingent region and a third encoding setting to further remaining pixels in a peripheral region of the image, wherein the second encoding setting and the third encoding setting are different from each other and from the first encoding setting.

### Claim 7 (depends on 1)

The system of claim 1, wherein when determining the gaze point and the gaze depth of the user's eyes, the at least one server is configured to perform one of:
identify the gaze point to be a point in a previous image at which the gaze directions of the user's eyes converged, and identify the gaze depth to be an optical depth associated with said point in an environment represented by the previous image;
identify the gaze point to be a point in a previous image at which the gaze directions of the user's eyes converged, and identify the gaze depth to be an optical depth at which at least one autofocus camera was focused for capturing a previous real-world image, wherein the previous image comprises the previous real-world image;
identify the gaze point to be a point in a previous image at which the gaze directions of the user's eyes converged, and determine the gaze depth using a depth value of a virtual object present at said point;
predict the gaze point to be a point in the image at which the gaze directions of the user's eyes are expected to converge at a future time instant, and identify the gaze depth to be an optical depth associated with said point in an environment represented by the image.

### Claim 8 (depends on 1)

The system of claim 1, wherein the image is any image for which per-pixel depth information is available to the at least one server, said image being at least one of:
a virtual-reality image,
a depth image captured by a depth sensor,
a real-world depth image captured by a depth sensor-based video see-through camera;
an image reconstructed using a three-dimensional data structure representing a three-dimensional environment.

### Claim 9

A method comprising:
receiving, from a client device, information indicative of gaze directions of a user's eyes;
determining a gaze point and a gaze depth of the user's eyes, using said information;
identifying, in an image, a gaze-contingent region that includes and surrounds the gaze point;
identifying a region of interest in the gaze-contingent region, wherein the region of interest comprises pixels whose optical depth lies within a first predefined distance (Δd1) from the gaze depth;
encoding the image by applying a first encoding setting to the pixels belonging to the region of interest; and
sending the image that is encoded, to the client device.

### Claim 10 (depends on 9)

The method of claim 9, wherein the region of interest further comprises pixels that lie within a second predefined distance (Δd2) from the gaze point.

### Claim 11 (depends on 9)

The method of claim 9, wherein the first encoding setting comprises:
emphasis values indicative of a relative importance of the pixels belonging to the region of interest with respect to other pixels in the image,
a first compression ratio to be employed while encoding the pixels belonging to the region of interest, when the encoded image is to be transported by utilizing at least a portion of an entire bandwidth of a communication channel.

### Claim 12 (depends on 11)

The method of claim 11, wherein the emphasis values are determined based on an emphasis curve that defines how emphasis values vary with respect to a distance of a given point from a three-dimensional position of the gaze point.

### Claim 13 (depends on 9)

The method of claim 9, further comprising:
obtaining a baseline encoding setting map comprising baseline foveated encoding settings for the image, wherein the baseline foveated encoding settings comprise a first baseline encoding setting for pixels belonging to the gaze-contingent region, and a second baseline encoding setting for pixels belonging to a peripheral region of the image;
determining a modulation that is to be applied to the first baseline encoding setting, wherein the first encoding setting is equal to the modulated first baseline encoding setting.

### Claim 14 (depends on 9)

The method of claim 9, wherein when encoding the image, the method comprises:
applying a second encoding setting to remaining pixels in the image, wherein the second encoding setting is different from the first encoding setting; or
applying a second encoding setting to remaining pixels in the gaze-contingent region and a third encoding setting to further remaining pixels in a peripheral region of the image, wherein the second encoding setting and the third encoding setting are different from each other and from the first encoding setting.

### Claim 15 (depends on 9)

The method of claim 9, wherein when determining the gaze point and the gaze depth of the user's eyes, the method comprises one of:
identifying the gaze point to be a point in a previous image at which the gaze directions of the user's eyes converged, and identify the gaze depth to be an optical depth associated with said point in an environment represented by the previous image;
identifying the gaze point to be a point in a previous image at which the gaze directions of the user's eyes converged, and identify the gaze depth to be an optical depth at which at least one autofocus camera was focused for capturing a previous real-world image, wherein the previous image comprises the previous real-world image;
identifying the gaze point to be a point in a previous image at which the gaze directions of the user's eyes converged, and determine the gaze depth using a depth value of a virtual object present at said point;
predicting the gaze point to be a point in the image at which the gaze directions of the user's eyes are expected to converge at a future time instant, and identify the gaze depth to be an optical depth associated with said point in an environment represented by the image.
