# US 2025/0061593 A1 · Reducing texture lookups in extended-reality applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0061593 A1
- Document ID: US20250061593A1
- Status: Published application
- Kind: A1
- Application number: 18/449,352
- Filed: 2023-08-14
- Published: 2025-02-20
- Earliest claimed: 2023-08-14
- Assignees: Varjo Technologies Oy
- Applicants: Varjo Technologies Oy
- Inventors: Mikko Strandborg; Antti Hirvonen
- Web page: https://patentgrep.com/patent/US20250061593A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 7/50: Depth or shape recovery
- G06T 7/11: Region-based segmentation
- H04N 13/128: Adjusting depth or disparity
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- H04N 13/344: with head-mounted left-right displays
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A display apparatus has a display area of an output image divided into a plurality of regions. For a given region of the output image, a corresponding first region in a video-see-through (VST) image and a corresponding second region in at least one virtual-reality (VR) image are determined. From a depth map corresponding to the VST image, a minimum optical depth and a maximum optical depth in the first region of the VST image are determined. From a depth map corresponding to the at least one VR image, a minimum optical depth and a maximum optical depth in the second region of the at least one VR image are determined. When the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, pixel data for pixels in the given region of the output image from the second region of the at least one VR image is fetched. Otherwise, when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image, the pixel data for the pixels in the given region of the output image from the first region of the VST image is fetched.

## Claims

### Claim 1

A method implemented by a display apparatus, the method comprising:
dividing a display area of an output image into a plurality of regions;
for a given region of the output image, determining a corresponding first region in a video-see-through (VST) image and a corresponding second region in at least one virtual-reality (VR) image;
determining, from a depth map corresponding to the VST image, a minimum optical depth and a maximum optical depth in the first region of the VST image;
determining, from a depth map corresponding to the at least one VR image, a minimum optical depth and a maximum optical depth in the second region of the at least one VR image;
when the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, fetching pixel data for pixels in the given region of the output image from the second region of the at least one VR image; and
when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image, fetching the pixel data for the pixels in the given region of the output image from the first region of the VST image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising when the minimum optical depth in the first region of the VST image is not greater than the maximum optical depth in the second region of the at least one VR image, and the minimum optical depth in the second region of the at least one VR image is not greater than the maximum optical depth in the first region of the VST image,
fetching, from the depth map corresponding to the VST image, a first optical depth for a given pixel in the given region of the output image;
fetching, from the depth map corresponding to the at least one VR image, a second optical depth for the given pixel;
when the first optical depth is greater than or equal to the second optical depth, fetching pixel data for the given pixel from the second region of the at least one VR image; and
when the first optical depth is smaller than the second optical depth, fetching pixel data for the given pixel from the first region of the VST image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
when the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, detecting whether a transparency of at least a part of the second region of the at least one VR image is more than a predefined threshold; and
when it is detected that the transparency of at least a part of the second region of the at least one VR image is more than the predefined threshold, fetching pixel data for at least a subset of the pixels in the given region of the output image from the first region of the VST image, wherein said subset of the pixels in the given region of the output image correspond to said part of the second region of the at least one VR image whose transparency is more than the predefined threshold.

### Claim 4

A method implemented by at least one server that is communicably coupled to at least one display apparatus, the method comprising:
receiving, from the at least one display apparatus, depth data corresponding to a video-see-through (VST) image captured at the at least one display apparatus;
dividing a display area into a plurality of regions;
for a given region of the display area, determining a corresponding first region in the VST image and a corresponding second region in at least one virtual-reality (VR) image;
determining, from the depth data corresponding to the VST image, a minimum optical depth and a maximum optical depth in the first region of the VST image;
determining, from at least one depth map corresponding to the at least one VR image, a minimum optical depth and a maximum optical depth in the second region of the at least one VR image;
when the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, classifying the given region of the display area as a VR region that is to be rendered from the at least one VR image only;
when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image, classifying the given region of the display area as a VST region that is to be rendered from the VST image only;
when the minimum optical depth in the first region of the VST image is not greater than the maximum optical depth in the second region of the at least one VR image, and the minimum optical depth in the second region of the at least one VR image is not greater than the maximum optical depth in the first region of the VST image, classifying the given region of the display area as a mixed region that is to be rendered from the at least one VR image and the VST image; and
sending, to the at least one display apparatus, information indicative of respective classifications of the plurality of regions.

### Claim 5 (depends on 4)

The method of claim 4, further comprising skipping encoding and transport of the second region of the at least one VR image to the at least one display apparatus, when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of regions of the display area are in a form of tiles.

### Claim 7 (depends on 1)

The method of claim 1, wherein the display area is divided into the plurality of regions iteratively.

### Claim 8

A display apparatus comprising a processor configured to:
divide a display area of an output image into a plurality of regions;
for a given region of the output image, determine a corresponding first region in a video-see-through (VST) image and a corresponding second region in at least one virtual-reality (VR) image;
determine, from a depth map corresponding to the VST image, a minimum optical depth and a maximum optical depth in the first region of the VST image;
determine, from a depth map corresponding to the at least one VR image, a minimum optical depth and a maximum optical depth in the second region of the at least one VR image;
when the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, fetch pixel data for pixels in the given region of the output image from the second region of the at least one VR image; and
when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image, fetch the pixel data for the pixels in the given region of the output image from the first region of the VST image.

### Claim 9 (depends on 8)

The display apparatus of claim 8, wherein the processor is configured to: when the minimum optical depth in the first region of the VST image is not greater than the maximum optical depth in the second region of the at least one VR image, and the minimum optical depth in the second region of the at least one VR image is not greater than the maximum optical depth in the first region of the VST image,
fetch, from the depth map corresponding to the VST image, a first optical depth for a given pixel in the given region of the output image;
fetch, from the depth map corresponding to the at least one VR image, a second optical depth for the given pixel;
when the first optical depth is greater than or equal to the second optical depth, fetch pixel data for the given pixel from the second region of the at least one VR image; and
when the first optical depth is smaller than the second optical depth, fetch pixel data for the given pixel from the first region of the VST image.

### Claim 10 (depends on 8)

The display apparatus claim 8, wherein the processor is configured to:
when the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, detect whether a transparency of at least a part of the second region of the at least one VR image is more than a predefined threshold; and
when it is detected that the transparency of at least a part of the second region of the at least one VR image is more than the predefined threshold, fetch pixel data for at least a subset of the pixels in the given region of the output image from the first region of the VST image, wherein said subset of the pixels in the given region of the output image correspond to said part of the second region of the at least one VR image whose transparency is more than the predefined threshold.

### Claim 11 (depends on 8)

The display apparatus claim 8, wherein the plurality of regions of the display area are in a form of tiles.

### Claim 12 (depends on 8)

The display apparatus claim 8, wherein the display area is divided into the plurality of regions iteratively.

### Claim 13

A system comprising at least one server that is communicably coupled to at least one display apparatus, the at least one server being configured to:
receive, from the at least one display apparatus, depth data corresponding to a video-see-through (VST) image captured at the at least one display apparatus;
divide a display area into a plurality of regions;
for a given region of the display area, determine a corresponding first region in the VST image and a corresponding second region in at least one virtual-reality (VR) image;
determine, from the depth data corresponding to the VST image, a minimum optical depth and a maximum optical depth in the first region of the VST image;
determine, from at least one depth map corresponding to the at least one VR image, a minimum optical depth and a maximum optical depth in the second region of the at least one VR image;
when the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, classify the given region of the display area as a VR region that is to be rendered from the at least one VR image only;
when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image, classify the given region of the display area as a VST region that is to be rendered from the VST image only;
when the minimum optical depth in the first region of the VST image is not greater than the maximum optical depth in the second region of the at least one VR image, and the minimum optical depth in the second region of the at least one VR image is not greater than the maximum optical depth in the first region of the VST image, classify the given region of the display area as a mixed region that is to be rendered from the at least one VR image and the VST image; and
send, to the at least one display apparatus, information indicative of respective classifications of the plurality of regions.

### Claim 14 (depends on 13)

The system of claim 13, wherein the at least one server is configured to skip encoding and transport of the second region of the at least one VR image to the at least one display apparatus, when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image.

### Claim 15 (depends on 13)

The system claim 13, wherein the display area is divided into the plurality of regions iteratively.
