# US 12,010,442 B1 · Systems and methods incorporating adjustable focus and extended depth-of-field correction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,442 B1
- Document ID: US12010442B1
- Status: Granted
- Kind: B1
- Application number: 17/991,384
- Filed: 2022-11-21
- Published: 2024-05-23
- Granted: 2024-06-11
- Earliest claimed: 2022-11-21
- Assignees: Varjo Technologies Oy
- Applicants: Varjo Technologies Oy
- Inventors: Mikko Ollila
- Web page: https://patentgrep.com/patent/US12010442B1

## CPC

- H04N 23/959: by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
- G06T 5/00: Image enhancement or restoration
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 23/675: {comprising setting of focusing regions}
- G06T 2207/20221: Image fusion; Image merging
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system including server(s) configured to: receive, from N client devices, information indicating optical depths at which N users are gazing; send, to imaging device including left camera and right camera, information indicating N optical depths; and in each cycle of N consecutive pairs of left images and right images, for M ranging from 1 to N: receive Mth left image captured by focusing left camera at Mth optical depth; select such image for user(s) gazing within focusing range; generate extended depth-of-field (EDOF)-corrected Mth left images; receive Mth right image captured by focusing right camera according to another optical depth; select such image for other user(s) gazing within another focusing range; generate EDOF-corrected Mth right images; and send Mth left image, EDOF-corrected Mth left images, Mth right image, EDOF-corrected Mth right images to client device(s) of user(s), users other than user(s), other user(s), users other than other user(s), respectively.

## Claims

### Claim 1

A system comprising at least one server communicably coupled to client devices of N users and an imaging device, wherein the imaging device comprises a left camera and a right camera, each of the left camera and the right camera having an adjustable optical focus, wherein the at least one server is configured to:
receive, from the client devices of the N users, information indicative of respective optical depths at which the N users are gazing;
send, to the imaging device, information indicative of N optical depths; and in each cycle of N consecutive pairs of left images and right images, for M ranging from 1 to N:
receive, from the imaging device, an Mth left image captured by adjusting the optical focus of the left camera according to an Mth optical depth from amongst the N optical depths;
select the Mth left image for at least one user who is gazing at an optical depth that lies in a focusing distance range covered by adjusting the optical focus of the left camera according to the Mth optical depth;
apply respective extended depth-of-field (EDOF) corrections to the Mth left image, based on respective optical depths at which one or more users other than the at least one user are gazing, to generate respective EDOF-corrected Mth left images;
receive, from the imaging device, an Mth right image captured by adjusting the optical focus of the right camera according to another optical depth from amongst the N optical depths, wherein each of N consecutive right images in said cycle corresponds to a different optical depth from amongst the N optical depths;
select the Mth right image for at least one other user who is gazing at an optical depth that lies in another focusing distance range covered by adjusting the optical focus of the right camera according to the another optical depth;
apply respective EDOF corrections to the Mth right image, based on respective optical depths at which one or more users other than the at least one other user are gazing, to generate respective EDOF-corrected Mth right images; and
send the Mth left image to a client device of the at least one user, the EDOF-corrected Mth left images to respective ones of client devices of the one or more users other than the at least one user, the Mth right image to a client device of the at least one other user, and the EDOF-corrected Mth right images to respective ones of client devices of the one or more users other than the at least one other user.

### Claim 2 (depends on 1)

The system of claim 1, wherein the N optical depths comprise fixed optical depths.

### Claim 3 (depends on 1)

The system of claim 1, wherein the at least one server is configured to select an optical depth based on two different optical depths at which at least two users from amongst the N users are gazing, when the two different optical depths correspond to a same focussing distance range that is covered by adjusting the optical focus of a given camera according to the selected optical depth, wherein the N optical depths comprise the selected optical depth.

### Claim 4 (depends on 1)

The system of claim 1, wherein the N optical depths comprise the optical depths at which the N users are gazing, and wherein the Mth optical depth is an optical depth at which an Mth user is gazing, the at least one user being the Mth user, further wherein the another optical depth is an optical depth at which another user other than the Mth user is gazing.

### Claim 5 (depends on 1)

The system of claim 1, wherein N is one of: 2, 3 or 4.

### Claim 6 (depends on 1)

The system of claim 1, further comprising a given client device of a given user from amongst the N users, wherein a processor of the given client device is configured to:
receive consecutive images for a given eye of the given user in batches, wherein a given batch of consecutive images comprises at least two EDOF-corrected images and, optionally, an image selected for the given user or captured by adjusting the optical focus of a given camera according to an optical depth at which the given user is gazing;
detect, prior to displaying a given EDOF-corrected image, whether an image selected for the given user or captured by adjusting the optical focus of the given camera according to the optical depth at which the given user is gazing is available in a recently-received batch of consecutive images; and
when it is detected that said image is available in the recently-received batch of consecutive images, use any one of:
said image,
a fusion of said image and the given EDOF-corrected image,
a fusion of said image and at least two EDOF-corrected images available in the recently-received batch,
for display at the given client device, instead of the given EDOF-corrected image.

### Claim 7 (depends on 6)

The system of claim 6, wherein, when it is detected that said image is not available in the recently-received batch of consecutive images, the processor of the given client device is configured to use any one of:
the given EDOF-corrected image,
another EDOF-corrected image available in the recently-received batch,
a fusion of the given EDOF-corrected image and the another EDOF-corrected image,
a fusion of at least two EDOF-corrected images available in the recently-received batch,
for display at the given client device.

### Claim 8 (depends on 1)

The system of claim 1, wherein the at least one server is configured to:
receive, from the client devices of the N users, information indicative of respective gaze directions of the N users;
send, to the imaging device, the information indicative of the respective gaze directions of the N users; and
receive, from the imaging device in each cycle of N consecutive pairs of left images and right images, only one image as an entire frame and cropped gaze regions of remaining images in said cycle, wherein a gaze region of a given image is determined based on a gaze direction of a given user for which the given image is to be selected.

### Claim 9

A method comprising:
receiving, from client devices of N users, information indicative of respective optical depths at which the N users are gazing;
sending, to an imaging device, information indicative of N optical depths, wherein the imaging device comprises a left camera and a right camera, each of the left camera and the right camera having an adjustable optical focus; and
in each cycle of N consecutive pairs of left images and right images, for M ranging from 1 to N:
receiving, from the imaging device, an Mth left image captured by adjusting the optical focus of the left camera according to an Mth optical depth from amongst the N optical depths;
selecting the Mth left image for at least one user who is gazing at an optical depth that lies in a focusing distance range covered by adjusting the optical focus of the left camera according to the Mth optical depth;
applying respective extended depth-of-field (EDOF) corrections to the Mth left image, based on respective optical depths at which one or more users other than the at least one user are gazing, for generating respective EDOF-corrected Mth left images;
receiving, from the imaging device, an Mth right image captured by adjusting the optical focus of the right camera according to another optical depth from amongst the N optical depths, wherein each of N consecutive right images in said cycle corresponds to a different optical depth from amongst the N optical depths;
selecting the Mth right image for at least one other user who is gazing at an optical depth that lies in another focusing distance range covered by adjusting the optical focus of the right camera according to the another optical depth;
applying respective EDOF corrections to the Mth right image, based on respective optical depths at which one or more users other than the at least one other user are gazing, for generating respective EDOF-corrected Mth right images; and
sending the Mth left image to a client device of the at least one user, the EDOF-corrected Mth left images to respective ones of client devices of the one or more users other than the at least one user, the Mth right image to a client device of the at least one other user, and the EDOF-corrected Mth right images to respective ones of client devices of the one or more users other than the at least one other user.

### Claim 10 (depends on 9)

The method of claim 9, wherein the N optical depths comprise fixed optical depths.

### Claim 11 (depends on 9)

The method of claim 9, further comprising selecting an optical depth based on two different optical depths at which at least two users from amongst the N users are gazing, when the two different optical depths correspond to a same focussing distance range that is covered by adjusting the optical focus of a given camera according to the selected optical depth, wherein the N optical depths comprise the selected optical depth.

### Claim 12 (depends on 9)

The method of claim 9, wherein the N optical depths comprise the optical depths at which the N users are gazing, and wherein the Mth optical depth is an optical depth at which an Mth user is gazing, the at least one user being the Mth user, further wherein the another optical depth is an optical depth at which another user other than the Mth user is gazing.

### Claim 13 (depends on 9)

The method of claim 9, wherein N is one of: 2, 3 or 4.

### Claim 14 (depends on 9)

The method of claim 9, further comprising:
receiving, at a given client device of a given user from amongst the N users, consecutive images for a given eye of the given user in batches, wherein a given batch of consecutive images comprises at least two EDOF-corrected images and, optionally, an image selected for the given user or captured by adjusting the optical focus of a given camera according to an optical depth at which the given user is gazing;
detecting, prior to displaying a given EDOF-corrected image at the given client device, whether an image selected for the given user or captured by adjusting the optical focus of the given camera according to the optical depth at which the given user is gazing is available in a recently-received batch of consecutive images; and
when it is detected that said image is available in the recently-received batch of consecutive images, using any one of:
said image,
a fusion of said image and the given EDOF-corrected image,
a fusion of said image and at least two EDOF-corrected images available in the recently-received batch,
for display at the given client device, instead of the given EDOF-corrected image.

### Claim 15 (depends on 14)

The method of claim 14, wherein, when it is detected that said image is not available in the recently-received batch of consecutive images, the method comprises using any one of:
the given EDOF-corrected image,
another EDOF-corrected image available in the recently-received batch,
a fusion of the given EDOF-corrected image and the another EDOF-corrected image,
a fusion of at least two EDOF-corrected images available in the recently-received batch,
for display at the given client device.

### Claim 16 (depends on 9)

The method of claim 9, further comprising:
receiving, from the client devices of the N users, information indicative of respective gaze directions of the N users;
sending, to the imaging device, the information indicative of the respective gaze directions of the N users; and
receiving, from the imaging device in each cycle of N consecutive pairs of left images and right images, only one image as an entire frame and cropped gaze regions of remaining images in said cycle, wherein a gaze region of a given image is determined based on a gaze direction of a given user for which the given image is to be selected.