# US 12,112,457 B2 · Selective extended depth-of-field correction for image reconstruction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,457 B2 - Document ID: US12112457B2 - Status: Granted - Kind: B2 - Application number: 17/991,343 - Filed: 2022-11-21 - Published: 2024-05-23 - Granted: 2024-10-08 - Earliest claimed: 2022-11-21 - Assignees: Varjo Technologies Oy - Applicants: Varjo Technologies Oy - Inventors: Mikko Ollila; Mikko Strandborg - Web page: https://patentgrep.com/patent/US12112457B2 ## CPC - G06T 5/73: Deblurring; Sharpening - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/60: using machine learning, e.g. neural networks - G06T 7/55: from multiple images - G06T 15/20: Perspective computation - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 2200/21: involving computational photography - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system including server(s) and data repository, wherein server(s) is/are configured to receive images of real-world environment captured using camera(s), corresponding depth maps, and at least one of: pose information, relative pose information; generate three-dimensional model (3D) of real-world environment; store 3D model; utilise 3D model to generate output image from perspective of new pose; determine whether extended depth-of-field (EDOF) correction is required to be applied to any one of: at least one of images captured by camera(s) representing given object(s), 3D model, output image, based on whether optical focus of camera(s) was adjusted according to optical depth of given object from given pose of camera; and when it is determined that EDOF correction is required to be applied, apply EDOF correction to at least portion of any one of: at least one of images captured by camera(s), 3D model, output image. ## Claims ### Claim 1 A system comprising at least one server and a data repository communicably coupled to the at least one server, wherein the at least one server is configured to: receive images of a real-world environment that are captured using at least one camera, corresponding depth maps, and at least one of: pose information indicative of corresponding poses of the at least one camera from which the images and the depth maps are captured, relative pose information indicative of a relative pose of a given camera with respect to a pose of at least one other given camera during capturing of the images, wherein the at least one camera comprises the given camera and the at least one other given camera; process the images to generate a three-dimensional model of the real-world environment, based on the corresponding depth maps and the at least one of: the pose information, the relative pose information; store the three-dimensional model of the real-world environment at the data repository; utilise the three-dimensional model of the real-world environment to generate an output image from a perspective of a new pose, the output image representing at least one given object present in the real-world environment; determine whether an extended depth-of-field correction is required to be applied to any one of: at least one of the images captured by the at least one camera representing the at least one given object, the three-dimensional model of the real-world environment, the output image, based on whether an optical focus of the at least one camera was adjusted for capturing the at least one of the images according to an optical depth of the at least one given object from a given pose of the at least one camera from which the at least one of the images was captured; and when it is determined that an extended depth-of-field correction is required to be applied, apply the extended depth-of-field correction to at least a portion of any one of: the at least one of the images captured by the at least one camera, the three-dimensional model of the real-world environment, the output image, wherein said portion to which the extended depth-of-field correction is applied represents the at least one given object or at least a part of the at least one given object. ### Claim 2 (depends on 1) The system of claim 1, wherein the at least one server is configured to: for a user viewing from the new pose, receive information indicative of gaze directions of the user's eyes; determine, based on the new pose and a convergence of the gaze directions, a gaze region in the real-world environment at which the user is gazing; select the at least one given object from amongst a plurality of objects present in the real-world environment, based on the gaze region. ### Claim 3 (depends on 2) The system of claim 2, wherein the at least one server is configured to determine whether there is at least one sub-region within the gaze region that is out-of-focus in the images captured by the at least one camera, wherein the at least one sub-region is determined to be out-of-focus when an optical depth of at least a part of the at least one given object present in the at least one sub-region from the given pose of the at least one camera lies outside a focussing distance range of a focus step that was used to capture the at least one of the images from the given pose of the at least one camera, wherein when there is at least one sub-region within the gaze region that is out-of-focus in the images captured by the at least one camera, it is determined that the extended depth-of-field correction is required to be applied, further wherein said portion to which the extended depth-of-field correction is applied represents at least the part of the at least one given object. ### Claim 4 (depends on 1) The system of claim 1, wherein the extended depth-of-field correction is applied to at least one of: colour data, depth data, luminance data corresponding to said portion. ### Claim 5 (depends on 1) The system of claim 1, wherein the at least one server is configured to store any one of: the at least one of the images, the three-dimensional model of the real-world environment, the output image, at the data repository after applying the extended depth-of-field correction. ### Claim 6 (depends on 1) The system of claim 1, wherein the extended depth-of-field correction is applied by utilising a neural network that corrects point spread functions specific to at least one optical element of the at least one camera. ### Claim 7 (depends on 6) The system of claim 6, wherein the neural network is trained using at least one of: images having a specific type of blur or at least one specific focal plane, in-focus images as ground truth material and corresponding blurred images as training material, along with corresponding depth information. ### Claim 8 (depends on 6) The system of claim 6, wherein the at least one server is configured to: determine the optical depth of the at least one given object from the given pose of the at least one camera using which the at least one of the images representing the at least one given object was captured; determine whether the optical depth of the at least one given object lies outside a focussing distance range of a focus step that was used to capture the at least one of the images from the given pose of the at least one camera; when the optical depth of the at least one given object lies outside the focussing distance range of the focus step that was used, determine a different focussing step that corresponds to the optical depth of the at least one given object; and select the neural network, from amongst a plurality of neural networks, based on the different focussing step that corresponds to the optical depth of the at least one given object. ### Claim 9 (depends on 1) The system of claim 1, wherein the extended depth-of-field correction is applied by utilising at least one of: defocus map estimation, blind deblurring deconvolution, non-blind deblurring deconvolution. ### Claim 10 A method comprising: receiving images of a real-world environment that are captured using at least one camera, corresponding depth maps, and at least one of: pose information indicative of corresponding poses of the at least one camera from which the images and the depth maps are captured, relative pose information indicative of a relative pose of a given camera with respect to a pose of at least one other given camera during capturing of the images, wherein the at least one camera comprises the given camera and the at least one other given camera; processing the images for generating a three-dimensional model of the real-world environment, based on the corresponding depth maps and the at least one of: the pose information, the relative pose information; storing the three-dimensional model of the real-world environment at a data repository; utilising the three-dimensional model of the real-world environment to generate an output image from a perspective of a new pose, the output image representing at least one given object present in the real-world environment; determining whether an extended depth-of-field correction is required to be applied to any one of: at least one of the images captured by the at least one camera representing the at least one given object, the three-dimensional model of the real-world environment, the output image, based on whether an optical focus of the at least one camera was adjusted for capturing the at least one of the images according to an optical depth of the at least one given object from a given pose of the at least one camera from which the at least one of the images was captured; and when it is determined that an extended depth-of-field correction is required to be applied, applying the extended depth-of-field correction to at least a portion of any one of: the at least one of the images captured by the at least one camera, the three-dimensional model of the real-world environment, the output image, wherein said portion to which the extended depth-of-field correction is applied represents the at least one given object or at least a part of the at least one given object. ### Claim 11 (depends on 10) The method of claim 10, further comprising: for a user viewing from the new pose, receiving information indicative of gaze directions of the user's eyes; determining, based on the new pose and a convergence of the gaze directions, a gaze region in the real-world environment at which the user is gazing; selecting the at least one given object from amongst a plurality of objects present in the real-world environment, based on the gaze region. ### Claim 12 (depends on 11) The method of claim 11, further comprising determining whether there is at least one sub-region within the gaze region that is out-of-focus in the images captured by the at least one camera, wherein the at least one sub-region is determined to be out-of-focus when an optical depth of at least a part of the at least one given object present in the at least one sub-region from the given pose of the at least one camera lies outside a focussing distance range of a focus step that was used to capture the at least one of the images from the given pose of the at least one camera, wherein when there is at least one sub-region within the gaze region that is out-of-focus in the images captured by the at least one camera, it is determined that the extended depth-of-field correction is required to be applied, further wherein said portion to which the extended depth-of-field correction is applied represents at least the part of the at least one given object. ### Claim 13 (depends on 10) The method of claim 10, wherein the extended depth-of-field correction is applied to at least one of: colour data, depth data, luminance data corresponding to said portion. ### Claim 14 (depends on 10) The method of claim 10, further comprising storing any one of: the at least one of the images, the three-dimensional model of the real-world environment, the output image, at the data repository after applying the extended depth-of-field correction. ### Claim 15 (depends on 10) The method of claim 10, wherein the step of applying the extended depth-of-field correction comprises utilising a neural network that corrects point spread functions specific to at least one optical element of the at least one camera. ### Claim 16 (depends on 15) The method of claim 15, wherein the neural network is trained using at least one of: images having a specific type of blur or at least one specific focal plane, in-focus images as ground truth material and corresponding blurred images as training material, along with corresponding depth information. ### Claim 17 (depends on 15) The method of claim 15, further comprising: determining the optical depth of the at least one given object from the given pose of the at least one camera using which the at least one of the images representing the at least one given object was captured; determining whether the optical depth of the at least one given object lies outside a focussing distance range of a focus step that was used to capture the at least one of the images from the given pose of the at least one camera; when the optical depth of the at least one given object lies outside the focussing distance range of the focus step that was used, determining a different focussing step that corresponds to the optical depth of the at least one given object; and selecting the neural network, from amongst a plurality of neural networks, based on the different focussing step that corresponds to the optical depth of the at least one given object. ### Claim 18 (depends on 10) The method of claim 10, wherein the step of applying the extended depth-of-field correction comprises utilising at least one of: defocus map estimation, blind deblurring deconvolution, non-blind deblurring deconvolution.