# US 2024/0005598 A1 · Modelling an environment using image data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005598 A1
- Document ID: US20240005598A1
- Status: Published application
- Kind: A1
- Application number: 18/469,395
- Filed: 2023-09-18
- Published: 2024-01-04
- Earliest claimed: 2021-03-19
- Applicants: Imperial College Innovations Limited
- Inventors: Edgar SUCAR; Shikun LIU; Joseph ORTIZ; Andrew DAVISON
- Web page: https://patentgrep.com/patent/US20240005598A1

## CPC

- G06T 7/579: from motion
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

A method comprising obtaining image data captured by a camera device. The image data represents an observation of at least part of an environment. A camera pose estimate associated with the observation is obtained. Rendered image data is generated based on the camera pose estimate and a model of the environment for generating a three-dimensional representation of the at least part of the environment. The rendered image data is representative of at least one rendered image portion corresponding to the at least part of the environment. The method includes evaluating a loss function based on the image data and the rendered image data, thereby generating a loss. At least the camera pose estimate and the model are jointly optimised based on the loss, thereby generating an update to the camera pose estimate, and an update to the model.

## Claims

### Claim 1

A method comprising:
obtaining image data captured by a camera device, the image data representing an observation of at least part of an environment;
obtaining a camera pose estimate associated with the observation;
generating rendered image data based on the camera pose estimate and a model of the environment, wherein the model is for generating a three-dimensional representation of the at least part of the environment, wherein the rendered image data is representative of at least one rendered image portion corresponding to the at least part of the environment;
evaluating a loss function based on the image data and the rendered image data, thereby generating a loss; and
jointly optimising at least the camera pose estimate and the model based on the loss, thereby generating:
an update to the camera pose estimate; and
an update to the model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the model is a neural network and the update to the model is an update to a set of parameters of the neural network.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the rendered image data comprises:
generating the three-dimensional representation using the model; and
performing a rendering process using the three-dimensional representation, wherein the rendering process is differentiable with respect to the camera pose estimate and a set of parameters of the model.

### Claim 4 (depends on 1)

The method of claim 1, comprising:
evaluating a first gradient of the at least one rendered image portion with respect to the camera pose estimate, thereby generating a first gradient value; and
evaluating a second gradient of the at least one rendered image portion with respect to a set of parameters of the model, thereby generating a second gradient value, wherein jointly optimising the camera pose estimate and the model comprises applying a gradient-based optimisation algorithm using the first gradient value and the second gradient value.

### Claim 5 (depends on 1)

The method of claim 1, wherein the model is configured to map a spatial coordinate corresponding to a location within the environment to:
a photometric value associated with the location within the environment; and
a volume density value for deriving a depth value associated with the location within the environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the image data comprises photometric data comprising at least one measured photometric image portion;
the at least one rendered image portion comprises at least one rendered photometric image portion; and
the loss function comprises a photometric error based on the at least one measured photometric image portion and the at least one rendered photometric image portion.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the image data comprises depth data comprising at least one measured depth image portion;
the at least one rendered image portion comprises at least one rendered depth image portion; and
the loss function comprises a geometric error based on the at least one measured depth image portion and the at least one rendered depth image portion.

### Claim 8 (depends on 7)

The method of claim 7,
wherein the depth data comprises a plurality of measured depth image portions, the at least one rendered image portion comprises a plurality of rendered depth image portions each corresponding to a respective one of the plurality of measured depth image portions, the geometric error comprises a plurality of geometric error terms, each corresponding to a different one of the plurality of measured depth image portions, and the method comprises reducing a contribution to the geometric error of a first geometric error term associated with a first one of the plurality of measured depth image portions relative to a second geometric error term associated with a second one of the plurality of measured depth image portions, based on at least one of: a first measure of uncertainty associated with the first one of the plurality of measured depth image portions or a second measure of uncertainty associated with the second one of the plurality of measured depth image portions.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the rendered image data comprises:
applying ray-tracing to identify a set of spatial coordinates along a ray, wherein the ray is determined based on the camera pose estimate and a pixel coordinate of a pixel of the at least one rendered image portion; and
processing the set of spatial coordinates using the model, thereby generating a set of photometric values and a set of volume density values, each associated with a respective one of the set of spatial coordinates;
combining the set of photometric values to generate a pixel photometric value associated with the pixel; and
combining the set of volume density values to generate a pixel depth value associated with the pixel.

### Claim 10 (depends on 9)

The method of claim 9, wherein the set of spatial coordinates is a first set of spatial coordinates, the set of photometric values is a first set of set of photometric values, the set of volume density values is a first set of volume density values, and applying the ray-tracing comprises applying the ray-tracing to identify a second set of spatial coordinates along the ray, wherein the second set of spatial coordinates are determined based on a probability distribution which is a function of the first set of volume density values and a distance between neighbouring spatial coordinates in the first set of spatial coordinates, and the method comprises:
processing the second set of spatial coordinates using the model, thereby generating a second set of photometric values and a second set of volume density values;
combining the first set of photometric values and the second set of photometric values to generate the pixel photometric value; and
combining the first set of volume density values and the second set of volume density values to generate the pixel depth value.

### Claim 11 (depends on 1)

The method of claim 1, wherein the observation is a first observation, the camera pose estimate is a first camera pose estimate and the method comprises, after jointly optimising the camera pose estimate and the model:
obtaining a second camera pose estimate associated with a second observation of the environment subsequent to the first observation; and
optimising the second camera pose estimate based on the second observation of the environment and the model, thereby generating an update to the second camera pose estimate.

### Claim 12 (depends on 1)

The method of claim 1, wherein the observation comprises a first frame and a second frame, and the rendered image data is representative of at least one rendered image portion corresponding to the first frame and at least one rendered image portion corresponding to the second frame, the camera pose estimate is a first frame camera pose estimate associated with the first frame, evaluating the loss function generates a first loss associated with the first frame and a second loss associated with the second frame, and the method comprises:
obtaining a second frame camera pose estimate corresponding to the second frame,
wherein jointly optimising at least the camera pose estimate and the model based on the loss comprises jointly optimising the first frame camera pose estimate, the second frame camera pose estimate and the model based on the first loss and second loss, thereby generating:
the update to the first frame camera pose estimate;
an update to the second frame camera pose estimate; and
the update to the model.

### Claim 13 (depends on 1)

The method of claim 1, wherein the image data is first image data, the observation is an observation of at least a first part of the environment, and the method comprises obtaining second image data captured by the camera device, the second image data representing an observation of at least a second part of an environment, wherein generating the rendered image data comprises generating the rendered image data for the first part of the environment without generating rendered image data for the second part of the environment.

### Claim 14 (depends on 1)

The method of claim 1, wherein the image data is first image data, the observation is an observation of at least a first part of the environment, and the method comprises obtaining second image data captured by the camera device, the second image data representing an observation of at least a second part of the environment, wherein the method comprises:
determining that further rendered image data is to be generated for the second part of the environment for further jointly optimising at least the camera pose estimate and the model; and
generating the further rendered image data, based on the camera pose estimate and the model, for further jointly optimising at least the camera pose estimate and the model,
wherein determining that the further rendered image data is to be generated for the second part of the environment comprises determining that the further rendered image data is to be generated based on the loss.

### Claim 15 (depends on 14)

The method of claim 14, wherein determining that the further rendered image data is to be generated for the second part of the environment comprises:
based on the loss, generating a loss probability distribution for a region of the environment comprising the first part and the second part; and
based on the loss probability distribution, selecting a set of pixels, corresponding to the second image data, for which the further rendered image data is to be generated.

### Claim 16 (depends on 1)

The method of claim 1, wherein the observation comprises at least a portion of at least one frame previously captured by the camera device, and the method comprises:
selecting the at least one frame from a plurality of frames previously captured by the camera device based on a difference between at least a portion of a respective frame of the plurality of frames and at least a corresponding portion of a respective rendered frame, rendered based on the camera pose estimate and the model.

### Claim 17

A non-transitory computer-readable storage medium comprising computer-executable instructions which, when executed by a processor, cause a computing device to perform operations comprising:
obtaining image data captured by a camera device, the image data representing an observation of at least part of an environment;
obtaining a camera pose estimate associated with the observation;
generating rendered image data based on the camera pose estimate and a model of the environment, wherein the model is for generating a three-dimensional representation of the at least part of the environment, wherein the rendered image data is representative of at least one rendered image portion corresponding to the at least part of the environment;
evaluating a loss function based on the image data and the rendered image data, thereby generating a loss; and
jointly optimising at least the camera pose estimate and the model based on the loss, thereby generating:
an update to the camera pose estimate; and
an update to the model.

### Claim 18

A system, comprising:
an image data interface to receive image data captured by a camera device, the image data representing an observation of at least part of an environment;
a rendering engine configured to:
obtain a camera pose estimate associated with the observation;
generate rendered image data based on the camera pose estimate and a model of the environment, wherein the model is for generating a three-dimensional representation of the at least part of the environment, wherein the rendered image data is representative of at least one rendered image portion corresponding to the at least part of the environment; and
evaluate a loss function based on the image data and the rendered image data, thereby generating a loss; and
an optimiser configured to:
jointly optimise at least the camera pose estimate and the model based on the loss, thereby generating:
an update to the camera pose estimate; and
an update to the model.

### Claim 19 (depends on 18)

The system of claim 18, being a robotic device, the system further comprising:
a camera device configured to obtain image data representing an observation of at least part of an environment; and
one or more actuators to enable the robotic device to navigate around the environment.

### Claim 20 (depends on 19)

The system of claim 19, configured to control the one or more actuators to control navigation of the robotic device around the environment based on the model.
