# US 2024/0005597 A1 · Modelling an environment using image data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005597 A1
- Document ID: US20240005597A1
- Status: Published application
- Kind: A1
- Application number: 18/469,392
- Filed: 2023-09-18
- Published: 2024-01-04
- Earliest claimed: 2021-03-19
- Applicants: Imperial College Innovations Limited
- Inventors: Edgar SUCAR; Shikun LIU; Joseph ORTIZ; Andrew DAVISON
- Web page: https://patentgrep.com/patent/US20240005597A1

## CPC

- G06T 7/579: from motion
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 7/11: Region-based segmentation
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method comprising obtaining image data captured by a camera device. The image data represents an observation of an environment. A two-dimensional representation of at least part of the environment is obtained using a model of the environment. The method includes evaluating a difference between the two-dimensional representation and at least part of the observation. The at least part of the observation is of the at least part of the environment represented by the two-dimensional representation. Based on the difference, a portion of the image data is selected for optimising the model. The portion of the image data represents a portion of the observation of the environment. The method comprises optimising the model using the portion of the image data.

## Claims

### Claim 1

A method, comprising:
obtaining image data captured by a camera device, the image data representing an observation of an environment;
obtaining a two-dimensional representation of at least part of the environment using a model of the environment;
evaluating a difference between the two-dimensional representation and at least part of the observation, wherein the at least part of the observation is of the at least part of the environment represented by the two-dimensional representation;
based on the difference, selecting a portion of the image data for optimising the model, wherein the portion of the image data represents a portion of the observation of the environment; and
optimising the model using the portion of the image data.

### Claim 2 (depends on 1)

The method of claim 1, comprising:
using the model of the environment to generate a three-dimensional representation of the at least part of the environment; and
obtaining the two-dimensional representation of the at least part of the environment using the three-dimensional representation.

### Claim 3 (depends on 1)

The method of claim 1, wherein the observation comprises at least one image, and selecting the portion of the image data comprises selecting a subset of pixels of the at least one image.

### Claim 4 (depends on 3)

The method of claim 3, wherein the at least one image comprises a plurality of frames, and selecting the portion of the image data comprises selecting a subset of pixels of at least two of the plurality of frames.

### Claim 5 (depends on 1)

The method of claim 1, wherein evaluating the difference comprises:
evaluating the difference between a first portion of the observation and a corresponding portion of the two-dimensional representation, thereby generating a first difference; and
evaluating the difference between a second portion of the observation and a corresponding portion of the two-dimensional representation, thereby generating a second difference less than the first difference, and
selecting the portion of the image data comprises:
selecting a first portion of the image data corresponding to the first portion of the observation, the first portion of the image data representing a first number of data points; and
selecting a second portion of the image data corresponding to the second portion of the observation, the second portion of the image data representing a second number of data points, smaller than the first number of data points.

### Claim 6 (depends on 1)

The method of claim 1, comprising:
evaluating a loss function based on the two-dimensional representation and the at least part of the observation of the environment, thereby generating a loss for optimising the model, wherein evaluating the loss function comprises evaluating the difference between the two-dimensional representation and the at least part of the observation; and
selecting the portion of the image data based on the loss.

### Claim 7 (depends on 6)

The method of claim 6, wherein the observation comprises at least one image and selecting the portion of the image data comprises selecting a subset of pixels of the at least one image with a distribution of the subset of pixels across the at least one image based on a loss probability distribution generated by evaluating the loss function for the at least part of the observation.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating the loss probability distribution comprises:
dividing the at least part of the observation into a plurality of regions;
evaluating the loss function for each of the plurality of regions, thereby generating a region loss for the each of the plurality of regions; and
generating the loss probability distribution based on the loss and the region loss of the each of the plurality of regions.

### Claim 9 (depends on 6)

The method of claim 6, wherein:
the observation comprises a first frame and a second frame;
evaluating the loss function comprises:
evaluating the loss function based on the first frame and a two-dimensional representation of the first frame, thereby generating a first loss;
evaluating the loss function based on the second frame and a two-dimensional representation of the second frame, thereby generating a second loss; and
selecting the portion of the image data based on the loss comprises, in response to determining that the first loss is greater than the second loss:
selecting a first number of pixels from the first frame; and
selecting a second number of pixels from the second frame, and wherein the first number of pixels is greater than the second number of pixels.

### Claim 10 (depends on 9)

The method of claim 9, comprising:
determining a total loss for a group of frames for optimising the model, the group of frames comprising the first frame and the second frame, the determining the total loss comprising evaluating the loss function based on the group of frames and a corresponding set of two-dimensional representations of the group of frames; and
determining the first number of pixels based on a contribution of the first loss to the total loss.

### Claim 11 (depends on 1)

The method of claim 1, wherein the observation comprises a plurality of frames, the evaluating the difference comprises evaluating the difference between a respective frame of the plurality of frames and a two-dimensional representation of the respective frame, and selecting the portion of the image data comprises selecting, based on the difference, a subset of the plurality of frames to be added to a set of frames for optimising the model.

### Claim 12 (depends on 11)

The method of claim 11, wherein the plurality of frames comprises a frame, and the method comprises:
obtaining a first set of pixels of the frame;
generating a second set of pixels of the two-dimensional representation of the frame, wherein evaluating the difference comprises evaluating the difference between each pixel in the first set of pixels and a corresponding pixel in the second set of pixels; and
determining a proportion of the first set of pixels for which the difference is lower than a first threshold, wherein the subset of the plurality of frames comprises the frame, and selecting the frame comprises selecting the frame in response to determining that the proportion is lower than a second threshold.

### Claim 13 (depends on 11)

The method of claim 11, comprising selecting a most recent frame captured by the camera device to be added to the set of frames.

### Claim 14 (depends on 1)

The method of claim 1, wherein:
the observation of the environment comprises a measured depth observation of the environment captured by the camera device;
the two-dimensional representation comprises a rendered depth representation of the at least part of the environment; and
the difference represents a geometric error based on the measured depth observation and the rendered depth representation.

### Claim 15 (depends on 1)

The method of claim 1, wherein:
the model of the environment comprises a neural network and obtaining the two-dimensional representation comprises applying a rendering process to an output of the neural network; and
optimising the model comprises optimising a set of parameters of the neural network, thereby generating an update to the set of parameters of the neural network.

### Claim 16 (depends on 15)

The method of claim 15, comprising:
obtaining a camera pose estimate for the observation of the environment, wherein the two-dimensional representation is generated based on the camera pose estimate; and
jointly optimising the camera pose estimate and the set of parameters of the neural network based on the difference; thereby generating:
an update to the camera pose estimate for the observation of the environment; and
the update to the set of parameters of the neural network.

### Claim 17

A non-transitory computer-readable storage medium comprising computer-executable instructions which, when executed by a processor, cause a computing device to perform operations comprising:
obtaining image data captured by a camera device, the image data representing an observation of an environment;
obtaining a two-dimensional representation of at least part of the environment using a model of the environment;
evaluating a difference between the two-dimensional representation and at least part of the observation, wherein the at least part of the observation is of the at least part of the environment represented by the two-dimensional representation;
based on the difference, selecting a portion of the image data for optimising the model, wherein the portion of the image data represents a portion of the observation of the environment; and
optimising the model using the portion of the image data.

### Claim 18

A system, comprising:
an image data interface to receive image data captured by a camera device, the image data representing an observation of an environment;
an image data portion selection engine configured to:
obtain a two-dimensional representation of at least part of the environment using a model of the environment;
evaluate a difference between the two-dimensional representation and at least part of the observation, wherein the at least part of the observation is of the at least part of the environment represented by the two-dimensional representation; and
based on the difference, select a portion of the image data for optimising the model, wherein the portion of the image data represents a portion of the observation of the environment; and
an optimiser configured to optimise the model using the portion of the image data.

### Claim 19 (depends on 18)

The system of claim 18, being a robotic device, further comprising:
a camera device configured to obtain image data representing an observation of an environment; and
one or more actuators to enable the robotic device to navigate around the environment.

### Claim 20 (depends on 19)

The system of claim 19, configured to control the one or more actuators to control navigation of the robotic device around the environment based on the model.
