# US 12,118,668 B2 · Scene representation using image processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,668 B2 - Document ID: US12118668B2 - Status: Granted - Kind: B2 - Application number: 17/687,053 - Filed: 2022-03-04 - Published: 2022-06-16 - Granted: 2024-10-15 - Earliest claimed: 2019-09-06 - Assignees: Imperial College Innovations Limited - Applicants: Imperial College of Science, Technology and Medicine - Inventors: Michael Bloesch; Tristan William Laidlow; Ronald Clark; Andrew Davison; Stefan Leutenegger - Web page: https://patentgrep.com/patent/US12118668B2 ## CPC - G06T 7/55: from multiple images - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 7/593: from stereo images - G06T 7/73: using feature-based methods - B25J 9/1602: {characterised by the control system, structure, architecture} - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) ## Abstract An image processing system configured to obtain a mesh representation of a scene, wherein the mesh representation comprises a plurality of polygons defined by respective vertices associated with an in-plane position, the in-plane position being in a plane comprising a first dimension and a second dimension, and the vertices having an associated vertex depth value in a third dimension different from the first dimension and the second dimension. The image processing system comprises an in-plane position estimation network configured to process image data representative of an image of the scene to estimate the in-plane positions associated with respective vertices of the mesh representation. The image processing system further comprises a depth estimation engine configured to process the in-plane positions and the image data to estimate the associated vertex depth values for the respective vertices of the mesh representation. ## Claims ### Claim 1 A system configured to obtain a mesh representation of a scene, wherein the mesh representation comprises a plurality of polygons defined by respective vertices associated with in-plane positions, the in-plane positions being in a plane comprising a first dimension and a second dimension, and the vertices having an associated vertex depth value in a third dimension different from the first dimension and the second dimension, and the system comprises: an in-plane position estimation network configured to process image data representative of an image of the scene to estimate the in-plane positions associated with respective vertices of the mesh representation, wherein the in-plane position estimation network comprises a neural network architecture comprising parameters obtained during training using a loss function based on a comparison between a predicted depth map and a ground-truth depth map, wherein the predicted depth map is computed from an output of the neural network architecture during the training; and a depth estimation engine configured to process the in-plane positions and the image data to estimate the associated vertex depth values for the respective vertices of the mesh representation. ### Claim 2 (depends on 1) The system according to claim 1, wherein the predicted depth map is computed from the output of the neural network architecture during the training using a transformation function for transforming vertex depth values associated with respective vertices of the mesh representation to pixel depth values associated with respective regions of the scene, wherein the transformation function depends on the output of the neural network architecture. ### Claim 3 (depends on 2) The system according to claim 2, wherein the transformation function is obtained by rendering the predicted depth map. ### Claim 4 (depends on 2) The system according to claim 2, wherein the transformation function (Original) comprises barycentric weights associated with respective polygons of the plurality of polygons. ### Claim 5 (depends on 1) The system according to claim 1, wherein the neural network architecture comprises: at least one convolutional layer to detect image features of the image; and at least one fully connected layer to process image features for a plurality of patches of the image to estimate the in-plane positions. ### Claim 6 (depends on 1) The system according to claim 1, wherein the associated vertex depth values for the respective vertices of the mesh representation are inverse depth values, the predicted depth map represents predicted inverse depth values associated with respective spatial regions of the scene and the ground-truth depth map represents ground-truth inverse depth values associated with the respective spatial regions. ### Claim 7 (depends on 1) The system according to claim 1, wherein each polygon of the plurality of polygons is a triangle. ### Claim 8 (depends on 1) The system according to claim 1, wherein the in-plane position estimation network is configured to estimate the in-plane positions such that the plurality of polygons satisfy a size condition, wherein optionally the size condition is satisfied by polygons with a length of less than or equal to a predetermined proportion of a length of the image, in at least one of the first and second dimensions. ### Claim 9 (depends on 1) The system according to claim 1, wherein the in-plane position estimation network is configured to estimate the in-plane positions such that the plurality of polygons are non-overlapping. ### Claim 10 (depends on 1) The system according to claim 1, wherein the depth estimation engine is configured to estimate the associated vertex depth values for the respective vertices of the mesh representation by optimising a cost function comprising a term dependent on the image data and the in-plane positions. ### Claim 11 (depends on 1) The system according to claim 1, further comprising: one or more actuators for interacting with a surrounding three-dimensional environment, wherein at least a portion of the surrounding three-dimensional environment is shown in the scene; and an interaction engine comprising at least one processor to control the one or more actuators. ### Claim 12 A system configured to obtain a mesh representation of a scene, wherein the mesh representation comprises a plurality of polygons defined by respective vertices associated with in-plane positions, the in-plane positions being in a plane comprising a first dimension and a second dimension, and the vertices having an associated depth value in a third dimension different from the first dimension and the second dimension, and the system comprises: an in-plane position estimation network configured to process image data representative of an image of the scene to estimate the in-plane positions associated with respective vertices of the mesh representation; and a depth estimation engine configured to: receive the in-plane positions and the image data; and optimise a cost function comprising a term dependent on the image data and the in-plane positions, to estimate the associated vertex depth values for the respective vertices of the mesh representation. ### Claim 13 (depends on 12) The system according to claim 12, wherein the image data comprises first image data representative of a first image of the scene and second image data representative of a second image of the scene which at least partly overlaps the first image, the term is a first term, and the cost function comprises a second term dependent on the first image data, the second image data, first pose data representative of a first pose of an image capture device during capture of the first image, and second pose data representative of a second pose of the image capture device during capture of the second image. ### Claim 14 (depends on 13) The system according to claim 13, wherein the depth estimation engine is configured to optimise the cost function to jointly estimate the associated vertex depth values and the first pose of the image capture device. ### Claim 15 (depends on 13) The system according to claim 13, wherein the term is a first term, and the depth estimation engine comprises a depth neural network architecture to predict at least one residual term based on the in-plane positions and the image data, wherein the at least one residual term comprises the first term and depends on the associated vertex depth values, and the cost function comprises the at least one residual term. ### Claim 16 (depends on 15) The system according to claim 15, wherein the depth neural network architecture comprises parameters obtained during a training process comprising: obtaining optimised values of the vertex depth values that optimise the cost function, given fixed values of the parameters of the depth neural network architecture; and obtaining optimised values of the parameters that optimise a loss function based on a comparison between a predicted depth map obtained using the optimised values of the vertex depth values and a ground-truth depth map. ### Claim 17 (depends on 16) The system according to claim 16, wherein: obtaining the optimised values of the vertex depth values comprises obtaining a respective value of the vertex depth values for each of a plurality of iterations; and obtaining the optimised values of the parameters comprises computing a loss value of the loss function for each of the respective value of the vertex depth values and computing a combined loss based on the loss values. ### Claim 18 (depends on 15) The system according to claim 15, wherein the depth neural network architecture is configured to: predict, using the in-plane positions, vertex residual terms associated with the respective vertices of the mesh representation; and obtain a term of the at least one residual term from the vertex residual terms. ### Claim 19 (depends on 15) The system according to claim 15, wherein: the term is a first term; the image data comprises first image data representative of a first image of the scene and second image data representative of a second image of the scene which at least partly overlaps the first image; and the depth neural network architecture comprises: a first neural network architecture to predict, using the first image data, the first term; and a second neural network architecture to predict, using the first image data and the second image data, a second term dependent on the first image data, the second image data, a first pose of an image capture device during capture of the first image, and a second pose of the image capture device during capture of the second image. ### Claim 20 (depends on 19) The system according to claim 19, wherein the depth estimation engine is configured to use the second neural network architecture to predict, using the second image data and further image data representative of at least one further image of the scene, at least one further term dependent on the second image data, the further image data, the second pose of the image capture device during capture of the second image, and at least one further pose of the image capture device during capture of the at least one further image. ### Claim 21 (depends on 19) The system according to claim 19, wherein the second neural network architecture is configured to generate first feature data representative of first image features of the first image and second feature data representative of second image features of the second image and the second term is based on a weighted comparison between the first image features for regions of the first image and the second image features for corresponding regions of the second image. ### Claim 22 (depends on 21) The system according to claim 21, wherein the depth neural network architecture comprises a third neural network architecture configured to process the first feature data and the second feature data to obtain weight data representative of weights for computing the weighted comparison. ### Claim 23 (depends on 12) The system according to claim 12, wherein the in-plane position estimation network comprises an in-plane neural network architecture comprising parameters obtained during training using a loss function based on a comparison between a predicted depth map and a ground-truth depth map, wherein the predicted depth map is computed from an output of the in-plane neural network architecture during the training. ### Claim 24 (depends on 12) The system according to claim 12, further comprising: one or more actuators for interacting with a surrounding three-dimensional environment, wherein at least a portion of the surrounding three-dimensional environment is shown in the scene; and an interaction engine comprising at least one processor to control the one or more actuators. ### Claim 25 A method of training an in-plane position estimation network, the method comprising, for a training iteration: obtaining a set of training samples from a set of training data, wherein each of the training samples comprises image data representative of a scene and a ground-truth depth map of the scene; for each given training sample in the set of training samples: processing the image data for the given training sample using a neural network architecture of the in-plane position estimation network to obtain estimated in-plane positions associated with respective vertices of polygons of a mesh representation of the scene, wherein the estimated in-plane positions are in a plane comprising a first dimension and a second dimension; using the estimated in-plane positions to obtain a predicted depth map of the scene, wherein the predicted depth map comprises depth values associated with the respective vertices in a third dimension different from the first and second dimensions; and computing a loss function based on a comparison between the predicted depth map and the ground-truth depth map for the given training sample, wherein the training iteration is repeated to optimise the loss function to determine parameter values for the neural network architecture. ### Claim 26 (depends on 25) The method according to claim 25, comprising: processing the image data for the given training sample using at least one convolutional layer of the neural network architecture to obtain feature data representative of image features for the given training sample; and processing the feature data for a plurality of image patches of the given training sample, corresponding to regions of the scene associated with respective vertices of the mesh representation with initial in-plane positions, using a fully-connected layer of the neural network architecture to compute a perturbation to the initial in-plane positions to obtain the estimated in-plane positions, wherein, prior to the perturbation, the vertices of the mesh representation are regularly spaced from each other in the plane.