# US 12,182,935 B2 · Method and apparatus for training a neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,935 B2 - Document ID: US12182935B2 - Status: Granted - Kind: B2 - Application number: 18/008,418 - Filed: 2021-06-03 - Published: 2023-07-20 - Granted: 2024-12-31 - Earliest claimed: 2020-06-05 - Assignees: Condense Reality Ltd. - Applicants: Condense Reality Ltd. - Inventors: Nicholas Fellingham; Oliver Moolan-Feroze - Web page: https://patentgrep.com/patent/US12182935B2 ## CPC - G06T 7/55: from multiple images - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/60: using machine learning, e.g. neural networks - G06T 7/593: from stereo images - G06T 2200/04: involving 3D image data - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A first aspect of the invention provides a method of training a neural network for capturing volumetric video, comprising: generating a 3D model of a scene; using the 3D model to generate a high fidelity depth map; capturing a perceived depth map of the scene, having a field of view that is aligned with a field of view of the high fidelity depth map; and training the neural network based on the high fidelity depth map and the perceived depth map, wherein the high fidelity depth map has a higher fidelity to the scene than the perceived depth map has. ## Claims ### Claim 1 A method of training a neural network for capturing volumetric video, comprising: generating a 3D model of a scene; using the 3D model to generate a high fidelity depth map; capturing a perceived depth map of the scene, having a field of view that is aligned with a field of view of the high fidelity depth map; and training the neural network based on the high fidelity depth map and the perceived depth map, wherein the high fidelity depth map has a higher fidelity to the scene than the perceived depth map has. ### Claim 2 (depends on 1) The method of claim 1, wherein training the neural network based on the high fidelity depth map and the perceived depth map comprises training the neural network to increase the fidelity to the scene of captured depth maps. ### Claim 3 (depends on 2) The method of claim 2, wherein training the neural network to increase the fidelity to the scene of captured depth maps comprises training the neural network to recognise that the perceived depth map is indicative of the high fidelity depth map. ### Claim 4 (depends on 2) The method of claim 2, wherein training the neural network to increase the fidelity to the scene of captured depth maps comprises training the neural network to increase, in less than 33 ms, the fidelity to the scene of captured depth maps. ### Claim 5 (depends on 1) The method of claim 1, further comprising: capturing a first colour map of the scene, having a field of view that is aligned with the field of view of the high fidelity depth map; and training the neural network based on the high fidelity depth map and the first colour map. ### Claim 6 (depends on 5) The method of claim 5, wherein training the neural network based on the high fidelity depth map and the first colour map comprises training the neural network to recognise that a colour feature of the first colour map is indicative of a depth feature of the high fidelity depth map. ### Claim 7 (depends on 1) The method of claim 1, further comprising: using the 3D model to generate a high fidelity colour map; capturing a perceived colour map of the scene, having a field of view that is aligned with a field of view of the high fidelity colour map; and training the neural network based on the high fidelity colour map and the perceived colour map, wherein: the high fidelity colour map has a higher fidelity to the scene than the perceived colour map has. ### Claim 8 (depends on 7) The method of claim 7, wherein training the neural network based on the high fidelity colour map and the perceived colour map comprises training the neural network to increase the fidelity to the scene of captured colour maps. ### Claim 9 (depends on 8) The method of claim 8, wherein training the neural network to increase the fidelity to the scene of captured colour maps comprises training the neural network to recognise that the perceived colour map is indicative of the high fidelity colour map. ### Claim 10 (depends on 1) The method of claim 1, wherein generating a 3D model of a scene comprises: capturing a plurality of depth maps of the scene; and generating a 3D model of the scene based on the plurality of depth maps. ### Claim 11 (depends on 1) The method of claim 1, further comprising repeating the method using at least one further depth map or scene. ### Claim 12 (depends on 1) A neural network that has been trained using the method of claim 1. ### Claim 13 (depends on 1) A method of capturing volumetric video, comprising capturing volumetric video using a neural network that has been trained using the method of claim 1. ### Claim 14 Apparatus for training a neural network for capturing volumetric video, comprising: a neural network; a plurality of cameras configured to generate a 3D model of a scene; and a processor configured to use the 3D model of the scene to generate a high fidelity depth map; wherein: a first subset of the plurality of cameras is further configured to capture a perceived depth map of the scene, the perceived depth map having a field of view that is aligned with a field of view of the high fidelity depth map; the neural network is configured to receive, as training inputs, the high fidelity depth map and the perceived depth map; and the high fidelity depth map has a higher fidelity to the scene than the perceived depth map has. ### Claim 15 (depends on 14) Apparatus according to claim 14, wherein when capturing the perceived depth map, the first subset of the plurality of cameras are co-located with cameras of the plurality of cameras when generating the 3D model. ### Claim 16 (depends on 14) Apparatus according to claim 14, wherein the plurality of cameras comprises stereoscopic cameras. ### Claim 17 (depends on 14) Apparatus according to any of claim 14, wherein the plurality of cameras comprises time of flight cameras. ### Claim 18 Apparatus for training a neural network for capturing volumetric video, comprising: a neural network; and a computer program configured to: generate a 3D model of a scene; use the 3D model to generate a high fidelity depth map; and generate a perceived depth map of the scene, the perceived depth map having a field of view that is aligned with a field of view of the high fidelity depth map, wherein: the neural network is configured to receive, as training inputs, the high fidelity depth map and the perceived depth map; and the high fidelity depth map has a higher fidelity to the scene than the perceived depth map has. ### Claim 19 Apparatus for training a neural network for capturing volumetric video, comprising: a neural network; a first plurality of cameras configured to generate a 3D model of a scene; a processor configured to use the 3D model of the scene to generate a high fidelity depth map; and a second camera, not included in the first plurality of cameras, configured to capture a perceived depth map of the scene, the perceived depth map having a field of view that is aligned with a field of view of the high fidelity depth map; wherein: the neural network is configured to receive, as training inputs, the high fidelity depth map and the perceived depth map; and the high fidelity depth map has a higher fidelity to the scene than the perceived depth map has. ### Claim 20 (depends on 19) Apparatus according to claim 19, wherein the second camera is configured to be positioned farther from the scene than the first plurality of cameras are from the scene.