# US 10,311,378 B2 · Depth from time-of-flight using machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,378 B2 - Document ID: US10311378B2 - Status: Granted - Kind: B2 - Application number: 15/672,261 - Filed: 2017-08-08 - Published: 2018-05-10 - Granted: 2019-06-04 - Earliest claimed: 2016-03-13 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Sebastian Nowozin; Amit Adam; Shai Mazor; Omer Yair - Web page: https://patentgrep.com/patent/US10311378B2 ## CPC - G06N 20/20: Ensemble learning - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 20/00: Machine learning - G06T 7/50: Depth or shape recovery - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06V 10/10: Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60) - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G01S 17/36: with phase comparison between the received signal and the contemporaneously transmitted signal - G01S 17/66: Tracking systems using electromagnetic waves other than radio waves - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A depth detection apparatus is described which has a memory storing raw time-of-flight sensor data received from a time-of-flight sensor. The depth detection apparatus also has a trained machine learning component having been trained using training data pairs. A training data pair comprises at least one simulated raw time-of-flight sensor data value and a corresponding simulated ground truth depth value. The trained machine learning component is configured to compute in a single stage, for an item of the stored raw time-of-flight sensor data, a depth value of a surface depicted by the item, by pushing the item through the trained machine learning component. ## Claims ### Claim 1 A depth detection apparatus comprising: a memory storing raw time-of-flight sensor data received from a time-of-flight sensor; and a processor comprising a trained machine learning component having been trained using training data pairs, a training data pair comprising at least one simulated raw time-of-flight sensor data value and a corresponding simulated ground truth depth value; the trained machine learning component configured to compute, in a single stage, for an item of the stored raw time-of-flight sensor data, a depth value of a surface depicted by the item by pushing the item through the trained machine learning component. ### Claim 2 (depends on 1) The apparatus of claim 1, the trained machine learning component having been trained using simulated raw time-of-flight sensor data values which incorporate simulated multi-path interference. ### Claim 3 (depends on 1) The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor data values computed using a computer graphics renderer which simulates multi-path interference. ### Claim 4 (depends on 1) The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor data values comprising, for an individual pixel, weighted intensity values at different depths potentially depicted by the pixel. ### Claim 5 (depends on 1) The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor data values where information about an exposure profile of the time-of-flight sensor is combined with the simulated raw time-of-flight sensor data values. ### Claim 6 (depends on 1) The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor data values where information about sensor noise of the time-of-flight sensor is combined with the simulated raw time-of-flight sensor data values. ### Claim 7 (depends on 1) The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor data values computed using a computer graphics renderer from a plurality of instances of a parametric 3D environment model, where the instances of the parametric 3D environment model are computer generated automatically at random. ### Claim 8 (depends on 7) The apparatus of claim 7 where parameters of the parametric 3D environment model comprise one or more of: geometry of an object in the 3D environment model, position of an object in the 3D environment model, presence of an object in the 3D environment model, orientation of an object in the 3D environment model, surface materials and reflectivity, ambient illumination. ### Claim 9 (depends on 1) The apparatus of claim 1 wherein a training data pair comprises a frame of simulated raw time-of-flight sensor data values and a corresponding simulated ground truth depth map. ### Claim 10 (depends on 1) The apparatus of claim 1, the trained machine learning component having been trained using simulated raw time-of-flight sensor data values computed using a computer graphics renderer for a plurality of randomly selected viewpoints of the time-of-flight sensor, and where any of the viewpoints which are within a threshold distance of a surface in a 3D environment model used by the computer graphics renderer are omitted. ### Claim 11 (depends on 1) The apparatus of claim 1 the trained machine learning component having been trained using simulated raw time-of-flight sensor data values aggregated over a neighborhood of a pixel, where the neighborhood is a spatial neighborhood, or a temporal neighborhood, or a spatial and temporal neighborhood. ### Claim 12 (depends on 1) The apparatus of claim 1 where the trained machine learning component is a pixel independent regressor. ### Claim 13 (depends on 1) The apparatus of claim 1 where the trained machine learning component is regressor which takes into account relationships between pixels of the stored time-of-flight sensor data. ### Claim 14 (depends on 1) The apparatus of claim 1 where the trained machine learning component is a convolutional neural network and where each training data pair comprises a frame of simulated raw time-of-flight sensor data and a ground truth depth map. ### Claim 15 (depends on 1) The apparatus of claim 1 where the trained machine learning component is at least partially implemented using hardware logic selected from any one or more of: a field-programmable gate array, an application-specific integrated circuit, an application-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit. ### Claim 16 One or more computer-readable storage devices comprising computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations storing frames of raw time-of-flight sensor data received from a time-of-flight sensor; and operating a trained machine learning component having been trained using training data pairs, a training data pair comprising at least one simulated raw time-of-flight sensor data value and a corresponding simulated ground truth depth value; wherein operating the trained machine learning component comprises computing, in a single stage, for an item of the stored raw time-of-flight sensor data, a depth value of a surface depicted by the item, by pushing the item through the trained machine learning component. ### Claim 17 (depends on 16) The one or more computer-readable storage device of claim 16 where operating the trained machine learning component further comprises operating the trained machine learning component in real time by computing the depth value at a rate which is equivalent to or faster than a frame rate of the time-of-flight sensor. ### Claim 18 (depends on 16) The one or more computer-readable storage device of claim 16 where the trained machine learning component comprises a convolutional neural network. ### Claim 19 (depends on 16) The one or more computer-readable storage device of claim 16 where the trained machine learning component comprises a pixel independent regressor which is a regressor that does not take into account relationships between pixels of a time-of-flight sensor frame. ### Claim 20 A computer-implemented method comprising: storing, at a memory, raw time-of-flight sensor data received from a time-of-flight sensor; and operating, by a processor, a trained machine learning component having been trained using training data pairs, a training data pair comprising at least one simulated raw time-of-flight sensor data value and a corresponding simulated ground truth depth value; wherein operating the trained machine learning component comprises computing, in a single stage, for an item of the stored raw time-of-flight sensor data, a depth value of a surface depicted by the item, by pushing the item through the trained machine learning component.