# US 11,288,842 B2 · Method and system for re-projecting and combining sensor data for visualization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,288,842 B2 - Document ID: US11288842B2 - Status: Granted - Kind: B2 - Application number: 16/791,203 - Filed: 2020-02-14 - Published: 2020-08-20 - Granted: 2022-03-29 - Earliest claimed: 2019-02-15 - Assignees: Interaptix Inc. - Applicants: Interaptix Inc. - Inventors: Dae Hyun Lee; Tyler James Doyle - Web page: https://patentgrep.com/patent/US11288842B2 ## CPC - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - H04N 5/265: Mixing - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06T 15/205: {Image-based rendering} - H04N 17/002: {for television cameras} - G06T 2210/56: Particle system, point based geometry or rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract There is provided a system and method of re-projecting and combining sensor data of a scene from a plurality of sensors for visualization. The method including: receiving the sensor data from the plurality of sensors; re-projecting the sensor data from each of the sensors into a new viewpoint; localizing each of the re-projected sensor data; combining the localized re-projected sensor data into a combined image; and outputting the combined image. In a particular case, the receiving and re-projecting can be performed locally at each of the sensors. ## Claims ### Claim 1 A computer-implemented method of re-projecting and combining sensor data of a scene from a plurality of sensors for visualization, the method comprising: receiving the sensor data from the plurality of sensors, wherein the sensor data comprises red-green-blue-depth (RGB-D) channel values for each of a plurality of pixels, and wherein the values of the RGB-D channels are from a combination of two or more sensors; re-projecting the sensor data from each of the sensors into a new viewpoint by: calibrating each of the sensors to determine one or more calibration values for the respective sensor; generating a point cloud for the respective sensor by applying the calibration values to the depth D channel values for each pixel; and applying a matrix representative of the new viewpoint to the point cloud for each sensor; localizing each of the re-projected sensor data; combining the localized re-projected sensor data into a combined image; and outputting the combined image. ### Claim 2 (depends on 1) The method of claim 1, wherein calibrating each of the sensors comprises determining intrinsic parameters for each of the sensors, the intrinsic parameters comprising distortions and camera matrix. ### Claim 3 (depends on 1) The method of claim 1, wherein the localization comprises performing simultaneous localization and mapping (SLAM) to position and orient the re-projected sensor data relative to each other. ### Claim 4 (depends on 3) The method of claim 3, wherein simultaneous localization and mapping comprises performing point-to-plane iterative closest point. ### Claim 5 (depends on 1) The method of claim 1, further comprising adding supporting elements to the combined image. ### Claim 6 (depends on 5) The method of claim 5, wherein the supporting elements comprise graphical elements associated with one or more objects located in the scene. ### Claim 7 (depends on 1) The method of claim 1, wherein the combined image comprises a top-down view. ### Claim 8 A system for re-projecting and combining sensor data of a scene from a plurality of sensors for visualization, the system comprising one or more processors and one or more data storages, the one or more processors in communication with the one or more data storages and configured to execute: an input module to receive the sensor data from the plurality of sensors; a projection module to re-project the sensor data from each of the sensors into a new viewpoint; a localization module to localize each of the re-projected sensor data; a combination module to combine the localized re-projected sensor data into a combined image; and an output module to output the combined image. ### Claim 9 (depends on 8) The system of claim 8, wherein one or more of the processors and one or more of the data storages are distributed and local to one or more of the sensors, and wherein the distributed one or more processors are configured to execute at least the input module and the projection module. ### Claim 10 (depends on 8) The system of claim 8, wherein the sensor data comprises red-green-blue-depth (RGB-D) channel values for each of a plurality of pixels. ### Claim 11 (depends on 10) The system of claim 10, wherein the values of the RGB-D channels are from a combination of two or more sensors. ### Claim 12 (depends on 11) The system of claim 11, wherein re-projecting the sensor data comprises: calibrating each of the sensors to determine one or more calibration values for the respective sensor; generating a point cloud for the respective sensor by applying the calibration values to each of the depth D channel values for each pixel; and applying a matrix representative of the new viewpoint to the point cloud for each sensor. ### Claim 13 (depends on 12) The system of claim 12, wherein calibrating each of the sensors comprises determining intrinsic parameters for each of the sensors, the intrinsic parameters comprising distortions and camera matrix. ### Claim 14 (depends on 8) The system of claim 8, wherein the localization comprises performing simultaneous localization and mapping (SLAM) to position and orient the re-projected sensor data relative to each other. ### Claim 15 (depends on 8) The system of claim 8, further comprising a support module to add supporting elements to the combined image. ### Claim 16 (depends on 15) The system of claim 15, wherein the supporting elements comprise graphical elements associated with one or more objects located in the scene. ### Claim 17 (depends on 8) The system of claim 8, wherein the combined image comprises a top-down view.