# US 12,002,292 B2 · Online calibration of 3D scan data from multiple viewpoints > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,292 B2 - Document ID: US12002292B2 - Status: Granted - Kind: B2 - Application number: 17/309,335 - Filed: 2019-11-26 - Published: 2022-01-13 - Granted: 2024-06-04 - Earliest claimed: 2018-11-28 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Kendai Furukawa - Web page: https://patentgrep.com/patent/US12002292B2 ## CPC - G01S 5/16: using electromagnetic waves other than radio waves - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 17/89: for mapping or imaging - G06T 7/251: {involving models} - G06T 7/596: {from three or more stereo images} - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/30196: Human being; Person - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A calibration system and method for online calibration of 3D scan data from multiple viewpoints is provided. The calibration system receives a set of depth scans and a corresponding set of color images of a scene that includes a human-object as part of a foreground of the scene. The calibration system extracts a first three-dimensional (3D) representation of the foreground based on a first depth scan and spatially aligns the extracted first 3D representation with a second 3D representation of the foreground. The first 3D representation and the second 3D representation are associated with a first viewpoint and a second viewpoint, respectively, in a 3D environment. The calibration system updates the spatially aligned first 3D representation based on the set of color images and a set of structural features of the human-object and reconstructs a 3D mesh of the human-object based on the updated first 3D representation of the foreground. ## Claims ### Claim 1 A calibration system, comprising: circuitry configured to: receive a set of depth scans and a corresponding set of color images of a scene comprising a human-object as part of a foreground of the scene; extract a first three-dimensional (3D) representation of the foreground based on a first depth scan of the set of depth scans, wherein the first 3D representation is associated with a first viewpoint in a 3D environment; spatially align the extracted first 3D representation with a second 3D representation of the foreground, wherein the second 3D representation is associated with a second viewpoint in the 3D environment; update the spatially aligned first 3D representation based on the corresponding set of color images and a set of structural features of the human-object, as a human-prior; extract a third 3D representation of a ground surface in the scene based on the first depth scan associated with the first viewpoint; spatially align the extracted third 3D representation of the ground surface with a fourth 3D representation of the ground surface, the fourth 3D representation being associated with the second viewpoint of the scene; update the spatially aligned first 3D representation further based on the spatial alignment of the extracted third 3D representation with the fourth 3D representation; and reconstruct a 3D mesh of the human-object based on the updated first 3D representation of the foreground. ### Claim 2 (depends on 1) The calibration system according to claim 1, further comprising a plurality of scanning devices configured to acquire the set of depth scans and the corresponding set of color images from a corresponding set of viewpoints in the 3D environment. ### Claim 3 (depends on 2) The calibration system according to claim 2, wherein the plurality of scanning devices, collectively, form a multi-camera network having a combined field-of-view that covers an entire surface of the human-object. ### Claim 4 (depends on 1) The calibration system according to claim 1, wherein the circuitry is further configured to: estimate a set of candidate transformations for the spatial alignment of the extracted first 3D representation with the second 3D representation; compute a visibility error-metric for each candidate transformation of the estimated set of candidate transformations; select, from the estimated set of candidate transformations, a candidate transformation for which the computed visibility error-metric is a minimum; and spatially align the extracted first 3D representation with the second 3D representation based on the selected candidate transformation. ### Claim 5 (depends on 1) The calibration system according to claim 1, wherein the set of structural features comprises a skeleton joint prior, a hand prior, and a face prior of the human-object. ### Claim 6 (depends on 1) The calibration system according to claim 1, wherein the circuitry is further configured to: estimate a first set of two-dimensional (2D) feature points for the set of structural features based on a first color image of the corresponding set of color images; and estimate a second set of 2D feature points for the set of structural features based on a second color image of the corresponding set of color images, wherein the first color image corresponds to the first depth scan associated with the first viewpoint, and the second color image corresponding to a second depth scan associated with the second viewpoint. ### Claim 7 (depends on 6) The calibration system according to claim 6, wherein the circuitry is further configured to compute a set of 3D feature points for the set of structural features based on the estimated first set of 2D feature points and the estimated second set of 2D feature points. ### Claim 8 (depends on 7) The calibration system according to claim 7, wherein the circuitry is further configured to update the spatially aligned first 3D representation based on the computed set of 3D feature points. ### Claim 9 (depends on 7) The calibration system according to claim 7, wherein the extracted first 3D representation comprises a first plurality of 3D points representing at least a surface portion of the human-object from the first viewpoint. ### Claim 10 (depends on 9) The calibration system according to claim 9, wherein the circuitry is further configured to: compute a distance between the computed set of 3D feature points and a portion of the first plurality of 3D points corresponding to the set of structural features in the extracted first 3D representation; estimate a global energy function based on the computed distance; and update the spatially aligned first 3D representation further based on the estimated global energy function being a minimum. ### Claim 11 (depends on 1) The calibration system according to claim 1, wherein the reconstructed 3D mesh corresponds to a watertight mesh that at least partially captures a texture of the human-object in the corresponding set of color images. ### Claim 12 (depends on 11) The calibration system according to claim 11, wherein the circuitry is further configured to: transfer texture values from reliable textured regions on temporally neighboring or distant meshes of the human-object to unreliable textured regions on the reconstructed 3D mesh; and refine the 3D mesh based on the transfer, wherein the refined 3D mesh corresponds to one temporal frame of a free-viewpoint video. ### Claim 13 A method, comprising: receiving a set of depth scans and a corresponding set of color images of a scene comprising a human-object as part of a foreground of the scene; extracting a first three-dimensional (3D) representation of the foreground based on a first depth scan of the set of depth scans, wherein the first 3D representation is associated with a first viewpoint in a 3D environment; spatially aligning the extracted first 3D representation with a second 3D representation of the foreground, wherein the second 3D representation is associated with a second viewpoint in the 3D environment; updating the spatially aligned first 3D representation based on the corresponding set of color images and a set of structural features of the human-object, as a human-prior; extracting a third 3D representation of a ground surface in the scene based on the first depth scan associated with the first viewpoint; spatially aligning the extracted third 3D representation of the ground surface with a fourth 3D representation of the ground surface, the fourth 3D representation being associated with the second viewpoint of the scene; updating the spatially aligned first 3D representation further based on the spatial alignment of the extracted third 3D representation with the fourth 3D representation; and reconstructing a 3D mesh of the human-object based on the updated first 3D representation of the foreground. ### Claim 14 (depends on 13) The method according to claim 13, further comprising: estimating a first set of two-dimensional (2D) feature points for the set of structural features based on a first color image of the corresponding set of color images; estimating a second set of 2D feature points for the set of structural features based on a second color image of the corresponding set of color images, wherein the first color image corresponds to the first depth scan associated with the first viewpoint, and the second color image corresponding to a second depth scan associated with the second viewpoint; and computing a set of 3D feature points for the set of structural features based on the estimated first set of 2D feature points and the estimated second set of 2D feature points. ### Claim 15 (depends on 14) The method according to claim 14, further comprising updating the spatially aligned first 3D representation based on the computed set of 3D feature points. ### Claim 16 A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a calibration system, causes the calibration system to execute operations, the operations comprising: receiving a set of depth scans and a corresponding set of color images of a scene comprising a human-object as part of a foreground of the scene; extracting a first three-dimensional (3D) representation of the foreground based on a first depth scan of the set of depth scans, wherein the first 3D representation is associated with a first viewpoint in a 3D environment; spatially aligning the extracted first 3D representation with a second 3D representation of the foreground, wherein the second 3D representation is associated with a second viewpoint in the 3D environment; updating the spatially aligned first 3D representation based on the corresponding set of color images and a set of structural features of the human-object, as a human-prior; extracting a third 3D representation of a ground surface in the scene based on the first depth scan associated with the first viewpoint; spatially aligning the extracted third 3D representation of the ground surface with a fourth 3D representation of the ground surface, the fourth 3D representation being associated with the second viewpoint of the scene; updating the spatially aligned first 3D representation further based on the spatial alignment of the extracted third 3D representation with the fourth 3D representation; and reconstructing a 3D mesh of the human-object based on the updated first 3D representation of the foreground.