# US 10,122,996 B2 · Method for 3D multiview reconstruction by feature tracking and model registration > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,996 B2 - Document ID: US10122996B2 - Status: Granted - Kind: B2 - Application number: 15/291,714 - Filed: 2016-10-12 - Published: 2017-09-14 - Granted: 2018-11-06 - Earliest claimed: 2016-03-09 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Ko-Kai Albert Huang; Ming-Chang Liu - Web page: https://patentgrep.com/patent/US10122996B2 ## CPC - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - G06T 7/579: from motion - G06T 7/593: from stereo images - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 13/128: Adjusting depth or disparity - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/243: using three or more two-dimensional [2D] image sensors - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A 3D multiview reconstruction method takes a sequence of 2D stereo images from a narrow field-of-view imager (e.g., camera) and reconstructs a 3D representation of the wide field-of-view object or scene. The 3D multiview reconstruction method tracks 2D image pixels across neighboring frames and constraints for frame integration via 3D model construction. ## Claims ### Claim 1 A method programmed in a non-transitory memory of a device comprising: acquiring one or more 2D stereo images; reconstructing a 3D multiview image using the one or more 2D stereo images, wherein reconstructing the 3D multiview image includes: rectifying the one or more images; estimating a disparity between images; implementing 2D pixel tracking across neighboring frames; estimating a 3D pose by mapping tracked 2D pixels to 3D coordinates; and constructing a 3D model based on the 3D pose; and displaying the 3D multiview image. ### Claim 2 (depends on 1) The method of claim 1 wherein acquiring the one or more 2D stereo images includes utilizing a plurality of cameras to capture the one or more 2D stereo images. ### Claim 3 (depends on 1) The method of claim 1 wherein acquiring the one or more 2D stereo images includes downloading the one or more 2D stereo images from a server. ### Claim 4 (depends on 1) The method of claim 1 wherein the one or more 2D stereo images have a narrow field of view. ### Claim 5 (depends on 1) The method of claim 1 wherein reconstructing the 3D multiview image includes: selecting a frame of the one or more images; performing a noise check to discard an image of the one or more images if noise of the image is above a threshold; and selecting a region of interest in the one or more images. ### Claim 6 (depends on 1) The method of claim 1 wherein the 3D image comprises a 3D surface. ### Claim 7 (depends on 1) The method of claim 1 wherein the 3D image comprises a 3D point cloud. ### Claim 8 An apparatus comprising: a non-transitory memory for storing an application, the application for: acquiring one or more 2D stereo images; reconstructing a 3D multiview image using the one or more 2D stereo images, wherein reconstructing the 3D multiview image includes: rectifying the one or more images; estimating a disparity between images; implementing 2D pixel tracking across neighboring frames; estimating a 3D pose by mapping tracked 2D pixels to 3D coordinates; and constructing a 3D model based on the 3D pose; and displaying the 3D multiview image; and a processing component coupled to the memory, the processing component configured for processing the application. ### Claim 9 (depends on 8) The apparatus of claim 8 wherein acquiring the one or more 2D stereo images includes utilizing a plurality of cameras to capture the one or more 2D stereo images. ### Claim 10 (depends on 8) The apparatus of claim 8 wherein acquiring the one or more 2D stereo images includes downloading the one or more 2D stereo images from a server. ### Claim 11 (depends on 8) The apparatus of claim 8 wherein the one or more 2D stereo images have a narrow field of view. ### Claim 12 (depends on 8) The apparatus of claim 8 wherein reconstructing the 3D multiview image includes: selecting a frame of the one or more images; performing a noise check to discard an image of the one or more images if noise of the image is above a threshold; and selecting a region of interest in the one or more images. ### Claim 13 (depends on 8) The apparatus of claim 8 wherein the 3D image comprises a 3D surface. ### Claim 14 (depends on 8) The apparatus of claim 8 wherein the 3D image comprises a 3D point cloud. ### Claim 15 A system comprising: a plurality of cameras for acquiring one or more 2D stereo images; and a computing device configured for: receiving the one or more 2D stereo images; reconstructing a 3D multiview image using the one or more 2D stereo images, wherein reconstructing the 3D multiview image includes: rectifying the one or more images; estimating a disparity between images; implementing 2D pixel tracking across neighboring frames; estimating a 3D pose by mapping tracked 2D pixels to 3D coordinates; and constructing a 3D model based on the 3D pose; and displaying the 3D multiview image. ### Claim 16 (depends on 15) The system of claim 15 wherein acquiring the one or more 2D stereo images includes utilizing a plurality of cameras to capture the one or more 2D stereo images. ### Claim 17 (depends on 15) The system of claim 15 wherein acquiring the one or more 2D stereo images includes downloading the one or more 2D stereo images from a server. ### Claim 18 (depends on 15) The system of claim 15 wherein the one or more 2D stereo images have a narrow field of view. ### Claim 19 (depends on 15) The system of claim 15 wherein reconstructing the 3D multiview image includes: selecting a frame of the one or more images; performing a noise check to discard an image of the one or more images if noise of the image is above a threshold; and selecting a region of interest in the one or more images. ### Claim 20 (depends on 15) The system of claim 15 wherein the 3D image comprises a 3D surface. ### Claim 21 (depends on 15) The system of claim 15 wherein the 3D image comprises a 3D point cloud.