# US 10,484,667 B2 · Generating 3D depth map using parallax > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,667 B2 - Document ID: US10484667B2 - Status: Granted - Kind: B2 - Application number: 15/798,914 - Filed: 2017-10-31 - Published: 2019-05-02 - Granted: 2019-11-19 - Earliest claimed: 2017-10-31 - Assignees: Sony Corporation; Sony Interactive Entertainment Inc. - Applicants: Sony Corporation; Sony Interactive Entertainment Inc. - Inventors: Peter Shintani; Morio Usami; Kazuyuki Shikama; Keith Resch - Web page: https://patentgrep.com/patent/US10484667B2 ## CPC - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48) - G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves - G01S 17/89: for mapping or imaging - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/55: from multiple images - G06T 7/593: from stereo images - G06T 15/205: {Image-based rendering} - H04N 13/128: Adjusting depth or disparity - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - H04N 23/60: Control of cameras or camera modules - G06T 2207/10028: Range image; Depth image; 3D point clouds - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Parallax views of objects are used to generate 3D depth maps of the objects using time of flight (TOF) information. In this way, the deleterious effects of multipath interference can be reduced to improve 3D depth map accuracy without using computationally intensive algorithms. ## Claims ### Claim 1 An assembly comprising: plural computerized devices; each computerized device comprising at least one laser emitter configured to output signals useful for generating at least one three-dimensional (3D) depth map; a first one of the computerized devices being programmed with instructions to: generate an initial two-dimensional (2D) camera image and an initial laser-based time of flight (TOF) image of a target object; move a camera generating the initial laser-based TOF image of the target object; generate a secondary laser-based TOF image of the target object; use the initial laser-based TOF image of the target object and secondary laser-based TOF image of the target object to generate a 3D depth map of the target object; identify whether a discrepancy between the initial 2D camera image and the initial laser-based TOF image satisfies at least one criterion; responsive to determining that a discrepancy between the initial 2D camera image and the initial laser-based TOF image satisfies the criterion, remove data points from the initial 3D depth map; and responsive to determining that a discrepancy between the initial 2D camera image and the initial laser-based TOF image does not satisfy the criterion, not remove data points from the initial 3D depth map. ### Claim 2 (depends on 1) The assembly of claim 1, wherein the camera is moved by moving the camera relative to an imaging device supporting the camera. ### Claim 3 (depends on 1) The assembly of claim 1, wherein the camera is moved by moving, relative to the target device, an imaging device supporting the camera. ### Claim 4 (depends on 1) The assembly of claim 1, wherein the instructions are executable to: use the initial laser-based TOF image of the target object and secondary laser-based TOF image of the target object to generate a 3D depth map of the target object by consolidating the initial laser-based TOF image of the target object and secondary laser-based TOF image of the target object. ### Claim 5 (depends on 1) The assembly of claim 1, wherein the discrepancy between the initial 2D camera image and the initial laser-based TOF image is identified at least in part using a comparison of the relative sizes of the images. ### Claim 6 (depends on 1) The assembly of claim 1, wherein the discrepancy between the initial 2D camera image and the initial laser-based TOF image is identified at least in part by identifying that the initial laser-based TOF image presents a portion of a target object that is not presented in the initial 2D camera image. ### Claim 7 (depends on 1) The assembly of claim 1, wherein the discrepancy between the initial 2D camera image and the initial laser-based TOF image is identified at least in part by identifying that the initial laser-based TOF image presents a flat surface larger than a threshold. ### Claim 8 An assembly comprising: plural computerized devices; each computerized device comprising at least one laser emitter configured to output signals useful for generating at least one three-dimensional (3D) depth map; a first one of the computerized devices being programmed with instructions to: generate an initial two-dimensional (2D) camera image and an initial laser-based time of flight (TOF) image of a target object; move a camera generating the initial laser-based TOF image of the target object; generate a secondary laser-based TOF image of the target object; use the initial laser-based TOF image of the target object and secondary laser-based TOF image of the target object to generate a 3D depth map of the target object; and move the camera and/or an imaging device supporting the camera at least in part based on a direction of a surface causing multipath errors in the initial 3D depth map. ### Claim 9 (depends on 8) The assembly of claim 8, wherein the camera and/or an imaging device supporting the camera is moved away from the surface causing multipath errors in the initial 3D depth map responsive to identifying a discrepancy between the initial 2D camera image and the initial laser-based TOF image meeting a threshold. ### Claim 10 (depends on 8) The assembly of claim 8, wherein the camera and/or an imaging device supporting the camera is moved away from the surface causing multipath errors in the initial 3D depth map along an axis that places a line of sight to the target device further away from the surface. ### Claim 11 (depends on 8) The assembly of claim 8, wherein the camera and/or an imaging device supporting the camera is moved along an axis to maximize parallax with at least one of the initial images. ### Claim 12 An assembly comprising: plural computerized devices; each computerized device comprising at least one laser emitter configured to output signals useful for generating at least one three-dimensional (3D) depth map; a first one of the computerized devices being programmed with instructions to: generate an initial two-dimensional (2D) camera image and an initial laser-based time of flight (TOF) image of a target object; move a camera generating the initial laser-based TOF image of the target object; generate a secondary laser-based TOF image of the target object; use at least one of the initial laser-based TOF image of the target object and secondary laser-based TOF image of the target object to generate a 3D depth map of the target object; and use the initial laser-based TOF image of the target object and secondary laser-based TOF image of the target object to generate a 3D depth map of the target object by determining which of the initial and secondary TOF images exhibits less noise, and using the image exhibiting less noise to establish the 3D depth map.