# US 10,503,265 B2 · Mixed-mode depth detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,265 B2 - Document ID: US10503265B2 - Status: Granted - Kind: B2 - Application number: 14/848,128 - Filed: 2015-09-08 - Published: 2017-03-09 - Granted: 2019-12-10 - Earliest claimed: 2015-09-08 - Assignees: Microvision, Inc. - Applicants: Microvision, Inc. - Inventors: P. Selvan Viswanathan - Web page: https://patentgrep.com/patent/US10503265B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G01S 7/4817: {relating to scanning} - G01S 17/46: Indirect determination of position data - G01S 17/89: for mapping or imaging - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - H04N 23/20: for generating image signals from infrared radiation only - H04N 23/56: provided with illuminating means - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Briefly, in accordance with one or more embodiments, a MEMS laser beam display to implement mixed-mode depth detection comprises an imaging engine to project an image in a field of view of the MEMS laser beam display, a time of flight (TOF) depth detector to determine a distance to a target within the field of view in a first mode of depth detection, wherein the TOF depth detector determines coarse depth data of the target, and a structured light depth detector to determine a distance to the target in a second mode of depth detection, wherein the structured light depth detector determines fine depth data of the target. Depth processing circuitry may generate a composite three-dimensional (3D) image of the target, or may identify a gesture region of the target for gesture recognition. ## Claims ### Claim 1 A MEMS laser beam display, comprising: at least one source of laser light configured to generate a laser beam; an imaging engine configured to reflect the laser beam in a pattern to project an image in a field of view of the MEMS laser beam display; a time of flight (TOF) depth detector configured to receive reflections of the laser beam from a target to determine a distance to the target within the field of view via a time of flight calculation in a first mode of depth detection, wherein the TOF depth detector determines three-dimensional (3D) coarse depth data of the target; depth processing circuitry to identify one or more regions in the field of view from the 3D coarse depth data, where the one or more regions include of only a portion of the field of view, and wherein the imaging engine is further configured to project at least one on structured light pattern in the identified one or more regions while not projecting the at least one structured light pattern outside the identified one or more regions in the field of view; a structured light depth detector configured to detect a reflection of the at least one structured light pattern from the target to determine a distance to the target in the one or more identified regions in a second mode of depth detection, wherein the structured light depth detector determines three-dimensional (3D) fine depth data of the target in the one or more identified regions; and wherein the depth processing circuitry is further configured to generate a composite three-dimensional (3D) image of the target using a combination of the 3D coarse depth data and the 3D fine depth data for the one or more identified regions. ### Claim 2 (depends on 1) The MEMS laser beam display of claim 1, wherein the depth processing circuitry is further configured to identify a gesture in the one or more identified regions using the 3D fine depth data. ### Claim 3 (depends on 1) The MEMS laser beam display of claim 1, wherein the imaging engine comprises a microelectromechanical system (MEMS) scanner, or a nanoscale scanner, or a combination thereof. ### Claim 4 (depends on 1) The MEMS laser beam display of claim 1, wherein the TOF depth detector comprises a photodetector to detect the received reflections of the laser beam from the target. ### Claim 5 (depends on 1) The MEMS laser beam display of claim 1, wherein the structured light depth detector comprises a camera to detect the reflection of the at least one structured light pattern, and to determine a distance to the target via distance calculation on the detected reflection of the structured light pattern. ### Claim 6 (depends on 1) The MEMS laser beam display of claim 1, further comprising an actuator to receive an input from a user to direct one or more structured light patterns emitted by the imaging engine to a selected region of the target. ### Claim 7 (depends on 1) The MEMS laser beam display of claim 1, wherein the depth processing circuitry is further configured to obtain a skeletal model of the target from the 3D coarse depth data obtained by the TOF depth detector, and to use the skeletal model to determine the one or more identified regions of the target to direct a structured light pattern to obtain the 3D fine depth data. ### Claim 8 (depends on 1) The MEMS laser beam display of claim 1, wherein the laser beam includes an infrared (IR) beam, and the TOF depth detector comprises an IR photodetector to detect the reflection of the IR beam off the target. ### Claim 9 (depends on 1) The MEMS laser beam display of claim 1, wherein the at least one structured light pattern includes a visible structured light pattern or an invisible structured light pattern, or a combination thereof, on the target, and the structured light depth detector comprise a camera to detect the visible structured light pattern or the invisible structured light pattern, or a combination thereof, reflected off the target. ### Claim 10 (depends on 1) The MEMS laser beam display of claim 1, wherein the 3D coarse depth data and the 3D fine depth data are determined from the pattern of the laser beam generated by the imaging engine during a same frame. ### Claim 11 (depends on 1) The MEMS laser beam display of claim 1, wherein the 3D coarse depth data is determined is determined from the pattern of the laser beam generated by the imaging engine during a first frame, and wherein the 3D fine depth data is determined from the pattern of the laser beam generated by the imaging engine during a second frame, where the second frame sequentially follows the first frame. ### Claim 12 A method to obtain a three-dimensional image of a target, comprising: projecting a scanned beam in a field of view onto the target; detecting a reflection of the scanned beam reflected off the target; determining a three-dimensional (3D) coarse depth map of the target from the reflection of the scanned beam by calculating a time of flight of the scanned beam; identifying one or more regions in the 3D coarse depth map in which to obtain fine depth data, where the one or more regions include of only a portion of the field of view; projecting a structured light pattern in the one or more regions with the scanned beam while not projecting the structured light pattern outside the one or more regions in the field of view; determining a three-dimensional (3D) fine depth map of the target in the one or more regions from a reflection of the structured light pattern by calculating depth information from a reflection of the structured light pattern from the one or more regions; and obtaining a three-dimensional image the target using a composite of the 3D coarse depth map and the 3D fine depth map. ### Claim 13 (depends on 12) The method of claim 12, wherein said projecting a scanned beam comprises projecting an infrared (IR) beam, and said detecting comprising detecting the reflection of the IR beam using an IR photodetector. ### Claim 14 (depends on 12) The method of claim 12, wherein said identifying comprises obtaining a skeletal model of the target from the 3D coarse depth map, wherein the one or more regions are located on the skeletal model of the target. ### Claim 15 (depends on 12) The method of claim 12, further comprising gesture recognition for the one or more regions to identify one or more gestures of the target. ### Claim 16 (depends on 12) The method of claim 12, further comprising gesture recognition for the one or more targets to identify one or more gestures for the one or more targets. ### Claim 17 (depends on 12) The method of claim 12, wherein said projecting a scanned beam and said projecting a structured light pattern comprises projecting the scanned beam and projecting the structured light patterned during a same frame. ### Claim 18 (depends on 12) The method of claim 12, wherein said projecting a scanned beam comprises projecting the scanned beam during a first frame and wherein said projecting a structured light pattern projecting comprises projecting the structured light patterned during a second frame, where the second frame sequentially follows the first frame. ### Claim 19 A method to obtain a three-dimensional image of a target, comprising: projecting a scanned beam in a field of view onto the target; detecting a reflection of the scanned beam reflected from the field of view; determining a three-dimensional (3D) coarse depth map from the reflection of the scanned beam by calculating a time of flight of the scanned beam; identifying one or more targets in the field of view using the 3D coarse depth map, where the one or more targets include only a portion of the field of view; drawing a structured light pattern on the one or more targets with the scanned beam while not drawing the structured light pattern outside the one or more targets in the field of view; capturing image data for the one or more targets from a reflection of the structured light pattern from the one or more targets; determining a three-dimensional (3D) fine depth map for the one or more targets from the image data by calculating depth information from a reflection of the structured light pattern from the one or more targets; and obtaining a three-dimensional image of the one or more targets using the 3D fine depth map. ### Claim 20 (depends on 19) The method of claim 19, wherein said projecting a scanned beam comprises projecting an infrared (IR) beam, and said detecting comprising detecting the reflection of the IR beam using an IR photodetector. ### Claim 21 (depends on 19) The method of claim 19, wherein said identifying comprises obtaining a skeletal model of the one or more targets from the 3D coarse depth map, wherein structured light patterns are drawn on one or more regions of the skeletal model.