# US 11,079,475 B2 · Optical device and depth information generating method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,079,475 B2 - Document ID: US11079475B2 - Status: Granted - Kind: B2 - Application number: 15/579,375 - Filed: 2016-02-24 - Published: 2018-06-07 - Granted: 2021-08-03 - Earliest claimed: 2015-06-08 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Sergii Gryshchenko; Iegor Vdovychenko; Ivan Safonov; Andrii But; Vitaliy Bulygin; Volodymyr Khristyan - Web page: https://patentgrep.com/patent/US11079475B2 ## CPC - G01S 7/4815: {using multiple transmitters} - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/48: Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G03B 21/008: {using micromirror devices} - G03B 21/2033: {LED or laser light sources} - G03B 21/28: Reflectors in projection beam {(in illumination beam G03B21/2066)} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An optical device is disclosed. An optical device of the present invention that generates a depth map for an object comprises: a projector for irradiating electromagnetic waves onto an object; an optical receiver for receiving reflected waves reflected from the object; and a processor for generating a first depth map for the object on the basis of the direction of arrival of the received reflected waves, generating a second depth map for the object on the basis of the arrival time of the received reflected waves, and combining the first depth map and the second depth map and generating a depth map for the object. ## Claims ### Claim 1 A depth information generating method of an optical device for generating a depth map for an object, comprising: irradiating an electromagnetic wave onto the object; receiving the electromagnetic wave reflected from the object through an optical receiver; generating a first depth map for the object based on a direction of arrival of the electromagnetic wave reflected from the object; generating a second depth map for the object based on a time during which the electromagnetic wave travels a round trip path from being reflected against the object until being received; and generating a depth map combining the first depth map and the second depth map. ### Claim 2 (depends on 1) The depth information generating method of claim 1, wherein the generating comprises: generating the first depth map from a direction of arrival spectrum calculated by performing a correlation operation on a steering vector calculated based on an impulse response of the optical receiver and an output vector of the optical receiver receiving the electromagnetic wave reflected from the object. ### Claim 3 (depends on 2) The depth information generating method of claim 2, wherein the optical receiver comprises a plurality of light receiving elements, and wherein the generating comprises performing the correlation operation using an equation below: P l ⁡ ( θ , ϕ ) = ( x → ⁡ ( t ) , a → ⁡ ( θ , ϕ , t , d l ) ) 2 = ∑ m = 0 M - 1 ⁢ ⁢ ( x m ⁡ ( t ) · a m ⁡ ( θ , ϕ , t , d l ) ) 2 where, M is a total number of the plurality of light receiving elements, {right arrow over (x)}(t) is the output vector, {right arrow over (a)}(t, θ, ϕ, d) is the steering vector, xm(t) is output of an m-th light receiving element, and am(θ,φ,t,dl) is a steering vector component corresponding the m-th light receiving element. ### Claim 4 (depends on 2) The depth information generating method of claim 2, wherein the generating comprises: calculating a direction of arrival spectrum from which sidelobes have been removed by performing the correlation operation using an equation below; and generating the first depth map from the direction of arrival spectrum from which the sidelobes have been removed, P l ⁡ ( θ , ϕ ) = [ ∑ m = 0 M - 1 ⁢ ⁢ 〈 x m , a m 〉 , if ⁢ ⁢ ∀ m ⁢ 〈 x m , a m 〉 ≠ 0 0 , otherwise where, is 〈 x m , a m 〉 = [ 1 M , if ⁢ ⁢ ( x m , a m )  x m  ·  a m  > ɛ , ɛ ∈ ( 0 , 1 ) 0 , otherwise , where, (xm, am) is ( x m , a m ) = ∫ ( k → , r → m ) c ( k → , r → m ) + Δ c ⁢ x m ⁡ ( t ) · a m ⁡ ( θ , ϕ , t , d l ) ⁢ ⁢ d ⁢ ⁢ t where, M is a number of the plurality of light receiving elements, {right arrow over (k)} is a wave vector representing the electromagnetic wave reflected from the object incident on the optical receiver, {right arrow over (r)}m is a radius vector representing an m-th light receiving element among the plurality of light receiving elements, Δ is a wavelength of the electromagnetic wave reflected from the object, and c is the speed of light. ### Claim 5 (depends on 1) The depth information generating method of claim 1, further comprising: irradiating the electromagnetic wave to a monitoring zone comprising the object; receiving some of signals of the electromagnetic wave scattered by the object; and analyzing the received scattered signal and identifying a position of the object, wherein, the irradiating an electromagnetic wave onto the object comprises irradiating the electromagnetic wave onto the object based on the identified position of the object. ### Claim 6 (depends on 5) The depth information generating method of claim 5, wherein the irradiating of the electromagnetic wave onto the object comprises: analyzing the received scattered signal to calculate a distance between a position at which the electromagnetic wave is irradiated and a point nearest to the object and a point farthest from the object; and irradiating the electromagnetic wave with a varying intensity according to the calculated distance. ### Claim 7 (depends on 5) The depth information generating method of claim 5, wherein the irradiating of the electromagnetic wave onto the object comprises, when a number of the objects is plural, irradiating the electromagnetic wave onto each of the plurality of objects based on the identified positions of the objects. ### Claim 8 (depends on 1) The depth information generating method of claim 1, wherein the electromagnetic wave irradiated onto the object is time and spatial light modulated electromagnetic wave. ### Claim 9 (depends on 1) The depth information generating method of claim 1, wherein the object is a user's hand, and further comprising controlling a function of a display device that outputs an image according to a gesture command corresponding to a movement of the user's hand determined from the generated depth map. ### Claim 10 (depends on 1) The depth information generating method of claim 1, wherein the generating of the depth map comprises combining the first depth map and the second depth map using super resolution technique or hybrid imaging technique. ### Claim 11 (depends on 1) The depth information generating method of claim 1, wherein the irradiating of the electromagnetic wave onto the object comprises irradiating onto the object while shifting the electromagnetic wave composed of a plurality of laser beams to a preset direction for irradiating the electromagnetic wave over an entirety of the object. ### Claim 12 (depends on 1) The depth information generating method of claim 1, wherein the irradiating of the electromagnetic wave onto the object comprises irradiating a time light modulated electromagnetic wave onto the object. ### Claim 13 An optical device for generating a depth map for an object, comprising: a projector for irradiating an electromagnetic wave onto the object; an optical receiver for receiving the electromagnetic wave reflected from the object; and at least one processor configured to: generate a first depth map for the object based on a direction of arrival of the electromagnetic wave reflected from the object, generate a second depth map for the object based on a time during which the electromagnetic wave travels a round trip path from being reflected against the object until being received, and generate a depth map for the object by combining the first depth map and the second depth map. ### Claim 14 (depends on 13) The optical device of claim 13, wherein the at least one processor is further configured to: generate the first depth map from a direction of arrival spectrum calculated by performing a correlation operation on a steering vector calculated based on an impulse response of the optical receiver and an output vector of the optical receiver receiving the electromagnetic wave reflected from the object. ### Claim 15 (depends on 14) The optical device of claim 14, wherein, the optical receiver comprises a plurality of light receiving elements, and wherein the at least one processor is further configured to perform the correlation operation using an equation below: P l ⁡ ( θ , ϕ ) = ( x → ⁡ ( t ) , a → ⁡ ( θ , ϕ , t , d l ) ) 2 = ∑ m = 0 M - 1 ⁢ ⁢ ( x m ⁡ ( t ) · a m ⁡ ( θ , ϕ , t , d l ) ) 2 where, M is a total number of the plurality of light receiving elements, {right arrow over (x)}(t) is the output vector, {right arrow over (a)}(t, θ, ϕ, d) is the steering vector, xm(t) is output of an m-th light receiving element, and am(θ,φ,t,dl) is a steering vector component corresponding the m-th light receiving element. ### Claim 16 (depends on 14) The optical device of claim 14, wherein the at least one processor is further configured to: calculate a direction of arrival spectrum from which sidelobes have been removed by performing the correlation operation using an equation below, and generate the first depth map from the direction of arrival spectrum from which the sidelobes have been removed, P l ⁡ ( θ , ϕ ) = [ ∑ m = 0 M - 1 ⁢ ⁢ 〈 x m , a m 〉 , if ⁢ ⁢ ∀ m ⁢ 〈 x m , a m 〉 ≠ 0 0 , otherwise where, is 〈 x m , a m 〉 = [ 1 M , if ⁢ ⁢ ( x m , a m )  x m  ·  a m  > ɛ , ɛ ∈ ( 0 , 1 ) 0 , otherwise , where, (xm, am) is ( x m , a m ) = ∫ ( k → , r → m ) c ( k → , r → m ) + Δ c ⁢ x m ⁡ ( t ) · a m ⁡ ( θ , ϕ , t , d l ) ⁢ ⁢ d ⁢ ⁢ t where, M is a number of the plurality of light receiving elements, {right arrow over (k)} is a wave vector representing the reflected wave incident on the optical receiver, {right arrow over (r)}m is a radius vector representing an m-th light receiving element among the plurality of light receiving elements, Δ is a wavelength of the electromagnetic wave reflected from the object, and c is the speed of light.