# US 2016/0366348 A1 · Using a Depth Map of a Monitored Scene to Identify Floors, Walls, and Ceilings > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2016/0366348 A1 - Document ID: US20160366348A1 - Status: Published application - Kind: A1 - Application number: 14/745,259 - Filed: 2015-06-19 - Published: 2016-12-15 - Earliest claimed: 2015-06-12 - Assignees: Google Inc. - Applicants: Google Inc. - Inventors: MICHAEL DIXON; DONGEEK SHIN - Web page: https://patentgrep.com/patent/US20160366348A1 ## CPC - H04N 7/188: {Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253)} - G06F 18/23: Clustering techniques - G06V 10/141: Control of illumination - G06V 10/143: Sensing or illuminating at different wavelengths - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06V 30/242: Division of the character sequences into groups prior to recognition; Selection of dictionaries - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - H04N 7/183: {for receiving images from a single remote source} - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - H04N 23/20: for generating image signals from infrared radiation only - H04N 23/56: provided with illuminating means - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N 23/74: by influencing the scene brightness using illuminating means - G06V 20/44: {Event detection} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A camera system includes memory, a lens assembly to direct light from a scene onto an image sensing element, an image sensing element configured to receive light from the scene via the lens assembly, at least one infrared illuminator configured to transmit infrared light, and a processor, coupled to the image sensing element and the at least one infrared illuminator. The processor is configured to operate the illuminators and the image sensing element in a first mode whereby infrared light transmitted by the illuminators and reflected from the scene is used to generate a two-dimensional image of the scene. The processor is also configured to operate the illuminators and the image sensing element in a second mode whereby infrared light transmitted by the illuminators and reflected from the scene is used to identify a planar surface in the scene. ## Claims ### Claim 1 A camera device, comprising: a memory configured to store information; a lens assembly arranged to direct light from a scene; a plurality of image sensing elements configured to receive light from the scene via the lens assembly; a plurality of infrared illuminators configured to transmit infrared light; and a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators and configured to: operate the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and operate the plurality of infrared illuminators and the image sensing element in a second mode, including: sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene; for each subset of illuminators, measuring an intensity of reflected light from the scene for each image sensing element of the plurality of image sensing elements; for each image sensing element: generating a light intensity vector based on the measured intensities for the element, wherein the light intensity vector has a component corresponding to each subset of illuminators; and estimating a depth in the scene using the light intensity vector and a lookup table; and identifying a planar surface in the scene using the estimated depths. ### Claim 2 (depends on 1) The camera device of claim 1, wherein the first mode is operated at night. ### Claim 3 (depends on 1) The camera device of claim 1, wherein in the first mode all of the infrared illuminators are activated to generate the two-dimensional image. ### Claim 4 (depends on 1) The camera device of claim 1, wherein the second mode is a three-dimensional surface detection mode. ### Claim 5 (canceled) ### Claim 6 (canceled) ### Claim 7 (depends on 1) The camera device of claim 1, wherein each of the subsets of illuminators is at a different angle relative to the plurality of image sensing elements. ### Claim 8 (depends on 1) The camera device of claim 1, wherein the processor is further configured to: aggregate the estimated pixel depths into a data structure for identifying planar surfaces in the scene. ### Claim 9 (depends on 8) The camera device of claim 8, wherein identifying planar surfaces in the scene using the estimated pixel depths comprises transmitting the data structure to a planar surface detection module configured to identify planar surfaces using a depth map constructed from the data structure. ### Claim 10 (depends on 8) The camera device of claim 8, wherein identifying planar surfaces in the scene using the estimated pixel depths comprises: using the data structure to identify a region in the scene; and fitting a plane to a set of points in the region. ### Claim 11 (depends on 10) The camera device of claim 10, wherein the processor is further configured to: determine that the region corresponds to a planar surface in the scene when a plane fitting residual error for the fitted plane is less than a predefined value. ### Claim 12 (depends on 8) The camera device of claim 8, wherein the processor is further configured to: construct a depth map of the scene using the data structure; use the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map identifies which points in the depth map comprise depth discontinuities; and select a contiguous region from the binary depth edge map. ### Claim 13 (depends on 10) The camera device of claim 10, wherein the processor is further configured to determine an orientation of the fitted plane. ### Claim 14 (depends on 13) The camera device of claim 13, wherein the orientation of the plane is upwards, and the fitted plane is determined to be a floor. ### Claim 15 (depends on 13) The camera device of claim 13, wherein the orientation of the plane is downwards, and the fitted plane is determined to be a ceiling. ### Claim 16 (depends on 13) The camera device of claim 13, wherein the orientation of the plane is horizontal, and the fitted plane is determined to be a wall. ### Claim 17 (canceled) ### Claim 18 (canceled) ### Claim 19 A method, comprising: at a camera device having a memory configured to store information, a lens assembly arranged to direct light from a scene, a plurality of image sensing elements configured to receive light from the scene via the lens assembly, a plurality of infrared illuminators configured to transmit infrared light, and a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators: operating the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and operating the plurality of infrared illuminators and the image sensing element in a second mode, including: sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene; for each subset of illuminators, measuring an intensity of reflected light from the scene for each pixel of a plurality of pixels, wherein each pixel corresponds to one or more image sensing elements of the plurality of image sensing elements; for each pixel: generating a light intensity vector based on the measured intensities for the pixel, wherein the light intensity vector has a component corresponding to each subset of illuminators; and estimating a pixel depth in the scene using the light intensity vector and a lookup table; and identifying a planar surface in the scene using the estimated pixel depths. ### Claim 20 A non-transitory computer-readable storage medium storing one or more programs configured for execution by a camera device having a memory configured to store information, a lens assembly arranged to direct light from a scene, plurality of image sensing elements configured to receive light from the scene via the lens assembly, a plurality of infrared illuminators configured to transmit infrared light, and a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators, wherein the one or more programs comprise instructions for: operating the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and operating the plurality of infrared illuminators and the image sensing element in a second mode, including: sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene; for each subset of illuminators, measuring an intensity of reflected light from the scene for each pixel of a plurality of pixels, wherein each pixel corresponds to one or more image sensing elements of the plurality of image sensing elements; for each pixel: generating a light intensity vector based on the measured intensities for the pixel, wherein the light intensity vector has a component corresponding to each subset of illuminators; and estimating a pixel depth in the scene using the light intensity vector and a lookup table; and identifying a planar surface in the scene using the estimated pixel depths. ### Claim 21 (depends on 19) The method of claim 19, wherein the first mode is configured to operate at night. ### Claim 22 (depends on 19) The method of claim 19, further comprising aggregating the estimated pixel depths into a data structure for identifying planar surfaces in the scene. ### Claim 23 (depends on 20) The non-transitory computer-readable storage medium of claim 20, wherein the first mode is configured to operate at night. ### Claim 24 (depends on 20) The non-transitory computer-readable storage medium of claim 20, wherein the one or more programs further comprise instructions for aggregating the estimated pixel depths into a data structure for identifying planar surfaces in the scene.