# US 12,001,024 B2 · Energy-efficient adaptive 3D sensing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,024 B2 - Document ID: US12001024B2 - Status: Granted - Kind: B2 - Application number: 18/299,923 - Filed: 2023-04-13 - Published: 2024-04-18 - Granted: 2024-06-04 - Earliest claimed: 2022-10-12 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Jian Wang; Sizhuo Ma; Brevin Tilmon; Yicheng Wu; Gurunandan Krishnan Gorumkonda; Ramzi Zahreddine; Georgios Evangelidis - Web page: https://patentgrep.com/patent/US12001024B2 ## CPC - G02B 27/0172: {characterised by optical features} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An energy-efficient adaptive 3D sensing system. The adaptive 3D sensing system includes one or more cameras and one or more projectors. The adaptive 3D sensing system captures images of a real-world scene using the one or more cameras and computes depth estimates and depth estimate confidence values for pixels of the images. The adaptive 3D sensing system computes an attention mask based on the one or more depth estimate confidence values and commands the one or more projectors to send a distributed laser beam into one or more areas of the real-world scene based on the attention mask. The adaptive 3D sensing system captures 3D sensing image data of the one or more areas of the real-world scene and generates 3D sensing data for the real-world scene based on the 3D sensing image data. ## Claims ### Claim 1 A system comprising: one or more cameras; one or more projectors; one or more processors operably connected to the one or more cameras and the one or more projectors; and a memory operably connected to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: capturing, using the one or more cameras, image data of a real-world scene; computing an attention mask based on the image data; commanding the one or more projectors to send a distributed laser beam into an area of the real-world scene based on the attention mask; capturing, using the one or more cameras, three-dimensional (3D) sensing image data of the area of the real-world scene while one or more projectors are sending the distributed laser beam into the area of the real-world scene; and generating 3D sensing data for the real-world scene based on the 3D sensing image data. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more projectors are Spatial Light Modulator (SLM)-based projectors. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more projectors are Microelectromechanical System (MEMS)+Diffractive Optical Element (DOE)-based projectors. ### Claim 4 (depends on 3) The system of claim 3, wherein the one or more MEMS+DOE-based projectors generates the distributed laser beam to include a randomized pattern. ### Claim 5 (depends on 1) The system of claim 1, wherein the operations further comprise: determining the area of the real-world scene based on a depth estimate confidence value. ### Claim 6 (depends on 1) The system of claim 1, wherein the operations further comprise: determining the area of the real-world scene based on a location in the real-world scene of a virtual object of an eXtended Reality (XR) user interface of an XR application. ### Claim 7 (depends on 1) The system of claim 1, wherein the operations further comprise: generating a 3D model of the real-world scene based on the image data of the real-world scene; determining that the area of the real-world scene is not mapped in the 3D model based on location data of the system; and specifying the area of the real-world scene in response to determining that the area of the real-world scene is not mapped in the 3D model. ### Claim 8 (depends on 1) The system of claim 1, wherein the system further comprises a head-worn device. ### Claim 9 (depends on 1) The system of claim 1, wherein the system further comprises a smartphone. ### Claim 10 A computer-implemented method comprising: capturing, using one or more cameras of a system, image data of a real-world scene; computing an attention mask based on the image data; commanding one or more projectors of the system to send a distributed laser beam into an area of the real-world scene based on the attention mask; capturing, using the one or more cameras, three-dimensional (3D) sensing image data of the area of the real-world scene while one or more projectors are sending the distributed laser beam into the area of the real-world scene; and generating 3D sensing data for the real-world scene based on the 3D sensing image data. ### Claim 11 (depends on 10) The computer-implemented method for a system of claim 10, wherein the one or more projectors are Spatial Light Modulator (SLM)-based projectors. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein the one or more projectors are Microelectromechanical System (MEMS)+Diffractive Optical Element (DOE)-based projectors. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the one or more MEMS+DOE-based projectors generates the distributed laser beam to include a randomized pattern. ### Claim 14 (depends on 10) The computer-implemented method of claim 10, further comprising: determining the area of the real-world scene based on a depth estimate confidence value. ### Claim 15 (depends on 10) The computer-implemented method of claim 10, further comprising: determining the area of the real-world scene based on a location in the real-world scene of a virtual object of an eXtended Reality (XR) user interface of an XR application. ### Claim 16 (depends on 10) The computer-implemented method of claim 10, further comprising: generating a 3D model of the real-world scene based on the image data of the real-world scene; determining that the area of the real-world scene is not mapped in the 3D model based on location data of the system; and specifying the area of the real-world scene in response to determining that the area of the real-world scene is not mapped in the 3D model. ### Claim 17 (depends on 10) The computer-implemented method of claim 10, wherein the system further comprises a head-worn device. ### Claim 18 (depends on 10) The computer-implemented method of claim 10, wherein the system further comprises a smartphone. ### Claim 19 A non-transitory computer-storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising: capturing, using one or more cameras, image data of a real-world scene; computing an attention mask based on the image data; commanding one or more projectors to send a distributed laser beam into an area of the real-world scene based on the attention mask; capturing, using the one or more cameras, three-dimensional (3D) sensing image data of the area of the real-world scene while one or more projectors are sending the distributed laser beam into the area of the real-world scene; and generating 3D sensing data for the real-world scene based on the 3D sensing image data. ### Claim 20 (depends on 19) The non-transitory computer-storage medium of claim 19, wherein the operations further comprise: determining the area of the real-world scene based on a depth estimate confidence value.