# US 12,015,842 B2 · Multi-purpose cameras for simultaneous capture and CV on wearable AR devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,015,842 B2 - Document ID: US12015842B2 - Status: Granted - Kind: B2 - Application number: 17/460,244 - Filed: 2021-08-29 - Published: 2022-03-31 - Granted: 2024-06-18 - Earliest claimed: 2020-09-30 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Bo Ding; Chintan Doshi; Alexander Kane; John James Robertson; Dmitry Ryuma - Web page: https://patentgrep.com/patent/US12015842B2 ## CPC - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 23/651: {for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera} - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56) - H04N 5/265: Mixing - H04N 13/296: Synchronisation thereof; Control thereof - H04N 23/65: Control of camera operation in relation to power supply - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Eyewear having an image signal processor (ISP) dynamically operable in a camera pipeline for augmented reality (AR) and computer vision (CV) systems. Multi-purpose cameras are used for simultaneous image capture and CV on wearable AR devices. The cameras are coupled to a frame and configured to generate images, wherein the cameras and the ISP are configured to operate in a first AR mode and capture images having a first resolution suitable for use in AR, and are configured to operate in a second CV mode to provide the images having a second resolution suitable for use in CV. The first resolution in the AR mode is higher than the second resolution in the CV mode, and the cameras and the ISP consume less power in the second CV mode than the first AR mode. The cameras and the ISP save significant system power by operating in the low power mode CV mode. ## Claims ### Claim 1 Eyewear, comprising: a frame; a display supported by the frame; a plurality of cameras coupled to the frame and configured to generate images; and a pipeline including an image signal processor (ISP) configurable to process the images for augmented reality (AR) and for computer vision (CV); wherein each of the plurality of cameras are configured to operate in a first mode and provide the images having a first resolution operable in AR, and wherein each of the plurality of cameras are configured to operate in a second mode and provide the images having a second resolution operable in CV, wherein the first resolution is higher than the second resolution; wherein the ISP is configured to stitch and synchronize the images from each of the plurality of cameras in the second mode in CV; and wherein the ISP is configured to process the images and operate at a first power in the first mode and at a second power in the second mode, wherein the second power is lower than the first power. ### Claim 2 (depends on 1) The eyewear of claim 1, wherein the pipeline has a first portion configured to process the images in the first mode, and a second portion configured to process the images in the second mode. ### Claim 3 (depends on 2) The eyewear of claim 2, wherein the ISP is configured to generate stitched, synchronized images. ### Claim 4 (depends on 3) The eyewear of claim 3, wherein the ISP is configured to match the frames from the plurality of cameras and put them in a buffer. ### Claim 5 (depends on 1) The eyewear of claim 1, wherein the ISP has CPU/DDR (central processing unit/double data read) rails, wherein the ISP is configured to scale up frequencies and voltages of the CPU/DDR rails in the first mode. ### Claim 6 (depends on 1) The eyewear of claim 1, wherein the cameras are configured to capture the images in the first resolution, and to downscale the images to the second resolution. ### Claim 7 (depends on 1) The eyewear of claim 1, wherein the ISP is configured to selectively turn a first of the plurality of cameras off in the second mode. ### Claim 8 (depends on 7) The eyewear of claim 7, wherein the ISP is configured to selectively turn the first camera back on and synchronize the plurality of cameras. ### Claim 9 A method of operating eyewear comprising a frame, a display supported by the frame, a plurality of cameras coupled to the frame and configured to generate images, and a pipeline including an image signal processor (ISP) configurable to process the images in augmented reality (AR) and in computer vision (CV); wherein: each of the plurality of cameras operate in a first mode and generate the images having a first resolution operable in AR, and each of the plurality of cameras operate in a second mode and provide the images having a second resolution operable in CV, wherein the first resolution is higher than the second resolution; the ISP stitches and synchronizes the images from each of the plurality of cameras in the second mode in CV; and the ISP processes the images and operates at a first power in the first mode and at a second power in the second mode, wherein the second power is lower than the first power. ### Claim 10 (depends on 9) The method of claim 9, wherein the pipeline includes a first portion processing the images in the first mode, and a second portion processing the images in the second mode. ### Claim 11 (depends on 10) The method of claim 10, wherein the ISP generates stitched, synchronized images. ### Claim 12 (depends on 11) The method of claim 11 wherein the ISP matches the frames from the plurality of cameras and puts them in a buffer. ### Claim 13 (depends on 9) The method of claim 9, wherein the ISP has CPU/DDR (central processing unit/double data read) rails, wherein the ISP scales up frequencies and voltages of the CPU/DDR rails in the first mode. ### Claim 14 (depends on 9) The method of claim 9, wherein the cameras capture the images in the first resolution, and downscale the images to the second resolution. ### Claim 15 (depends on 9) The method of claim 9, wherein the ISP selectively turns a first of the plurality of cameras off in the second mode. ### Claim 16 (depends on 15) The method of claim 15, wherein the ISP selectively turns the first camera back on and synchronizes the plurality of cameras. ### Claim 17 A non-transitory computer-readable medium storing program code which, when executed by a pipeline including an image signal processor (ISP) of eyewear having a frame, a display supported by the frame, and a plurality of cameras coupled to the frame and configured to generate images, is operative to cause the ISP to selectively process the images in augmented reality (AR) and in computer vision (CV), to perform the steps of: each of the plurality of cameras operating in a first mode such that each of the plurality of cameras generate the images having a first resolution operable in AR, and each of the plurality of cameras operating in a second mode such that each of the plurality of cameras provide the images having a second resolution operable in CV, wherein the first resolution is higher than the second resolution; the ISP stitching and synchronizing the images from each of the cameras in the second mode in CV; and process the images and operate at a first power in the first mode and at a second power in the second mode, wherein the second power is lower than the first power. ### Claim 18 (depends on 17) The non-transitory computer-readable medium as specified in claim 17, wherein the program code, when executed by the ISP, is operative to have the pipeline having a first portion processing the images in the first mode, and a second portion processing the images in the second mode. ### Claim 19 (depends on 18) The non-transitory computer-readable medium as specified in claim 18, wherein the program code, when executed by the ISP, is operative to generate stitched, synchronized images. ### Claim 20 (depends on 17) The non-transitory computer-readable medium as specified in claim 17, wherein the program code, when executed, is operative to cause the ISP to scale up frequencies and voltages of CPU/DDR (central processing unit/double data read) rails in the first mode.