# US 12,223,663 B2 · Frame processing and/or capture instruction systems and techniques > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,663 B2 - Document ID: US12223663B2 - Status: Granted - Kind: B2 - Application number: 17/240,837 - Filed: 2021-04-26 - Published: 2022-05-05 - Granted: 2025-02-11 - Earliest claimed: 2020-10-30 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Mark Gintsburg; Nir Strauss; Ron Gaizman; Eran Pinhasov; Yuri Dolgin; Victor Pinto - Web page: https://patentgrep.com/patent/US12223663B2 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06T 7/215: Motion-based segmentation - G06V 10/10: Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60) - G06V 10/764: using classification, e.g. of video objects - H04N 5/144: {Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20)} - H04N 23/6811: {based on the image signal} - H04N 23/6812: {based on additional sensors, e.g. acceleration sensors} - H04N 23/72: Combination of two or more compensation controls - H04N 23/73: by influencing the exposure time - H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors - H04N 23/743: Bracketing, i.e. taking a series of images with varying exposure conditions - H04N 23/81: for suppressing or minimising disturbance in the image signal generation - G06N 3/048: Activation functions - G06T 2207/10144: Varying exposure - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20208: High dynamic range [HDR] image processing - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Techniques and systems are provided for processing one or more frames or images. For instance, a process for determining exposure for one or more frames includes obtaining a motion map for one or more frames. The process includes determining, based on the motion map, motion associated with the one or more frames of a scene. The motion corresponds to movement of one or more objects in the scene relative to a camera used to capture the one or more frames. The process includes determining, based on the determined motion, a number of frames and an exposure for capturing the number of frames. The process further includes sending a request to capture the number of frames using the determined exposure duration. ## Claims ### Claim 1 An apparatus for processing one or more frames, comprising: a memory; and one or more processors coupled to the memory and configured to: obtain a motion map for the one or more frames, wherein the motion map includes a plurality of pixels and, for each pixel, a value indicating a respective amount of motion for the respective pixel and a respective confidence value associated with the respective amount of motion for the respective pixel; determine, based on the motion map, motion associated with the one or more frames of a scene, the motion corresponding to movement of one or more objects in the scene; determine, based on one or more sensor measurements, global motion associated with a camera used to capture the one or more frames; determine a weighted combination of the determined motion and the global motion using a first weight for the determined motion and a second weight for the global motion, the first weight representing a sensitivity of frame processing to movement of the one or more objects in the scene; determine, based on the weighted combination of the determined motion and the global motion, an amount of motion in the one or more frames is less than a motion threshold; determine a number of frames and an exposure duration for capturing the number of frames, wherein the determined number of frames is less than a previously-captured number of frames used to generate a first frame for display and the determined exposure duration is greater than an exposure duration used to capture the previously-captured number of frames based on the amount of motion in the one or more frames being less than the motion threshold; send a request to capture the number of frames using the determined exposure duration; and output a second frame for display based on the number of frames captured using the determined exposure duration. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the one or more frames are obtained before a capture command for capturing the number of frames is received. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the one or more processors are configured to: perform temporal blending on the number of frames captured using the determined exposure duration to generate a temporally blended frame. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the one or more processors are configured to: perform, using a machine learning based image signal processor, spatial processing on the temporally blended frame. ### Claim 5 (depends on 4) The apparatus of claim 4, wherein the machine learning based image signal processor uses the motion map as input for performing the spatial processing on the temporally blended frame. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the determined exposure duration is based on a gain. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the number of frames and the exposure duration for capturing the number of frames are determined based on the determined motion and the global motion. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein the one or more processors are configured to: determine a final motion indication based on the determined motion and the global motion; and wherein the amount of motion is based on the final motion indication. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the final motion indication is based on the weighted combination of the determined motion and the global motion. ### Claim 10 (depends on 8) The apparatus of claim 8, wherein the one or more processors are configured to: determine, based on a final motion indication associated with at least one additional frame, an amount of motion in the at least one additional frame is greater than the motion threshold; and based on the amount of motion in the at least one additional frame being greater than the motion threshold, increase a frame count for the number of frames and decrease an exposure amount for the determined exposure duration. ### Claim 11 (depends on 1) The apparatus of claim 1, further comprising at least one of a camera configured to capture at least one frame and a display configured to display the first frame and the second frame. ### Claim 12 A method of processing one or more frames, the method comprising: obtaining a motion map for the one or more frames, wherein the motion map includes-a plurality of pixels and, for each pixel, a value indicating a respective amount of motion for the respective pixel and a respective confidence value associated with the respective amount of motion for the respective pixel; determining, based on the motion map, motion associated with the one or more frames of a scene, the motion corresponding to movement of one or more objects in the scene; determining, based on one or more sensor measurements, global motion associated with a camera used to capture the one or more frames; determining a weighted combination of the determined motion and the global motion using a first weight for the determined motion and a second weight for the global motion, the first weight representing a sensitivity of frame processing to movement of the one or more objects in the scene; determining, based on the weighted combination of the determined motion and the global motion, an amount of motion in the one or more frames is less than a motion threshold; determining a number of frames and an exposure duration for capturing the number of frames, wherein the determined number of frames is less than a previously-captured number of frames used to generate a first frame for display and the determined exposure duration is greater than an exposure duration used to capture the previously-captured number of frames based on the amount of motion in the one or more frames being less than the motion threshold; sending a request to capture the number of frames using the determined exposure duration; and displaying a second frame based on the number of frames captured using the determined exposure duration. ### Claim 13 (depends on 12) The method of claim 12, wherein the one or more frames are obtained before a capture command for capturing the number of frames is received. ### Claim 14 (depends on 12) The method of claim 12, further comprising: performing temporal blending on the number of frames captured using the determined exposure duration to generate a temporally blended frame. ### Claim 15 (depends on 14) The method of claim 14, further comprising: performing, using a machine learning based image signal processor, spatial processing on the temporally blended frame. ### Claim 16 (depends on 15) The method of claim 15, wherein the machine learning based image signal processor uses the motion map as input for performing the spatial processing on the temporally blended frame. ### Claim 17 (depends on 12) The method of claim 12, wherein the determined exposure duration is based on a gain. ### Claim 18 (depends on 12) The method of claim 12, wherein the number of frames and the exposure duration for capturing the number of frames are determined based on the determined motion and the global motion. ### Claim 19 (depends on 18) The method of claim 18, further comprising: determining a final motion indication based on the determined motion and the global motion; wherein the amount of motion is based on the final motion indication. ### Claim 20 (depends on 19) The method of claim 19, wherein the final motion indication is based on the weighted combination of the determined motion and the global motion. ### Claim 21 (depends on 19) The method of claim 19, further comprising: determining, based on a final motion indication associated with at least one additional frame, an amount of motion in the at least one additional frame is greater than the motion threshold; and based on the amount of motion in the at least one additional frame being greater than the motion threshold, increasing a frame count for the number of frames and decreasing an exposure duration amount for the determined exposure duration. ### Claim 22 A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtain a motion map for one or more frames, wherein the motion map includes-a plurality of pixels and, for each pixel, a value indicating a respective amount of motion for the respective pixel and a respective confidence value associated with the respective amount of motion for the respective pixel; determine, based on the motion map, motion associated with the one or more frames of a scene, the motion corresponding to movement of one or more objects in the scene; determine, based on one or more sensor measurements, global motion associated with a camera used to capture the one or more frames; determine a weighted combination of the determined motion and the global motion using a first weight for the determined motion and a second weight for the global motion, the first weight representing a sensitivity of frame processing to movement of the one or more objects in the scene; determine, based on the weighted combination of the determined motion and the global motion, an amount of motion in the one or more frames is less than a motion threshold; determine a number of frames and an exposure duration for capturing the number of frames, wherein the determined number of frames is less than a previously-captured number of frames used to generate a first frame for display and the determined exposure duration is greater than an exposure duration used to capture the previously-captured number of frames based on the amount of motion in the one or more frames being less than the motion threshold; send a request to capture the number of frames using the determined exposure duration; and output a second frame for display based on the number of frames captured using the determined exposure duration. ### Claim 23 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein the one or more frames are obtained before a capture command for capturing the number of frames is received. ### Claim 24 (depends on 22) The non-transitory computer-readable storage medium of claim 22, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: perform temporal blending on the number of frames captured using the determined exposure duration to generate a temporally blended frame. ### Claim 25 (depends on 22) The non-transitory computer-readable storage medium of claim 22, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determine a final motion indication based on the determined motion and the global motion; wherein the amount of motion is based on the final motion indication. ### Claim 26 (depends on 25) The non-transitory computer-readable storage medium of claim 25, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determine, based on a final motion indication associated with at least one additional frame, whether an amount of motion in the at least one additional frame is greater than the motion threshold; and increase a frame count for the number of frames and decrease an exposure amount for the determined exposure duration based on the amount of motion in the at least one additional frame is greater than the motion threshold.