# US 2020/0396384 A1 · Systems, methods, and apparatuses for implementing an image sensor reconfiguration framework for seamless resolution-based tradeoffs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0396384 A1 - Document ID: US20200396384A1 - Status: Published application - Kind: A1 - Application number: 16/898,791 - Filed: 2020-06-11 - Published: 2020-12-17 - Earliest claimed: 2019-06-11 - Assignees: Arizona Board of Regents on behalf of Arizona State University - Applicants: Arizona Board of Regents on Behalf of Arizona State University - Inventors: Jinhan Hu; Robert LiKamWa; Alexander Shearer; Saranya Rajagopalan - Web page: https://patentgrep.com/patent/US20200396384A1 ## CPC - H04N 23/60: Control of cameras or camera modules - H04N 23/651: {for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera} - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06T 1/60: Memory management - H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image} - H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In accordance with embodiments disclosed herein, there are provided herein systems, methods, and apparatuses for implementing an image sensor reconfiguration framework for seamless resolution-based tradeoffs. For example, there is disclosed a system for performing a resolution reconfiguration of an image sensor without dropping frames; in which such a system includes: means for opening an image device having the image sensor embedded therein; means for setting the image sensor to a first sensor format: means for requesting and mapping buffers for a vision data stream from the image sensor in accordance with the first sensor format; means for receiving a resolution reconfiguration request to change the image sensor from the first sensor format to a second sensor format; and means for performing a parallel reconfiguration of the image sensor, wherein the image device continues to process the vision data stream from the image sensor in accordance with the first sensor format in parallel with the image device resetting the image sensor to the second sensor format. Other related embodiments are disclosed. ## Claims ### Claim 1 A method for performing a resolution reconfiguration of an image sensor hardware without dropping frames, wherein the method comprises: accessing image device circuitry having the image sensor hardware embedded therein; setting the image sensor hardware to operate utilizing a first sensor format; requesting and mapping buffers for a vision data stream from the image sensor hardware in accordance with the first sensor format; streaming the vision data stream from the image sensor hardware in accordance with the first sensor format; receiving a resolution reconfiguration request to modify operation of the image sensor hardware from the first sensor format to a second sensor format; and performing a parallel reconfiguration of the image sensor hardware, wherein the image device circuitry continues to process the vision data stream from the image sensor hardware in accordance with the first sensor format in parallel with the image device circuitry resetting the image sensor hardware to the second sensor format. ### Claim 2 (depends on 1) The method of claim 1: wherein performing the parallel reconfiguration of the image sensor hardware comprises transitioning streaming of the vision data stream from the image sensor hardware in accordance with the first sensor format to streaming the vision data stream from the image sensor in accordance with the second sensor format without dropping a single frame. ### Claim 3 (depends on 1) The method of claim 1: wherein performing the parallel reconfiguration of the image sensor hardware comprises transitioning streaming of the vision data stream via the first sensor format to streaming of the vision data stream via the second sensor format with frame-to-frame resolution reconfiguration latency being fully hidden from any application receiving the streaming of the vision data stream from the image sensor hardware. ### Claim 4 (depends on 1) The method of claim 1, wherein performing the parallel reconfiguration of the image sensor hardware comprises: executing a main thread for application processing of the vision data stream from the image sensor hardware; spawning a new reconfiguration thread to process reconfiguration requests received; and joining the main thread and the new reconfiguration thread before a capture thread for the image sensor hardware wakes to resume capturing frames from the image sensor. ### Claim 5 (depends on 1) The method of claim 1, wherein performing the parallel reconfiguration of the image sensor hardware further comprises: prohibiting reconfiguration of the image sensor hardware while a capture thread is actively capturing frames from the image sensor hardware. ### Claim 6 (depends on 5) The method of claim 5, further comprising: defining Tinterval as an interval between consecutive frame captures; identifying Tcapture as the time required for each frame capture; defining a reconfiguration timing budget Tbudget as a function of a frame interval Tinterval, where Tbudget is equal to frame interval Tinterval minus the required capture time Tcapture; and requiring the parallel reconfiguration of the image sensor hardware to complete within the defined reconfiguration timing budget Tbudget. ### Claim 7 A mobile computing device, comprising: a memory to store instructions; a processor to execute instructions stored within the memory an image device having an image sensor embedded therein; wherein the instructions cause the mobile device to perform operations including: opening the image device having the image sensor embedded therein; setting the image sensor to a first sensor format; requesting and mapping buffers for a vision data stream from the image sensor in accordance with the first sensor format; streaming the vision data stream from the image sensor in accordance with the first sensor format; receiving a resolution reconfiguration request to change the image sensor from the first sensor format to a second sensor format; and performing a parallel reconfiguration of the image sensor, wherein the image device continues to process the vision data stream from the image sensor in accordance with the first sensor format in parallel with the image device resetting the image sensor to the second sensor format. ### Claim 8 (depends on 7) The mobile computing device of claim 7: wherein performing the parallel reconfiguration of the image sensor comprises transitioning streaming of the vision data stream from the image sensor in accordance with the first sensor format to streaming the vision data stream from the image sensor in accordance with the second sensor format without dropping a single frame. ### Claim 9 (depends on 7) The mobile computing device of claim 7: wherein performing the parallel reconfiguration of the image sensor comprises transitioning streaming of the vision data stream via the first sensor format to streaming of the vision data stream via the second sensor format with frame-to-frame resolution reconfiguration latency being fully hidden from any application receiving the streaming of the vision data stream from the image sensor. ### Claim 10 (depends on 7) The mobile computing device of claim 7, wherein performing the parallel reconfiguration of the image sensor comprises: executing a main thread for application processing of the vision data stream from the image sensor; spawning a new reconfiguration thread to process reconfiguration requests received; and joining the main thread and the new reconfiguration thread before a capture thread for the image sensor wakes to resume capturing frames from the image sensor. ### Claim 11 (depends on 7) The mobile computing device of claim 7, wherein performing the parallel reconfiguration of the image sensor further comprises: prohibiting reconfiguration of the image sensor while a capture thread is actively capturing frames from the image sensor. ### Claim 12 (depends on 11) The mobile computing device of claim 11, wherein the instructions cause the mobile device to perform operations further including: defining Tinterval as an interval between consecutive frame captures; identifying Tcapture as the time required for each frame capture; defining a reconfiguration timing budget Tbudget as a function of a frame interval Tinterval, where Tbudget is equal to frame interval Tinterval minus the required capture time Tcapture; and requiring the parallel reconfiguration of the image sensor to complete within the defined reconfiguration timing budget Tbudget. ### Claim 13 Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a processor of a image device, the instructions cause the image device to perform operations including: opening the image device having an image sensor embedded therein; setting the image sensor to a first sensor format; requesting and mapping buffers for a vision data stream from the image sensor in accordance with the first sensor format; streaming the vision data stream from the image sensor in accordance with the first sensor format; receiving a resolution reconfiguration request to change the image sensor from the first sensor format to a second sensor format; and performing a parallel reconfiguration of the image sensor, wherein the image device continues to process the vision data stream from the image sensor in accordance with the first sensor format in parallel with the image device resetting the image sensor to the second sensor format. ### Claim 14 (depends on 13) The non-transitory computer readable storage media of claim 13: wherein performing the parallel reconfiguration of the image sensor comprises transitioning streaming of the vision data stream from the image sensor in accordance with the first sensor format to streaming the vision data stream from the image sensor in accordance with the second sensor format without dropping a single frame. ### Claim 15 (depends on 13) The non-transitory computer readable storage media of claim 13: wherein performing the parallel reconfiguration of the image sensor comprises transitioning streaming of the vision data stream via the first sensor format to streaming of the vision data stream via the second sensor format with frame-to-frame resolution reconfiguration latency being fully hidden from any application receiving the streaming of the vision data stream from the image sensor. ### Claim 16 (depends on 13) The non-transitory computer readable storage media of claim 13, wherein performing the parallel reconfiguration of the image sensor comprises: executing a main thread for application processing of the vision data stream from the image sensor; spawning a new reconfiguration thread to process reconfiguration requests received; and joining the main thread and the new reconfiguration thread before a capture thread for the image sensor wakes to resume capturing frames from the image sensor. ### Claim 17 (depends on 13) The non-transitory computer readable storage media of claim 13, wherein performing the parallel reconfiguration of the image sensor further comprises: prohibiting reconfiguration of the image sensor while a capture thread is actively capturing frames from the image sensor. ### Claim 18 (depends on 17) The non-transitory computer readable storage media of claim 17, wherein the instructions cause the image device to perform operations further comprising: defining Tinterval as an interval between consecutive frame captures; identifying Tcapture as the time required for each frame capture; defining a reconfiguration timing budget Tbudget as a function of a frame interval Tinterval, where Tbudget is equal to frame interval Tinterval minus the required capture time Tcapture; and requiring the parallel reconfiguration of the image sensor to complete within the defined reconfiguration timing budget Tbudget.