# US 11,908,100 B2 · Transform matrix learning for multi-sensor image capture devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,100 B2 - Document ID: US11908100B2 - Status: Granted - Kind: B2 - Application number: 17/201,660 - Filed: 2021-03-15 - Published: 2022-09-15 - Granted: 2024-02-20 - Earliest claimed: 2021-03-15 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Shizhong Liu; Jincheng Huang; Weiliang Liu; Jianfeng Ren; Sahul Madanayakanahalli Phaniraj Venkatesh - Web page: https://patentgrep.com/patent/US11908100B2 ## CPC - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 5/80: Geometric correction - G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12) - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 7/55: from multiple images - G06T 7/571: from focus - H04N 5/268: Signal distribution or switching - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20208: High dynamic range [HDR] image processing - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Artefacts in a sequence of image frames may be reduced or eliminated through modification of an input image frame to match another image frame in the sequence, such as by geometrically warping to generate a corrected image frame with a field of view matched to another frame in sequence of frames with the image frame. The warping may be performed based on a model generated from data regarding the multi-sensor device. The disparity between image frames may be modeled based on image captures from the first and second image sensor for scenes at varying depths. The model may be used to predict disparity values for captured images, and those predicted disparity values used to reduce artefacts resulting from image sensor switching. The predicted disparity values may be used in image conditions resulting in erroneous actual disparity values. ## Claims ### Claim 1 A method, comprising: receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for a disparity between the first image sensor and the second image sensor at a plurality of depth values, the model based, at least in part, on the first disparity value and the first depth; receiving an input image from one of the first image sensor or the second image sensor; receiving a second depth value corresponding to the input image; determining a predicted disparity value based on the second depth value and the model; determining a transformation matrix for warping the input image to a field of view of the other of the first image sensor or the second image sensor; determining a confidence level associated with the transformation matrix is below a threshold level; and determining a corrected image frame based on the input image and the predicted disparity value. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the corrected image comprises warped warping the input image received from one of the first image sensor or the second image sensor to match a field of view of the other of the first image sensor or the second image sensor. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the transformation matrix comprises determining the transformation matrix with computer vision processing (CVP). ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining an image characteristic of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the image characteristic is below the threshold level. ### Claim 5 (depends on 1) The method of claim 1, further comprising: determining a brightness of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the brightness is below the threshold level. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining a video sequence comprising a first image from the first image sensor, a second image from the second image sensor, and the corrected image, wherein the corrected image appears in the video sequence between the first image and the second image. ### Claim 7 (depends on 1) The method of claim 1, wherein the step of determining the corrected image is further based on an image of the other of the first image sensor or the second image sensor. ### Claim 8 (depends on 1) The method of claim 1, wherein: the first disparity value indicates a difference in field, of view along a first axis, and the method further comprising: receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, wherein the step of determining the model is further based, at least in part, on the second disparity value. ### Claim 9 (depends on 1) The method of claim 1, wherein determining the model comprises: storing a plurality of disparity values, wherein the model is based on the plurality of disparity values; and replacing a previous value of the plurality of disparity values with the first disparity value based on at least one of a number of values in the plurality of disparity values or a time associated with the previous value. ### Claim 10 (depends on 1) The method of claim 1, wherein the first depth value comprises an auto-focus depth corresponding to the input image. ### Claim 11 (depends on 1) The method of claim 1, further comprising determining the first depth value based on range imaging. ### Claim 12 (depends on 11) The method of claim 11, wherein determining the first depth value comprises determining the first depth value based on a time of flight (ToF) measurement. ### Claim 13 (depends on 11) The method of claim 11, wherein determining the first depth value comprises determining the first depth value based on a light detection and ranging (LIDAR) measurement. ### Claim 14 A device, comprising: one or more processors; and a memory coupled to the one or more processors and storing instruction that, when executed by the one or more processors, cause the device to perform operations comprising: receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for a disparity between the first image sensor and the second image sensor at a plurality of depth values, the model based, at least, in part, on the first disparity value and the first depth; receiving an input image from one of the first image sensor or the second image sensor; receiving a second depth value corresponding to the input image; determining a predicted disparity value based on the second depth value and the model; determining a transformation matrix for warping the input image to a field of view of the other of the first image sensor or the second image sensor; determining a confidence level associated with the transformation matrix is below a threshold level; and determining a corrected image based on the input image and the predicted disparity value. ### Claim 15 (depends on 14) The device of claim 14, wherein determining the corrected image comprises warping the input image received from one of the first image sensor or the second image sensor to match a field of view of the other of the first image sensor or the second image sensor. ### Claim 16 (depends on 14) The device of claim 14, wherein determining the transformation matrix comprises determining the transformation matrix with computer vision processing (CVP). ### Claim 17 (depends on 14) The device of claim 14, wherein the instructions cause the device to perform operations further comprising: determining an image characteristic of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the image characteristic is below the threshold level. ### Claim 18 (depends on 14) The device of claim 14, wherein the instructions cause the device to perform operations further comprising: determining a brightness of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the brightness is below the threshold level. ### Claim 19 (depends on 14) The device of claim 14, wherein the instructions cause the device to perform operations further comprising: determining a video sequence comprising a first image from the first image sensor, a second image from the second image sensor, and the corrected image, wherein the corrected image appears in the video sequence between the first image and the second image. ### Claim 20 (depends on 14) The device of claim 14, wherein the step of determining the corrected image is further based on an image of the other of the first image sensor or the second image sensor. ### Claim 21 (depends on 14) The device of claim 14, wherein: the first disparity value indicates a difference in field of view along a first axis, and the instructions cause the device to perform operations further comprising: receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, wherein the step of determining the model is, further based, at least in part, on the second disparity value. ### Claim 22 (depends on 14) The device of claim 14, wherein determining the model comprises: storing a plurality of disparity values, wherein the model is based on the plurality of disparity values; and replacing a previous value of the plurality of disparity values with the first disparity value based on at least one of a number of values in the plurality of disparity values or a time associated with the previous value. ### Claim 23 (depends on 14) The device of claim 14, wherein the first depth value comprises an auto-focus depth corresponding to the input image. ### Claim 24 (depends on 14) The device of claim 14, wherein the instructions cause the device to perform operations further comprising determining the first depth value based on range imaging. ### Claim 25 (depends on 24) The device of claim 24, wherein determining the first depth value comprises determining the first depth value based on a time of flight (ToF) measurement. ### Claim 26 (depends on 24) The device of claim 24, wherein determining the first depth value comprises determining the first depth value based on a light detection and ranging (LIDAR) measurement. ### Claim 27 A non-transitory computer-readable medium storing instructions that, when executed by a processor of a device, cause the device to perform operations comprising: receiving a first disparity value indicating a difference in field of view between a first image sensor and a second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for a disparity between the first image sensor and the second image sensor at a plurality of depth values, the model based, at least in part, on the first disparity value and the first depth; receiving an input image from one of the first image sensor or the second image sensor; receiving a second depth value corresponding to the input image; determining a predicted disparity value based on the second depth value and the model; determining a transformation matrix for warping the input image to a field of view of the other of the first image sensor or the second image sensor; determining a confidence level associated with the transformation matrix is below a threshold level; and determining a corrected image based on the input image and the predicted disparity value. ### Claim 28 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein determining the corrected image comprises warping the input image received from one of the first image sensor or the second image sensor to match a field of view of the other of the first image sensor or the second image sensor. ### Claim 29 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein determining the transformation matrix comprises determining the transformation matrix with computer vision processing (CVP). ### Claim 30 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein the instructions, when executed by a processor of a device, cause the device to perform operations further comprising: determining an image characteristic of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the image characteristic is below the threshold level. ### Claim 31 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein the instructions, when executed by a processor of a device, cause the device to perform operations further comprising: determining a brightness of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the brightness is below the threshold level. ### Claim 32 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein the instructions, when executed by a processor of a device, cause the device to perform operations further comprising: determining a video sequence comprising a first image from the first image sensor, a second image from the second image sensor, and the corrected image, wherein the corrected image appears in the video sequence between the first image and the second image. ### Claim 33 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein the step of determining the corrected image is further based on an image of the other of the first image sensor or the second image sensor. ### Claim 34 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein: the first disparity value indicates a difference in field of view along a first axis, and the instructions, when executed by a processor of a device, cause the device to perform operations further comprising: receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, wherein the step of determining the model is further based, at least in part, on the second disparity value. ### Claim 35 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein determining the model comprises: storing a plurality of disparity values, wherein the model is based on the plurality of disparity values; and replacing a previous value of the plurality of disparity values with the first disparity value based on at least one of a number of values in the plurality of disparity values or a time associated with the previous value. ### Claim 36 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein the first depth value comprises an auto-focus depth corresponding to the input image. ### Claim 37 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein the instructions, when executed by a processor of a device, cause the device to perform operations further comprising: determining the first depth value based on range imaging. ### Claim 38 (depends on 37) The non-transitory computer-readable medium of claim 37, wherein determining the first depth value comprises determining the first depth value based on a time of flight (ToF) measurement. ### Claim 39 (depends on 37) The non-transitory computer-readable medium of claim 37, wherein determining the first depth value comprises determining the first depth value based on a light detection and ranging (LIDAR) measurement. ### Claim 40 A device, comprising: a first image sensor configured with a first field of view; a second image sensor configured with a second field of view at least partially overlapping the first field of view; one or more processors coupled to the first image sensor and coupled to the second image sensor; and a memory coupled to the one or more processors, wherein the one or more processors are configured to perform steps comprising: receiving a first disparity value indicating a difference in field of view between the first image sensor and the second image sensor; receiving a first depth value corresponding to the first disparity value; determining a model for a disparity between the first image sensor and the second image sensor at a plurality of depth values, the model based, at least in part, on the first, disparity value and the first depth; receiving an input image from one of the first image sensor or the second image sensor; receiving a second depth value corresponding to the input image; determining a predicted disparity value based on the second depth value and the model; determining a transformation matrix for warping the input image to a field of view of the other of the first image sensor or the second image sensor; determining a confidence level associated with the transformation matrix is below a threshold level; and determining a corrected image based on the input image and the predicted disparity value. ### Claim 41 (depends on 40) The device of claim 40, wherein determining the corrected image comprises warping the input image received from one of the first image sensor or the second image sensor to match a field of view of the other of the first image sensor or the second image sensor. ### Claim 42 (depends on 40) The device of claim 40, wherein determining the transformation matrix comprises determining the transformation matrix with computer vision processing (CVP). ### Claim 43 (depends on 40) The device of claim 40, wherein the one or more processors are further configured to perform steps comprising: determining an image characteristic of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the image characteristic is below the threshold level. ### Claim 44 (depends on 40) The device of claim 40, wherein the one or more processors are further configured to perform steps comprising: determining a brightness of the input image is below a threshold level, wherein the step of determining the corrected image is performed based on determining the brightness is below the threshold level. ### Claim 45 (depends on 40) The device of claim 40, wherein the one or more processors are further configured to perform steps comprising: determining a video sequence comprising a first image from the first image sensor, a second image from the second image sensor, and the corrected image, wherein the corrected image appears in the video sequence between the first image and the second image. ### Claim 46 (depends on 40) The device of claim 40, wherein the step of determining the corrected image is further based on an image of the other of the first image sensor or the second image sensor. ### Claim 47 (depends on 40) The device of claim 40, wherein: the first disparity value indicates a difference in field of view along a first axis, and the one or more processors are configured to perform steps further comprising: receiving a second disparity value indicating a difference in field of view between the first image sensor and the second image sensor along a second axis different from the first axis, wherein the step of determining the model is further based, at least in part, on the second disparity value. ### Claim 48 (depends on 40) The device of claim 40, wherein determining the model comprises: storing a plurality of disparity values, wherein the model is based on the plurality of disparity values; and replacing a previous value of the plurality of disparity values with the first disparity value based on at least one of a number of values in the plurality of disparity values or a time associated with the previous value. ### Claim 49 (depends on 40) The device of claim 40, further comprising a depth sensor, wherein the one or more processors are coupled to the depth sensor and is configured to receive the first depth value from the depth sensor. ### Claim 50 (depends on 49) The device of claim 49, wherein the depth sensor comprises a range imaging system. ### Claim 51 (depends on 50) The device of claim 50, wherein the depth sensor comprises a time of flight (ToF) system, and wherein determining the first depth value comprises determining the first depth value based on a ToF measurement from the ToF system. ### Claim 52 (depends on 50) The device of claim 50, wherein the depth sensor comprises a light detection and ranging (LIDAR) system, and wherein determining the first depth value comprises determining the first depth value based on a LIDAR measurement from the LIDAR system. ### Claim 53 (depends on 40) The device of claim 40, further comprising: a computer vision processor (CVP) coupled to the one or more processors and configured to perform operations comprising: receiving a first image frame from the first image sensor; receiving a second image frame from the second image sensor; and feature matching between the first image frame and the second image frame to determine a difference in field of view between the first image sensor and the second image sensor, wherein the processor is configured to receive the difference in field of view between the first image sensor and the second image sensor as the first disparity value from the CVP.