# US 2025/0095182 A1 · Unified simultaneous optical flow and depth estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0095182 A1 - Document ID: US20250095182A1 - Status: Published application - Kind: A1 - Application number: 18/468,656 - Filed: 2023-09-15 - Published: 2025-03-20 - Earliest claimed: 2023-09-15 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Jisoo JEONG; Hong CAI; Babak EHTESHAMI BEJNORDI; Risheek GARREPALLI; Rajeev YASARLA; Fatih Murat PORIKLI - Web page: https://patentgrep.com/patent/US20250095182A1 ## CPC - G06T 7/593: from stereo images - G06V 10/82: using neural networks - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/285: using a sequence of stereo image pairs - G06T 2207/10012: Stereo images - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20228: Disparity calculation for image-based rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques and systems are provided for image processing. For instance, a process can include correlating a first set of features from a first viewpoint with a second set of features from a second viewpoint at a first time period to generate a first disparity cost volume; correlating a third set of features from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume; gating the first disparity cost volume to generate first intermediate disparity information; gating the first optical flow cost volume to generate first intermediate optical flow information; correlating the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output; and correlating the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output. ## Claims ### Claim 1 A method for image processing, comprising: correlating a first set of features obtained from a first viewpoint at a first time period with a second set of features obtained from a second viewpoint at the first time period to generate a first disparity cost volume; correlating a third set of features obtained from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume; gating the first disparity cost volume to generate first intermediate disparity information; gating the first optical flow cost volume to generate first intermediate optical flow information; correlating the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output; and correlating the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output. ### Claim 2 (depends on 1) The method of claim 1, wherein: correlating the first set of features, second set of features, and the first intermediate optical flow information to generate a second disparity cost volume; and correlating the third set of features, the first set of features, and the first intermediate disparity information to generate a second optical flow cost volume. ### Claim 3 (depends on 2) The method of claim 2, further comprising: gating the second disparity cost volume to generate second intermediate disparity information; gating the second optical flow cost volume to generate second intermediate optical flow information; outputting disparity information based on the second intermediate disparity information; and outputting optical flow information based on the second intermediate optical flow information. ### Claim 4 (depends on 3) The method of claim 3, further comprising: concatenating the first intermediate optical flow information and the second intermediate optical flow information to generate concatenated optical flow information; and concatenating the first intermediate disparity information and the second intermediate disparity information to generate concatenated disparity information. ### Claim 5 (depends on 4) The method of claim 4, wherein outputting disparity information and optical flow information comprising outputting the concatenated disparity information and the concatenated optical flow information. ### Claim 6 (depends on 2) The method of claim 2, wherein correlating the first set of features, the second set of features, and the first intermediate optical flow information comprises: combining the first set of features with the first intermediate optical flow information to refine the first set of features; combining the second set of features with the first intermediate optical flow information to refine the second set of features; and comparing the refined first set of features and the refined second set of features to generate the second disparity cost volume. ### Claim 7 (depends on 6) The method of claim 6, wherein: combining the first set of features with the first intermediate optical flow information comprises concatenating the first set of features with the first intermediate optical flow information; and combining the second set of features with the first intermediate optical flow information comprises concatenating the second set of features with the first intermediate optical flow information. ### Claim 8 (depends on 6) The method of claim 6, wherein comparing the refined first set of features and the refined second set of features comprises determining a dot product between a feature of the refined first set of features and a feature of the refined second set of features. ### Claim 9 (depends on 2) The method of claim 2, wherein correlating the third set of features, the first set of features, and the first intermediate disparity information comprises: combining the first set of features with the first intermediate disparity information to refine the first set of features; combining the third set of features with the first intermediate disparity information to refine the second set of features; and comparing the refined first set of features and the refined third set of features to generate the second optical flow cost volume. ### Claim 10 (depends on 9) The method of claim 9, wherein: combining the first set of features with the first intermediate disparity information comprises concatenating the first set of features with the first intermediate disparity information; and combining the third set of features with the first intermediate disparity information comprises concatenating the third set of features with the first intermediate disparity information. ### Claim 11 (depends on 9) The method of claim 9, wherein comparing the refined first set of features and the refined third set of features comprises determining a dot product between a feature of the refined first set of features and a feature of the refined third set of features. ### Claim 12 An apparatus for image processing, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: correlate a first set of features obtained from a first viewpoint at a first time period with a second set of features obtained from a second viewpoint at the first time period to generate a first disparity cost volume; correlate a third set of features obtained from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume; gate the first disparity cost volume to generate first intermediate disparity information; gate the first optical flow cost volume to generate first intermediate optical flow information; correlate the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output; and correlate the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the at least one processor is further configured to: correlate the first set of features, second set of features, and the first intermediate optical flow information to generate a second disparity cost volume; and correlate the third set of features, the first set of features, and the first intermediate disparity information to generate a second optical flow cost volume. ### Claim 14 (depends on 13) The apparatus of claim 13 wherein the at least one processor is further configured to: gate the second disparity cost volume to generate second intermediate disparity information; gate the second optical flow cost volume to generate second intermediate optical flow information; output disparity information based on the second intermediate disparity information; and output optical flow information based on the second intermediate optical flow information. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the at least one processor is further configured to: concatenate the first intermediate optical flow information and the second intermediate optical flow information to generate concatenated optical flow information; and concatenate the first intermediate disparity information and the second intermediate disparity information to generate concatenated disparity information. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein, to output disparity information and optical flow information, the at least one processor is configured to output the concatenated disparity information and the concatenated optical flow information. ### Claim 17 (depends on 13) The apparatus of claim 13, wherein, to correlate the first set of features, the second set of features, and the first intermediate optical flow information, the at least one processor is configured to: combine the first set of features with the first intermediate optical flow information to refine the first set of features; combine the second set of features with the first intermediate optical flow information to refine the second set of features; and compare the refined first set of features and the refined second set of features to generate the second disparity cost volume. ### Claim 18 (depends on 17) The apparatus of claim 17, wherein: to combine the first set of features with the first intermediate optical flow information, the at least one processor is further configured to concatenate the first set of features with the first intermediate optical flow information; and to combine the second set of features with the first intermediate optical flow information, the at least one processor is further configured to concatenate the second set of features with the first intermediate optical flow information. ### Claim 19 (depends on 17) The apparatus of claim 17, wherein, to compare the refined first set of features and the refined second set of features, the at least one processor is configured to determine a dot product between a feature of the refined first set of features and a feature of the refined second set of features. ### Claim 20 (depends on 13) The apparatus of claim 13, wherein, to correlate the third set of features, the first set of features, and the first intermediate disparity information, the at least one processor is configured to: combine the first set of features with the first intermediate disparity information to refine the first set of features; combine the third set of features with the first intermediate disparity information to refine the second set of features; and compare the refined first set of features and the refined third set of features to generate the second optical flow cost volume. ### Claim 21 (depends on 20) The apparatus of claim 20, wherein: to combine the first set of features with the first intermediate disparity information, the at least one processor is configured to concatenate the first set of features with the first intermediate disparity information; and to combine the third set of features with the first intermediate disparity information, the at least one processor is configured to concatenate the third set of features with the first intermediate disparity information. ### Claim 22 (depends on 20) The apparatus of claim 20, wherein, to compare the refined first set of features and the refined third set of features, the at least one processor is further configured to determine a dot product between a feature of the refined first set of features and a feature of the refined third set of features. ### Claim 23 A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: correlate a first set of features obtained from a first viewpoint at a first time period with a second set of features obtained from a second viewpoint at the first time period to generate a first disparity cost volume; correlate a third set of features obtained from the first viewpoint at a second time period with the first set of features to generate a first optical flow cost volume; gate the first disparity cost volume to generate first intermediate disparity information; gate the first optical flow cost volume to generate first intermediate optical flow information; correlate the first set of features, the second set of features, and the first intermediate optical flow information to generate disparity information for output; and correlate the third set of features, the first set of features, and the first intermediate disparity information to generate optical flow information for output. ### Claim 24 (depends on 23) The non-transitory computer-readable medium of claim 23, wherein the instructions cause the at least one processor to: correlate the first set of features, second set of features, and the first intermediate optical flow information generates a second disparity cost volume; and correlate the third set of features, the first set of features, and the first intermediate disparity information generates a second optical flow cost volume. ### Claim 25 (depends on 24) The non-transitory computer-readable medium of claim 24 wherein the instructions cause the at least one processor to: gate the second disparity cost volume to generate second intermediate disparity information; gate the second optical flow cost volume to generate second intermediate optical flow information; output disparity information based on the second intermediate disparity information; and output optical flow information based on the second intermediate optical flow information. ### Claim 26 (depends on 25) The non-transitory computer-readable medium of claim 25, wherein the instructions cause the at least one processor to: concatenate the first intermediate optical flow information and the second intermediate optical flow information to generate concatenated optical flow information; and concatenate the first intermediate disparity information and the second intermediate disparity information to generate concatenated disparity information. ### Claim 27 (depends on 26) The non-transitory computer-readable medium of claim 26, wherein, to output disparity information and optical flow information, the instructions cause the at least one processor to output the concatenated disparity information and the concatenated optical flow information. ### Claim 28 (depends on 24) The non-transitory computer-readable medium of claim 24, wherein, to correlate the first set of features, the second set of features, and the first intermediate optical flow information, the instructions cause the at least one processor to: combine the first set of features with the first intermediate optical flow information to refine the first set of features; combine the second set of features with the first intermediate optical flow information to refine the second set of features; and compare the refined first set of features and the refined second set of features to generate the second disparity cost volume. ### Claim 29 (depends on 28) The non-transitory computer-readable medium of claim 28, wherein: to combine the first set of features with the first intermediate optical flow information, the instructions cause the at least one processor to concatenate the first set of features with the first intermediate optical flow information; and to combine the second set of features with the first intermediate optical flow information, the instructions cause the at least one processor to concatenate the second set of features with the first intermediate optical flow information. ### Claim 30 (depends on 28) The non-transitory computer-readable medium of claim 28, wherein, to compare the refined first set of features and the refined second set of features, the instructions cause the at least one processor to determine a dot product between a feature of the refined first set of features and a feature of the refined second set of features.