# US 10,504,234 B2 · Image processing apparatus and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,234 B2 - Document ID: US10504234B2 - Status: Granted - Kind: B2 - Application number: 15/860,662 - Filed: 2018-01-03 - Published: 2018-05-24 - Granted: 2019-12-10 - Earliest claimed: 2015-07-03 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Jacek Konieczny; Marek Domanski; Olgierd Stankiewicz; Krzysztof Wegner; Adam Łuczak - Web page: https://patentgrep.com/patent/US10504234B2 ## CPC - G06T 7/55: from multiple images - G06T 2207/10012: Stereo images - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20228: Disparity calculation for image-based rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present disclosure can provide an image processing apparatus. In some embodiments, the image processing apparatus in accordance with the disclosure comprises memory configured to store a previously selected depth information value (dprev), (dprev; dprevi) having been selected for a previously processed fragment and signal processing logic (103). The signal processing logic is configured to calculate a similarity measure for the current fragment based on a depth information value candidate defining a reference fragment candidate of a digital reference image, compare the depth information value candidate with the previously selected depth information value (dprev), apply a weighting function to the calculated similarity measure to obtain a weighted similarity measure for the current fragment based on the comparison, and determine whether to select the depth information value candidate as the depth information value (dbest) for the current fragment based on the weighted similarity measure of the depth information value candidate. ## Claims ### Claim 1 An image processing apparatus for selecting a depth information value dbest for a current fragment of a current digital image, the image processing apparatus comprising: memory configured to store a previously selected depth information value (dprev; dprevk), (dprev; dprevk) having been selected for a previously processed fragment; and signal processing logic configured to: calculate a similarity measure, MCurrent(di), for the current fragment based on a depth information value candidate di, di defining a reference fragment candidate of a digital reference image, wherein the similarity measure indicates a difference between a value of the current digital image at the current fragment and a value of the digital reference image at the reference fragment candidate; compare the depth information value candidate di with the previously selected depth information value (dprev; dprev,k); apply a weighting function to the calculated similarity measure to obtain a weighted similarity measure, Ccurrent(di) for the current fragment based on the comparison between the depth information value candidate di and the previously selected depth information value (dprev; dprev,k); and determine whether to select the depth information value candidate di as the depth information value dbest for the current fragment based on the weighted similarity measure of the depth information value candidate di. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the similarity measure is a matching cost and the weighted similarity measure is a weighted matching cost, and wherein the weighting function is configured such that: when the depth information value candidate di is different from the previously selected depth information value (dprev; dprev,k), the matching cost is increased with a penalty, and/or when the depth information value candidate di is identical to the previously selected depth information value (dprev; dprev,k), the matching cost is maintained or increased to a smaller extent compared to the case where the depth information value candidate di is different from the previously selected depth information value (dprev; dprev,k). ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the weighting function is configured such that: when the depth information value candidate di is different from the previously selected depth information value (dprev; dprev,k), the matching cost is increased by adding a first matching cost penalty to the matching cost to obtain the weighted matching cost, or by multiplying the matching cost with a first matching cost penalty to obtain the weighted matching cost, and/or when the depth information value candidate di is identical to the previously selected depth information value (dprev; dprev,k), the matching cost is maintained or increased by adding a second matching cost penalty to the matching cost to obtain the weighted matching cost, or by multiplying the matching cost with a second matching cost penalty to obtain the weighted matching cost, wherein the second matching cost penalty is smaller than the first matching cost penalty. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the similarity measure is a matching probability and the weighted similarity measure is a weighted matching probability, and wherein the weighting function is configured such that, when the depth information value candidate di is different from the previously selected depth information value (dprev; dprev,k), the matching probability is decreased, and/or such that, when the depth information value candidate di is identical to the previously selected depth information value (dprev; dprev,k), the matching probability is maintained or decreased to a smaller extent compared to the case where the depth information value candidate di is different from the previously selected depth information value (dprev; dprev,k). ### Claim 5 (depends on 4) The image processing apparatus (100) according to claim 4, wherein the weighting function is configured such that, when the depth information value candidate (di) is different from the previously selected depth information value (dprev; dprev,k), the matching probability is decreased by subtracting a first matching probability penalty from the matching probability to obtain the weighted matching probability, or by dividing the matching probability by a first matching probability penalty to obtain the weighted matching probability, and/or such that, when the depth information value candidate di is identical to the previously selected depth information value (dprev; dprev,k), the matching probability is maintained or decreased by subtracting a second matching probability penalty from the matching probability to obtain the weighted matching probability, or by dividing the matching probability by a second matching probability penalty to obtain the weighted matching probability, wherein the first matching probability penalty is larger than the second matching probability penalty. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the signal processing logic is configured to: calculate a similarity measure for each depth information value candidate of a set of depth information value candidates, each depth information value candidate di defining a reference fragment candidate of the digital reference image; compare each of the depth information value candidates di of the set of depth information value candidates with the previously selected depth information value (dprev; dprev,k); apply, for each of the depth information value candidates di, a weighting function to the calculated similarity measure to obtain for each of the depth information value candidates di a corresponding weighted similarity measure for the current fragment, wherein for each of the depth information value candidates di the weighting function is applied based on the corresponding result of the comparison; and select the depth information value dbest for the current fragment from the depth information value candidates di of the set of depth information value candidates based on the weighted similarity measures. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the similarity measure is a matching cost, and the signal processing logic is configured to select the depth information value dbest from the depth information value candidates di of the set of depth information value candidates, which has the smallest weighted matching cost; or wherein the similarity measure is a matching probability, and the signal processing logic is configured to select the depth information value dbest from the depth information value candidates (di) of the set of depth information value candidates, which has the largest weighted matching probability. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the signal processing logic is configured to determine the depth information value dbest for the current fragment by sequentially processing depth information value candidates di of a set of depth information value candidates; wherein the signal processing logic is configured to: calculate a similarity measure for a current depth information value candidate (di) of the set of depth information value candidates; compare the current depth information value candidate (di) with the previously selected depth information value (dprev; dprev,k); apply the weighting function to the similarity measure of the current depth information value candidate di of the set of depth information value candidates to obtain a weighted similarity measure depending on the comparison; compare the weighted similarity measure of the current depth information value candidate with a currently best similarity measure costbest associated with a currently best depth information value dbest; when the weighted similarity measure of the current depth information value candidate does not fulfill an update condition with regard to the currently best similarity measure costbest maintain the currently best depth information value dbest as currently best depth information value and the currently best similarity measure as the currently best similarity measure, and/or when the weighted similarity measure of the current depth information value candidate fulfills an update condition with regard to the currently best similarity measure costbest, write the current depth information value candidate di as currently best depth information value dbest to the memory and write the weighted similarity measure of the current depth information value candidate (di) as currently best similarity measure costbest to the memory; and after all depth information value candidates di have been processed, select the currently best depth information value dbest as depth information value dbest for the current fragment of the current digital image. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the similarity measure is a matching cost and the weighted similarity measure is a weighted matching cost, and the update condition is that the weighted matching cost is smaller than, or smaller than or equal to the currently best matching cost costbest, or wherein the similarity measure is a matching probability and the weighted similarity measure is a weighted matching probability, and the update condition is that the weighted matching probability is larger than, or larger than or equal to the currently best matching probability. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the memory is configured to store a set of previously selected depth information values (dprev,k) of a set of previously processed fragments; and wherein the signal processing logic is further configured to: compare the current depth information value candidate di with at least one of the depth information values (dprev,k) of the set of previously selected depth information values dbest; and apply a weighting function to the calculated similarity measure of the depth information value candidate di to obtain the weighted similarity measure for the current depth information value candidate di based on whether any of the depth information values (dprev,k) of the set of previously selected depth information values (dprev,k) is identical to the current depth information value candidate di. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein the signal processing logic is configured to: compare the current depth information value candidate di with a set of depth information values within a range around the previously selected depth information value (dprev); and apply a weighting function to the calculated similarity measure of the depth information value candidate di to obtain the weighted similarity measure for the current depth information value candidate di based on whether any of the depth information values (dprev,k) of the set of depth information values within the range around the previously selected depth information value (dprev) is identical to the current depth information value candidate di. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein the signal processing logic is configured to: calculate the similarity measure MCurrent(di), for the current fragment based on the previously selected depth information value (dprev; dprev,k); compare the similarity measure with a threshold; and when the result of the comparison fulfills a skipping condition, select the previously selected depth information value (dprev; dprev,k) as the depth information value (dbest) of the current fragment of the current digital image; and/or when the result of the comparison does not fulfill the skipping condition: calculate a similarity measure, MCurrent(di), for the current fragment based on a further depth information value candidate (di) defining a further reference fragment candidate of a digital reference image; compare the further depth information value candidate (di) with the previously selected depth information value (dprev; dprev,k); apply a weighting function to the calculated similarity measure to obtain a weighted similarity measure, CCurrent(di), for the current fragment based on the comparison between the further depth information value candidate di and the previously selected depth information value (dprev; dprev,k); and determine whether to select the further depth information value candidate (di) as the depth information value dbest for the current fragment depending on the weighted matching cost of the depth information value candidate di. ### Claim 13 (depends on 12) The image processing apparatus according to claim 12, wherein the similarity measure is a matching cost and the skipping condition is that the matching cost is smaller than, or smaller than or equal to the threshold; wherein the similarity measure is a matching cost and the skipping condition is that the matching cost is smaller than, or smaller than or equal to the threshold, and wherein the threshold is equal to a penalty of the weighting function applied to the matching cost when the depth information value candidate is different to the previously selected depth information value or different to any depth information value comprised in a set of reference or previously selected depth information value candidates; or wherein the similarity measure is a matching probability and the skipping condition is that the matching probability is larger than, or larger than or equal to the threshold. ### Claim 14 An image processing method for selecting a depth information value dbest for a current fragment of a current digital image, the image processing method comprising: storing a previously selected depth information value (dprev; dprevi), (dprev; dprevi) having been selected for a previously processed fragment; calculating a similarity measure MCurrent(di) for the current fragment based on a depth information value candidate di defining a reference fragment candidate of a digital reference image, wherein the similarity measure indicates a difference between a value of the current digital image at the current fragment and a value of the digital reference image at the reference fragment candidate; comparing the depth information value candidate di with a previously selected depth information value (dprev; dprevi) selected for a previously processed fragment; applying a weighting function to the calculated similarity measure to obtain a weighted similarity measure CCurrent(di), for the current fragment based on the comparison between the depth information value candidate di and the previously selected depth information value (dprev; dprev,k); and determining whether to select the depth information value candidate di as the depth information value (dbest) for the current fragment based on the weighted similarity measure of the depth information value candidate di. ### Claim 15 (depends on 14) A non-transient computer readable medium containing program instructions for causing a computer to perform the method of claim 14.