# US 12,112,495 B2 · Depth data filtering method and apparatus, electronic device, and readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,495 B2 - Document ID: US12112495B2 - Status: Granted - Kind: B2 - Application number: 17/559,103 - Filed: 2021-12-22 - Published: 2022-04-14 - Granted: 2024-10-08 - Earliest claimed: 2019-07-11 - Assignees: Guangdong Oppo Mobile Telecommunications Corp., Ltd. - Applicants: Guangdong Oppo Mobile Telecommunications Corp., Ltd. - Inventors: Jian Kang - Web page: https://patentgrep.com/patent/US12112495B2 ## CPC - G06T 7/55: from multiple images - G01B 11/22: for measuring depth - G01S 7/4865: Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak (peak detection in noise, signal conditioning G01S7/487) - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G06T 5/70: Denoising; Smoothing - G06V 10/225: {based on a marking or identifier characterising the area} - G06V 10/30: Noise filtering - G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation - G06V 10/72: Data preparation, e.g. statistical preprocessing of image or video features - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Provided are a depth data filtering method and apparatus, an electronic device, and a readable storage medium. The method includes: obtaining, for each pixel, a depth difference value between two consecutive frames of depth maps; marking an area formed by pixels as a first environment change area, the depth difference value of the pixels is smaller than a predetermined absolute depth deviation; marking an area formed by pixels as a second environment change area, the depth difference value of the pixels is greater than or equal to the predetermined absolute depth deviation; respectively filtering the first environment change area and the second environment change area. ## Claims ### Claim 1 A depth data filtering method, comprising: obtaining, for each pixel, a depth difference value between two consecutive frames of depth maps; marking an area formed by pixels as a first environment change area, the depth difference value of the pixels being smaller than a predetermined absolute depth deviation; marking an area formed by pixels as a second environment change area, the depth difference value of the pixels being greater than or equal to the predetermined absolute depth deviation; generating a first similarity weight corresponding to each pixel in the first environment change area based on the depth difference value, an enlarged predetermined original smoothing coefficient, and a pixel depth deviation value of a current frame; generating a second similarity weight corresponding to each pixel in the second environment change area based on the depth difference value, a reduced predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame; and filtering the first environment change area based on the first similarity weight and filtering the second environment change area based on the second similarity weight. ### Claim 2 (depends on 1) The method according to claim 1, further comprising: marking the first environment change area with a first area mask. ### Claim 3 (depends on 1) The method according to claim 1, further comprising: marking the second environment change area with a second area mask. ### Claim 4 (depends on 1) The method according to claim 1, wherein said filtering the first environment change area based on the first similarity weight comprises: obtaining a first original depth value of the previous frame and a first original depth value of the current frame corresponding to each pixel in the first environment change area in a predetermined coordinate system; and obtaining a first current frame depth value corresponding to each pixel in the first environment change area by summing-up a product of the first similarity weight and the first original depth value of the previous frame and a product of a third similarity weight and the first original depth value of the current frame, wherein a sum of the first similarity weight and the third similarity weight is equal to 1. ### Claim 5 (depends on 4) The method according to claim 4, wherein a similarity weight is generated based on the depth difference value for each pixel, a predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame by applying the following predetermined formula: w ⁢ ⁢ 1 = s * exp ⁡ ( - diff 2 2 ⁢ σ 2 ) , where s represents the predetermined original smoothing coefficient, diff represents the depth difference value, and σ represents the pixel depth deviation value of the current frame. ### Claim 6 (depends on 1) The method according to claim 1, wherein said filtering the second environment change area based on the second similarity weight comprises: obtaining a second original depth value of the previous frame and a second original depth value of the current frame corresponding to each pixel in the second environment change area in a predetermined coordinate system; and obtaining a second current frame depth value corresponding to each pixel in the second environment change area by summing-up a product of the second similarity weight and the second original depth value of the previous frame and a product of a fourth similarity weight and the second original depth value of the current frame, wherein a sum of the second similarity weight and the fourth similarity weight is equal to 1. ### Claim 7 (depends on 5) The method according to claim 5, wherein a similarity weight is generated based on the depth difference value for each pixel, a predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame by applying the following predetermined formula: w ⁢ ⁢ 1 = s * exp ⁡ ( - diff 2 2 ⁢ σ 2 ) , where s represents the predetermined original smoothing coefficient, diff represents the depth difference value, and σ represents the pixel depth deviation value of the current frame. ### Claim 8 An electronic device, comprising an image sensor, a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image sensor is electrically connected to the processor, and the program, when executed by the processor, implements a depth data filtering method, comprising: obtaining, for each pixel, a depth difference value between two consecutive frames of depth maps; marking an area formed by respective pixels as a first environment change area, the depth difference value of the pixels being smaller than a predetermined absolute depth deviation; marking an area formed by pixels as a second environment change area, the depth difference value of the pixels being greater than or equal to the predetermined absolute depth deviation; generating a first similarity weight corresponding to each pixel in the first environment change area based on the depth difference value, an enlarged predetermined original smoothing coefficient, and a pixel depth deviation value of a current frame; generating a second similarity weight corresponding to each pixel in the second environment change area based on the depth difference value, a reduced predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame; and filtering the first environment change area based on the first similarity weight and filtering the second environment change area based on the second similarity weight. ### Claim 9 (depends on 8) The electronic device according to claim 8, wherein the program, when executed by the processor, further implements: marking the first environment change area with a first area mask. ### Claim 10 (depends on 8) The electronic device according to claim 8, wherein the program, when executed by the processor, further implements: marking the second environment change area with a second area mask. ### Claim 11 (depends on 8) The electronic device according to claim 8, wherein the program, when executed by the processor, implements: obtaining a first original depth value of the previous frame and a first original depth value of the current frame corresponding to each pixel in the first environment change area in a predetermined coordinate system; and obtaining a first current frame depth value corresponding to each pixel in the first environment change area by summing-up a product of the first similarity weight and the first original depth value of the previous frame and a product of a third similarity weight and the first original depth value of the current frame, wherein a sum of the first similarity weight and the third similarity weight is equal to 1. ### Claim 12 (depends on 11) The electronic device according to claim 11, wherein the program, when executed by the processor, implements: generating a similarity weight based on the depth difference value for each pixel, a predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame by applying the following predetermined formula: w ⁢ ⁢ 1 = s * exp ⁡ ( - diff 2 2 ⁢ σ 2 ) , where s represents the predetermined original smoothing coefficient, diff represents the depth difference value, and σ represents the pixel depth deviation value of the current frame. ### Claim 13 (depends on 8) The electronic device according to claim 8, wherein the program, when executed by the processor, implements: obtaining a second original depth value of the previous frame and a second original depth value of the current frame corresponding to each pixel in the second environment change area in a predetermined coordinate system; and obtaining a second current frame depth value corresponding to each pixel in the second environment change area by summing-up a product of the second similarity weight and the second original depth value of the previous frame and a product of a fourth similarity weight and the second original depth value of the current frame, wherein a sum of the second similarity weight and the fourth similarity weight is equal to 1. ### Claim 14 (depends on 13) The electronic device according to claim 13, wherein the program, when executed by the processor, implements: generating a similarity weight based on the depth difference value for each pixel, a predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame by applying the following predetermined formula: w ⁢ ⁢ 1 = s * exp ⁡ ( - diff 2 2 ⁢ σ 2 ) , where s represents the predetermined original smoothing coefficient, diff represents the depth difference value, and σ represents the pixel depth deviation value of the current frame. ### Claim 15 A non-transitory computer-readable storage medium, comprising a computer program stored thereon, wherein the computer program, when executed by a processor, implements a depth data filtering method comprising: obtaining, for each pixel, a depth difference value between two consecutive frames of depth maps for each pixel; marking an area formed by pixels as a first environment change area, the depth difference value of the pixels being smaller than a predetermined absolute depth deviation; marking an area formed by respective pixels as a second environment change area, the depth difference value of the pixels being greater than or equal to the predetermined absolute depth deviation; generating a first similarity weight corresponding to each pixel in the first environment change area based on the depth difference value, an enlarged predetermined original smoothing coefficient, and a pixel depth deviation value of a current frame; generating a second similarity weight corresponding to each pixel in the second environment change area based on the depth difference value, a reduced predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame; and filtering the first environment change area based on the first similarity weight and filtering the second environment change area based on the second similarity weight. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein the computer program, when executed by the processor, further implements: marking the first environment change area with a first area mask. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein the computer program, when executed by the processor, further implements: marking the second environment change area with a second area mask. ### Claim 18 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein the computer program, when executed by the processor, implements: obtaining a first original depth value of the previous frame and a first original depth value of the current frame corresponding to each pixel in the first environment change area in a predetermined coordinate system; and obtaining a first current frame depth value corresponding to each pixel in the first environment change area by summing-up a product of the first similarity weight and the first original depth value of the previous frame and a product of a third similarity weight and the first original depth value of the current frame, wherein a sum of the first similarity weight and the third similarity weight is equal to 1. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein the computer program, when executed by the processor, implements: obtaining a second original depth value of the previous frame and a second original depth value of the current frame corresponding to each pixel in the second environment change area in a predetermined coordinate system; obtaining a second current frame depth value corresponding to each pixel in the second environment change area by summing-up a product of the second similarity weight and the second original depth value of the previous frame and a product of a fourth similarity weight and the second original depth value of the current frame, wherein a sum of the second similarity weight and the fourth similarity weight is equal to 1. ### Claim 20 (depends on 18) The non-transitory computer-readable storage medium according to claim 18, wherein the computer program, when executed by the processor, implements: generating a similarity weight based on the depth difference value for each pixel, a predetermined original smoothing coefficient, and the pixel depth deviation value of the current frame by applying the following predetermined formula: w ⁢ ⁢ 1 = s * exp ⁡ ( - diff 2 2 ⁢ σ 2 ) , where s represents the predetermined original smoothing coefficient, cuff represents the depth difference value, and σ represents the pixel depth deviation value of the current frame.