# US 2003/0198378 A1 · Method and system for 3D smoothing within the bound of error regions of matching curves > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2003/0198378 A1 - Document ID: US20030198378A1 - Status: Published application - Kind: A1 - Application number: 10/125,028 - Filed: 2002-04-18 - Published: 2003-10-23 - Earliest claimed: 2002-04-18 - Assignees: STMicroelectronics S.r.l. - Inventors: Kim Chai Ng - Web page: https://patentgrep.com/patent/US20030198378A1 ## CPC - G06T 7/55: from multiple images - G06V 10/10: Image acquisition (document image scanning and transmission H04N1/00; control of digital cameras H04N23/60) - G06V 2201/12: Acquisition of 3D measurements of objects - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An image processing system and method for smoothing irregularities from 3D image information that was reconstructed from a plurality of 2D views of a scene, and particularly from homogeneous surfaces of objects in a scene. The method defines a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene. Each pixel is associated with predefined 3D depth information, and further is associated with a matching curve. A subject pixel is located within the plurality of pixels overlapped by the window. The method calculates an average 3D depth information associated with the plurality of pixels overlapped by the window, and assigns the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel. ## Claims ### Claim 1 A method of smoothing 3D image information, the method comprising the steps of: defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels, other than the subject pixel, being overlapped by the window; and assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel. ### Claim 2 (depends on 1) The method of claim 1, wherein the defining, the calculating, and the assigning steps are operated on each and every pixel in the one of the plurality of 2D image views of the scene, each pixel being defined as a subject pixel with the window overlapping the subject pixel and a plurality of its neighboring pixels, and wherein the assigning step assigns the calculated average 3D depth information to the 3D depth information associated with the subject pixel while the calculated average 3D depth information is outside of a smoothing threshold range from the 3D depth information associated with the subject pixel. ### Claim 3 (depends on 1) The method of claim 1, wherein the defining, the calculating, and the assigning steps are substantially simultaneously operated on each and every pixel in the one of the plurality of 2D image views of the scene, each pixel being defined as a subject pixel with the window overlapping the subject pixel and a plurality of its neighboring pixels, and wherein the assigning step assigns the calculated average 3D depth information to the 3D depth information associated with the subject pixel while the calculated average 3D depth information is outside of a smoothing threshold range from the 3D depth information associated with the subject pixel. ### Claim 4 (depends on 3) The method of claim 3, wherein the defining, the calculating, and the assigning steps are iteratively operated on each and every subject pixel in the one of the plurality of 2D image views of the scene until the calculated average 3D depth information for each and every respective subject pixel is within a smoothing threshold range from the 3D depth information associated with the respective subject pixel. ### Claim 5 (depends on 1) The method of claim 1, wherein the window is defined to overlap a plurality of pixels about a high contrast region. ### Claim 6 (depends on 1) The method of claim 1, wherein, at a first processing of the 3D image information, the window is defined to overlap a plurality of pixels about an edge in the scene and the subject pixel is an edgel. ### Claim 7 (depends on 6) The method of claim 6, wherein, at a second processing of the 3D image information, the window is defined to overlap a plurality of pixels about a homogeneous region in the scene and the subject pixel is in the homogeneous region. ### Claim 8 (depends on 1) The method of claim 1, further comprising the steps of: redefining the window to overlap a different subject pixel and a plurality of pixels of the one of the plurality of 2D image views of the scene; and repeating the calculating and the assigning steps. ### Claim 9 (depends on 8) The method of claim 8, wherein the redefining and repeating steps are repeated while advancing the window from a high contrast region to a low contrast region of the one of the plurality of 2D image views of the scene. ### Claim 10 (depends on 8) The method of claim 8, wherein the redefining and repeating steps are repeated while advancing the window from an edge region to a homogeneous region of the one of the plurality of 2D image views of the scene. ### Claim 11 A method of smoothing 3D image information, the method comprising the steps of: defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels, other than the subject pixel, being overlapped by the window; assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel; and smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by repeating the defining, calculating, and assigning steps, for each and every subject pixel in the one 2D image view of the scene, while the calculated average 3D depth information for the plurality of pixels, other than the subject pixel, that is being overlapped by the window is outside of a smoothing threshold range from the 3D depth information associated with the subject pixel. ### Claim 12 (depends on 11) The method of claim 11, wherein the defining, calculating, and assigning steps, are repeatedly substantially simultaneously operated on each and every subject pixel in the one 2D image view of the scene until the calculated average 3D depth information for each and every subject pixel is within a smoothing threshold range from the 3D depth information associated with the respective subject pixel. ### Claim 13 A method of smoothing 3D image information, the method comprising the steps of: defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels overlapped by the window; assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel; and smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by repeating the defining, calculating, and assigning steps, while advancing the window from a high contrast region to a low contrast region in the scene. ### Claim 14 (depends on 13) The method of claim 13, wherein the smoothing step comprises the step of: smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by repeating the defining, calculating, and assigning steps, while advancing the window from an edge region to a homogeneous region in the scene. ### Claim 15 A computer readable medium including computer instructions for smoothing 3D image information, the computer instructions comprising instructions for: defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels, other than the subject pixel, being overlapped by the window; and assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel. ### Claim 16 (depends on 15) The computer readable medium of claim 15, further including computer instructions for: redefining the window to overlap a different subject pixel and a plurality of pixels of the one of the plurality of 2D image views of the scene; and repeating the calculating and the assigning. ### Claim 17 (depends on 16) The computer readable medium of claim 16, wherein the redefining and repeating steps are repeated while advancing the window from a high contrast region to a low contrast region of the one of the plurality of 2D image views of the scene. ### Claim 18 (depends on 16) The computer readable medium of claim 16, wherein the redefining and repeating steps are repeated while advancing the window from an edge region to a homogeneous region of the one of the plurality of 2D image views of the scene. ### Claim 19 A computer readable medium including computer instructions for smoothing 3D image information, the computer instructions comprising instructions for: defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels, other than the subject pixel, being overlapped by the window; assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel; and smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by repeating the defining, calculating, and assigning steps, for each and every subject pixel in the one 2D image view of the scene, while the calculated average 3D depth information for the plurality of pixels, other than the subject pixel, that is being overlapped by the window is outside of a smoothing threshold range from the 3D depth information associated with the subject pixel. ### Claim 20 (depends on 19) The computer readable medium of claim 19, wherein the defining, calculating, and assigning, are repeatedly substantially simultaneously operated for each and every subject pixel in the one 2D image view of the scene until the calculated average 3D depth information for each and every subject pixel is within a smoothing threshold range from the 3D depth information associated with the respective subject pixel. ### Claim 21 A computer readable medium including computer instructions for smoothing 3D image information, the computer instructions comprising instructions for: defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels overlapped by the window; assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel; and smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by repeating the defining, calculating, and assigning steps, while advancing the window from a high contrast region to a low contrast region in the scene. ### Claim 22 (depends on 21) The computer readable medium of claim 21, wherein the smoothing step comprises: smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by repeating the defining, calculating, and assigning steps, while advancing the window from an edge region to a homogeneous region in the scene. ### Claim 23 An image processing system comprising: a memory; a controller/processor electrically coupled to the memory; and a means for smoothing 3D image information, electrically coupled to the controller/processor and to the memory, for: defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels, other than the subject pixel, being overlapped by the window; assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel; and smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by repeating the defining, calculating, and assigning steps, for each and every subject pixel in the one 2D image view of the scene, while the calculated average 3D depth information for the plurality of pixels, other than the subject pixel, that is being overlapped by the window is outside of a smoothing threshold range from the 3D depth information associated with the subject pixel. ### Claim 24 (depends on 23) The image processing system of claim 23, wherein the means for smoothing 3D image information is further for smoothing 3D image information associated with pixels of the one 2D image view of the plurality of 2D image views by substantially simultaneously repeating the defining, calculating, and assigning, instructions being operated on each and every subject pixel in the one 2D image view of the scene until the calculated average 3D depth information for each and every subject pixel is within a smoothing threshold range from the 3D depth information associated with the respective subject pixel. ### Claim 25 (depends on 23) The image processing system of claim 23, further comprising at least one camera interface, electrically coupled to the controller/processor, for sending image information from at least one camera to the controller/processor. ### Claim 26 (depends on 23) The image processing system of claim 23, wherein the controller/processor, the memory, and the means for smoothing 3D image information, are implemented in at least one of an integrated circuit and a circuit supporting substrate. ### Claim 27 (depends on 23) The image processing system of claim 23, further comprising at least one video interface for displaying a 3D image corresponding to the 3D depth information of the scene. ### Claim 28 An electronic device comprising: a data memory; a program memory including computer instructions for smoothing 3D image information; and a controller/processor, electrically coupled to the data memory and the program memory, for defining a window that overlaps a plurality of pixels of one of a plurality of 2D image views of a scene, each pixel being associated with predefined 3D depth information therefor, and further being associated with a matching curve, a subject pixel being located within the plurality of pixels overlapped by the window; calculating an average 3D depth information associated with the plurality of pixels, other than the subject pixel, being overlapped by the window; and assigning the calculated average 3D depth information to the 3D depth information of the subject pixel, if the calculated average 3D depth information is within an error region of a matching curve associated with the subject pixel. ### Claim 29 (depends on 28) The electronic device of claim 28, wherein the controller/processor for, at a first processing of the 3D image information, defining the window to overlap a plurality of pixels about an edge in the scene and the subject pixel is an edgel. ### Claim 30 (depends on 28) The electronic device of claim 28, wherein the controller/processor for, at a second processing of the 3D image information, defining the window to overlap a plurality of pixels about a homogeneous region in the scene and the subject pixel is in the homogeneous region. ### Claim 31 (depends on 28) The electronic device of claim 28, wherein the controller/processor for, operating the defining, the calculating, and the assigning instructions for each and every pixel in the one of the plurality of 2D image views of the scene, each pixel being defined as a subject pixel with the window overlapping the subject pixel and a plurality of its neighboring pixels, and wherein the assigning instructions assign the calculated average 3D depth information to the 3D depth information associated with the subject pixel while the calculated average 3D depth information is outside of a smoothing threshold range from the 3D depth information associated with the subject pixel. ### Claim 32 (depends on 28) The electronic device of claim 28, wherein the controller/processor for, substantially simultaneously operating the defining, the calculating, and the assigning instructions for each and every pixel in the one of the plurality of 2D image views of the scene, each pixel being defined as a subject pixel with the window overlapping the subject pixel and a plurality of its neighboring pixels, and wherein the assigning step assigns the calculated average 3D depth information to the 3D depth information associated with the subject pixel while the calculated average 3D depth information is outside of a smoothing threshold range from the 3D depth information associated with the subject pixel. ### Claim 33 (depends on 32) The electronic device of claim 32, wherein the controller/processor for, iteratively operating the defining, the calculating, and the assigning instructions for each and every subject pixel in the one of the plurality of 2D image views of the scene until the calculated average 3D depth information for each and every respective subject pixel is within a smoothing threshold range from the 3D depth information associated with the respective subject pixel. ### Claim 34 (depends on 28) The electronic device of claim 28, further comprising a video interface for providing, to a display, 3D image display information corresponding to the 3D depth information of the scene. ### Claim 35 (depends on 28) The electronic device of claim 28, wherein the electronic device comprises at least one of an integrated circuit and a circuit supporting substrate.