# US 2022/0036573 A1 · Method and apparatus with image depth estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036573 A1 - Document ID: US20220036573A1 - Status: Published application - Kind: A1 - Application number: 17/116,063 - Filed: 2020-12-09 - Published: 2022-02-03 - Earliest claimed: 2020-07-29 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Seungeon KIM; Hyunsung CHANG - Web page: https://patentgrep.com/patent/US20220036573A1 ## CPC - G06T 7/50: Depth or shape recovery - G06T 7/593: from stereo images - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06T 2207/10004: Still image; Photographic image - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method and apparatus with image depth estimation are provided. The method includes obtaining a first statistical value associated with a depth for each of plural pixels included in an input image based on a first channel of output data obtained by applying the input image to a neural network, obtaining a second statistical value associated with a depth for each of the plural pixels in the input image based on a second channel of the output data, and estimating depth information of each of the plural pixels in the input image based on the first statistical value and the second statistical value. The neural network may be trained based on a probability distribution for a depth of each pixel in an image based on a first statistical value and a second statistical value that are obtained corresponding to an image with predetermined depth information in the training data. ## Claims ### Claim 1 A neural network apparatus, the apparatus comprising: a camera configured to capture an input image; and one or more processors configured to: apply the input image to a neural network; obtain a first statistical value associated with a depth of a pixel included in the input image; obtain another first statistical value associated with a depth of another pixel included in the input image; obtain a second statistical value associated with the depth of the pixel included in the input image; and obtain another second statistical value associated with the depth of the other pixel included in the input image; estimate depth information of the pixel included in the input image based on the first statistical value and the second statistical value; and estimate depth information of the other pixel included in the input image based on the other first statistical value and the other second statistical value, wherein the first statistical value and the other first statistical value are of a same first type of statistical value, and the second statistical value and the other second statistical value are a same second type of statistical value different from the first type of statistical value. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the second statistical value represents a confidence in the first statistical value, and the other second statistical value represents a confidence in the other first statistical value. ### Claim 3 (depends on 2) The apparatus of claim 2, wherein the second statistical value is a standard deviation or variation for the depth value of the pixel included in the input image, and the other second statistical value is a standard deviation or variation for the depth value of the other pixel included in the input image. ### Claim 4 (depends on 2) The apparatus of claim 2, wherein the one or more processors are further configured to: selectively, dependent on the represented confidence by the second statistical value, perform a correction of the first statistical value; and selectively, dependent on the represented confidence by the other second statistical value, perform a correction of the other first statistical value. ### Claim 5 (depends on 4) The apparatus of claim 4, wherein the one or more processors are further configured to: generate a reconstructed input image from a reference image, the generating being dependent on the first statistical value, the second statistical value, the other first statistical value, and the other second statistical value; and perform training of the neural network based on a comparison result of the generated reconstructed input image and the input image. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the first statistical value and the other first statistical value are obtained from image information output by the neural network as a first channel with the first statistical value and the other first statistical value, and wherein the second statistical value and the other second statistical value are obtained from image information output by the neural network as a second channel with the second statistical value and the other second statistical value. ### Claim 7 A neural network method, the method comprising: obtaining a first statistical value associated with a depth for each of plural pixels included in an input image based on a first channel of output data obtained by applying the input image to a neural network; obtaining a second statistical value associated with a depth for each of the plural pixels in the input image based on a second channel of the output data; and estimating depth information of each of the plural pixels in the input image based on the first statistical value and the second statistical value. ### Claim 8 (depends on 7) The method of claim 7, wherein each first statistical value comprises a mean of depth values of a corresponding each of the plural pixels in the input image, based on probabilities of the depth values, and each second statistical value comprises a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image, based on the probabilities of the depth values. ### Claim 9 (depends on 7) The method of claim 7, wherein the estimating of the depth information comprises: determining a confidence level of each of the first statistical values based on a corresponding second statistical value for each of the plural pixels in the input image; and respectively determining whether to use each of the first statistical values as depth information based on the correspondingly determined confidence level of each of the first statistical values. ### Claim 10 (depends on 9) The method of claim 9, wherein, with respect to the determining of the confidence level of each of the first statistical values, there is an inverse relationship between the determined confidence levels and the corresponding second statistical values in that when values of second statistical values increase corresponding confidence levels decrease, and when other values of the second statistical value decrease corresponding confidence levels increase. ### Claim 11 (depends on 7) The method of claim 7, wherein the estimating of the depth information comprises: obtaining a probability distribution based on a first statistical value for a depth of a pixel in the input image and a corresponding second statistical value of the first statistical value for the depth of the pixel; and performing random sampling based on the probability distribution to correct the first statistical value for the pixel. ### Claim 12 (depends on 11) The method of claim 11, wherein the performing of the random sampling is selectively performed dependent on a confidence level, of the first statistical value for the depth of the pixel, as represented by the corresponding second statistical value. ### Claim 13 (depends on 7) The method of claim 7, further comprising: capturing the input image using a camera of a terminal; and generating three-dimensional (3D) location information corresponding to the terminal and a surrounding object included in the input image, based on the estimated depth information. ### Claim 14 (depends on 7) The method of claim 7, wherein the input image comprises at least one of a monocular image and a stereo image. ### Claim 15 (depends on 7) A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 7. ### Claim 16 A neural network method, the method comprising: obtaining a first statistical value and a second statistical value associated with a depth, for each of plural pixels included in an input image, by applying the input image to a neural network; obtaining a probability distribution for a depth of each of the plural pixels in the input image, based on the first statistical value and the second statistical value; and training the neural network using an objective function that is based on the obtained probability distribution and predetermined depth information of each of the plural pixels in the input image. ### Claim 17 (depends on 16) The method of claim 16, wherein each first statistical value comprises a mean of depth values of a corresponding each of the plural pixels in the input image, based on probabilities of the depth values, and each second statistical value comprises a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image, based on the probabilities of the depth values. ### Claim 18 (depends on 16) The method of claim 16, wherein the obtaining of the first statistical value and the second statistical value comprises: obtaining each of the first statistical values based on a first channel of output data of the neural network to which the input image is applied; and obtaining each of the second statistical values based on a second channel of the output data. ### Claim 19 (depends on 16) The method of claim 16, wherein the obtaining of the first statistical value includes applying a feature map, extracted from the input image by a previous portion of the neural network, to a first network included in the neural network; and wherein the obtaining of the second statistical value includes applying the feature map to a second network included in the neural network. ### Claim 20 (depends on 16) The method of claim 16, wherein the training of the neural network comprises: estimating depth information of each of the plural pixels in the input image based on the probability distribution and a reference image corresponding to the input image; and training the neural network using an objective function that is based on the estimated depth information and the predetermined depth information of each of the plural pixels in the input image. ### Claim 21 (depends on 20) The method of claim 20, wherein the reference image comprises at least one of: a stereo image; and a monocular video and pose information of a camera used to capture the monocular video. ### Claim 22 (depends on 16) The method of claim 16, wherein the training of the neural network further comprises obtaining the predetermined depth information, of each of the plural pixels in the input image, based on the input image and a reference image corresponding to the input image. ### Claim 23 (depends on 16) The method of claim 16, wherein the training of the neural network comprises: synthesizing an output image based on the probability distribution and a reference image corresponding to the input image; and training the neural network using an objective function that is based on the input image and the synthesized output image. ### Claim 24 (depends on 23) The method of claim 23, wherein the synthesizing of the output image comprises, based on a probability distribution for a depth of a first pixel in the input image and pixels corresponding to the depth of a corresponding first pixel in the reference image, generating a second pixel in the output image corresponding to the first pixel. ### Claim 25 (depends on 24) The method of claim 24, wherein the generating of the second pixel comprises generating the second pixel by calculating a weighted sum of values of the pixels corresponding to the depth of the corresponding first pixel in the reference image, based on the probability distribution for the depth of the first pixel. ### Claim 26 (depends on 23) The method of claim 23, wherein the training of the neural network comprises training the neural network using an objective function for minimizing a difference between a first pixel in the input image and a corresponding first pixel in the output image. ### Claim 27 (depends on 16) The method of claim 16, wherein the training of the neural network comprises: generating a mask corresponding to each of the plural pixels in the input image, based on the respective first statistical value and the respective second statistical value; and training the neural network using the mask and the objective function. ### Claim 28 (depends on 16) The method of claim 16, wherein the training of the neural network comprises performing optimization of parameters for each of plural layers in the neural network, dependent on the objective function. ### Claim 29 (depends on 16) The method of claim 16, wherein the input image comprises at least one of a monocular image and a stereo image. ### Claim 30 (depends on 16) A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 16. ### Claim 31 A neural network apparatus, the apparatus comprising: one or more processors configured to: obtain a first statistical value associated with a depth for each of plural pixels included in an input image based on a first channel of output data obtained by applying the input image to a neural network; obtain a second statistical value associated with a depth for each of the plural pixels in the input image based on a second channel of the output data; and estimate depth information of each of the plural pixels in the input image based on the first statistical value and the second statistical value. ### Claim 32 (depends on 31) The apparatus of claim 31, wherein each first statistical value comprises a mean of depth values of a corresponding each of the plural pixels in the input image, based on probabilities of the depth values, and each second statistical value comprises a variance or a standard deviation of the depth values of the corresponding each of the plural pixels in the input image, based on the probabilities of the depth values. ### Claim 33 (depends on 31) The apparatus of claim 31, wherein, for the estimation of the depth information, the one or more processors are configured to: determine a confidence level of each of the first statistical values based on a corresponding second statistical value for each of the plural pixels in the input image; and respectively determine whether to use each of the first statistical values as depth information based on the correspondingly determined confidence level of each of the first statistical values. ### Claim 34 (depends on 31) The apparatus of claim 31, further comprising a camera configured to capture the input image, wherein the one or more processors are configured to generate three-dimensional (3D) location information corresponding to the apparatus and a surrounding object included in the input image, based on the estimated depth information. ### Claim 35 (depends on 31) The apparatus of claim 31, wherein the input image comprises at least one of a captured monocular image and an image of a captured stereo image. ### Claim 36 (depends on 31) The apparatus of claim 31, wherein the one or more processors are further configured to train the neural network based on a probability distribution that is obtained corresponding to a first image in training data based on obtained first statistical values and second statistical values that are associated with a depth of each pixel in the first image. ### Claim 37 (depends on 31) The apparatus of claim 31, wherein, for the obtaining of the first statistical values, the one or more processors are configured to obtain each of the first statistical values from a first channel of output data of the neural network to which the input image is applied, wherein, for the obtaining of the second statistical values, the one or more processors are configured to obtain each of the second statistical values from a second channel of the output data of the neural network, and wherein the output of the neural network represents the neural network as having been trained on a probability distribution that is obtained corresponding to a first image in training data based on obtained first statistical values and second statistical values that are associated with a depth of each pixel in the first image.