Method for training convolutional neural network to reconstruct an image and system for depth map generation from an image
AssigneeSamsung Electronics Co., Ltd.
InventorsValery Valerievich Anisimovskiy, Andrey Yurievich Shcherbinin, Sergey Alexandrovich Turko
Life of this patent
Estimating the term…
Abstract
A method for training a convolutional neural network to reconstruct an image. The method includes forming a common loss function basing on the left and right images (IL, IR), reconstructed left and right images (I′L, I′R), disparity maps (dL, dR), reconstructed disparity maps (d′L, d′R) for the left and right images (IL, IR) and the auxiliary images (I″L, I″R) and training the neural network based on the formed loss function.
Claims15
A method for training a convolutional neural network to reconstruct an image, comprising:
- obtaining stereo pair images including left and right images (IL, IR) using a camera;
- inputting each of the left and right images (IL, IR) to a corresponding siamese convolutional neural network for depth map estimation;
- generating high-level feature maps and inverse depth maps for the inputted left and right images (IL, IR), respectively;
- processing the generated high-level feature maps based on the convolutional neural network to identify parameters of the camera;
- generating disparity maps (dL, dR) for the left and right images (IL, IR) by applying an affine transform to the inverse depth maps for the left and right images (IL, IR), respectively, based on the identified camera parameters;
- performing bilinear-interpolation sampling for the left and right images (IL, IR) based on the generated disparity maps (dL, dR) to obtain reconstructed left and right images (IL, I′R), reconstructed disparity maps (d′L, d′R) and auxiliary images (I″L, I″R);
- forming a common loss function based on the left and right images (IL, IR), the reconstructed left and right images (I′L, I′R), the disparity maps (dL, dR), the reconstructed disparity maps (d′L, d′R) and the auxiliary images (I″L, I″R), and
- training the neural network based on the formed loss function.
A method for training a convolutional neural network to reconstruct an image, comprising:
- obtaining stereo pair images including the left and right images (IL, IR) using a camera;
- inputting each of the left and right images (IL, IR) to a corresponding siamese convolutional neural network for depth map estimation;
- generating high-level feature maps, inverse depth maps and correction maps for the inputted left and right images (IL, IR), respectively;
- processing the generated high-level feature maps based on the convolutional neural network to identify parameters of the camera;
- generating disparity maps (dL, dR) for the left and right images (IL, IR) by applying an affine transform to inverse depth maps for the left and right images (IL, IR), respectively, based on the identified camera parameters;
- generating corrected left and right images using the correction maps;
- performing bilinear-interpolation sampling for the corrected left and right images based on the generated disparity maps (dL, dR) to obtain reconstructed left and right images (I′L, I′R), reconstructed disparity maps (d′L, d′R) and auxiliary images (I″L, I″R);
- forming a common loss function based on the corrected left and right images, reconstructed left and right images (I′L, I′R), correction maps, disparity maps (dL, dR), reconstructed disparity maps (d′L, d′R) for the left and right images (IL, IR) and the auxiliary images (I″L, I″R), and
- training the convolutional neural network based on the formed loss function.
Dependencies are inferred from the claim text.