Method for depth estimation and head-mounted display
AssigneeVarjo Technologies Oy
InventorMikko Ollila
Life of this patent
Estimating the term…
Abstract
Disclosed is a method for depth estimation including receiving first and second colour images and sparse depth map corresponding to first and second colour images; generating first and second depth prediction maps by providing first and second colour images and sparse depth map as input to neural network, wherein neural network has first branch for receiving first colour image and sparse depth map as input to generate first depth prediction map and second branch for receiving second colour image and sparse depth map as input to generate second depth prediction map, and wherein first and second branches of neural network share weights by inter-branch weight sharing mechanism; and fusing first and second depth prediction maps to generate dense depth map for depth estimation.
Claims13
A method for depth estimation, the method comprising:
- receiving first and second colour images and a sparse depth map corresponding to the first and second colour images;
- generating first and second depth prediction maps by providing the first and second colour images and the sparse depth map as an input to a neural network, wherein the neural network comprises a first branch for receiving the first colour image and the sparse depth map as the input to generate the first depth prediction map and a second branch for receiving the second colour image and the sparse depth map as the input to generate the second depth prediction map, and wherein the first and second branches of the neural network share weights by an inter-branch weight sharing mechanism;
- fusing the first and second depth prediction maps to generate a dense depth map for the depth estimation;
- generating a third depth prediction map by providing the dense depth map and the sparse depth map as input to a third branch of the neural network, wherein the second and third branches of the neural network share weights by an additional inter-branch weight sharing mechanism;
- and
- fusing the third depth prediction map and the dense depth map to generate a modified-dense depth map for the depth estimation.
A head-mounted display device comprising:
- a stereo camera for capturing first and second colour images;
- and
- a processor operatively coupled to the stereo camera, wherein the processor is configured to
- receive the first and second colour images;
- generate a sparse depth map corresponding to the first and second colour images;
- generate first and second depth prediction maps by providing the first and second colour images and the sparse depth map as an input to a neural network, wherein the neural network comprises a first branch for receiving the first colour image and the sparse depth map as the input to generate the first depth prediction map and a second branch for receiving the second colour image and the sparse depth map as the input to generate the second depth prediction map, and wherein the first and second branches of the neural network share weights by an inter-branch weight sharing mechanism;
- fuse the first and second depth prediction maps to generate a dense depth map for the depth estimation;
- generate a third depth prediction map by providing the dense depth map and the sparse depth map as input to a third branch of the neural network, wherein the second and third branches of the neural network share weights by an additional inter-branch weight sharing mechanism, and
- fuse the third depth prediction map and the dense depth map to generate a modified-dense depth map for the depth estimation.
Dependencies are inferred from the claim text.