Room layout estimation methods and techniques
AssigneeMagic Leap, Inc.
InventorsChen-Yu Lee, Vijay Badrinarayanan, Tomasz Jan Malisiewicz, Andrew Rabinovich
Life of this patent
Estimating the term…
Abstract
Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.
Claims38
A method for estimating a layout of a room comprising:
- accessing a room image for room layout estimation;
- determining, using an encoder-decoder sub-network of a neural network for estimating a layout of a room and the room image, a plurality of predicted two-dimensional (2D) keypoint maps corresponding to a plurality of room types;
- determining, using the encoder-decoder sub-network, a classifier sub-network of the neural network connected to the encoder-decoder sub-network, and the room image, a predicted room type from the plurality of room types;
- determining, using the plurality of predicted 2D keypoint maps and the predicted room type, a plurality of ordered keypoints associated with the predicted room type;
- and
- determining, using the plurality of ordered keypoints, a predicted layout of the room in the room image.
A method for training a neural network for estimating a layout of a room comprising:
- receiving a training room image, wherein the training room image is associated with a reference room type from a plurality of room types, and reference keypoints associated with a reference room layout;
- generating a neural network for room layout estimation, wherein the neural network comprises an encoder-decoder sub-network configured to output predicted two-dimensional (2D) keypoints associated with a predicted room layout associated with each of the plurality of room types, and a side sub-network connected to the encoder-decoder sub-network configured to output a predicted room type from the plurality of room types;
- and
- optimizing a loss function based on a first loss for the predicted 2D keypoints and a second loss for the predicted room type;
- and
- updating parameters of the neural network based on the optimized loss function.
Dependencies are inferred from the claim text.