Room layout estimation methods and techniques
AssigneeMagic Leap, Inc.
InventorsChen-Yu Lee, Vijay Badrinarayanan, Tomasz Jan Malisiewicz, Andrew Rabinovich
Life of this patent
Estimating the term…
Abstract
Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.
Claims24
A system comprising:
- non-transitory memory configured to store:
- an image of a room;
- and
- a neural network comprising:
- an encoder-decoder sub-network comprising an encoder and a decoder;
- and
- a classifier sub-network in communication with the encoder-decoder sub-network configured to classify a room type associated with the image;
- a hardware processor in communication with the non-transitory memory, the hardware processor programmed to:
- access the image;
- predict, using the encoder, decoder, and the image, two-dimensional (2D) keypoint maps corresponding to room types;
- determine, using the encoder, the classifier sub-network, and the image, a predicted room type from the room types;
- and
- determine a predicted layout of the room in the image based at least in part on the 2D keypoint maps and the predicted room type.
A system comprising:
- non-transitory memory configured to store parameters for the neural network;
- and
- a hardware processor in communication with the non-transitory memory, the hardware processor programmed to:
- receive an image, wherein the image is associated with:
- a reference room type from a plurality of room types, and
- reference keypoints associated with a reference room layout;
- generate a neural network comprising:
- an encoder-decoder sub-network configured to output two-dimensional (2D) keypoints associated with a predicted room layout, wherein the encoder-decoder sub-network comprises an encoder and a decoder, and
- a side sub-network in communication with the encoder-decoder network configured to classify the reference room type associated with the image by outputting a predicted room type from the plurality of room types;
- and
- update the parameters for the neural network based at least in part on an optimized loss function.
A wearable display system comprising:
- an outward-facing imaging system configured to obtain an image of an environment of the wearer of the wearable display system;
- non-transitory memory configured to store the image;
- and
- a hardware processor in communication with the non-transitory memory, the processor programmed to:
- access the image;
- and
- analyze the image to determine a predicted layout of a room in the image, wherein to analyze the image, the processor is programmed to:
- use a neural network to determine two-dimensional (2D) keypoints associated with a room type associated with the image, wherein the neural network comprises:
- an encoder-decoder sub-network configured to output the 2D keypoints, and
- a side sub-network in communication with the encoder-decoder sub-network configured to classify a room type associated with the image by outputting the room type;
- and
- provide the room layout based at least partly on the 2D keypoints and the room type.
A method comprising:
- accessing an image of a room;
- predicting two-dimensional (2D) keypoint maps corresponding to room types based at least in part on the image;
- determining a predicted room type from the room types based at least in part on the image and a side sub-network configured to classify a room type associated with the image;
- and
- determining a predicted layout of the room in the image based at least in part on the 2D keypoint maps and the predicted room type.
Dependencies are inferred from the claim text.