Back to results
US 11,636,658 B1Granted
Dynamic augmented reality overlay display
AssigneeGoogle Inc.
InventorsYi-Hsuan Tsai, Chen-Ping Yu, Myvictor Tran
Life of this patent
Estimating the term…
Abstract
Three-dimensional occlusion can be used when generating AR display overlays. Depth information can be used to delete portions of an AR element, based on intervening objects between a viewer and the AR element. In cases where the depth information does not impart a complete picture of the intervening objects, additional image processing and object detection systems and techniques can be used to further improve the precision of the occlusion.
Claims15
A method comprising:
- receiving, by one or more processors of a computing device, a video stream of an environment, the video stream including a plurality of image frames, each respective image frame from the plurality of image frames including a projection of an environment point into one or more pixels of the respective image frame;
- generating, by the one or more processors, a depth map of the image frame, the depth map including respective comprising distances of environment points to a user for each of the image pixels of the image frame;
- determining, by the one or more processors, icon location of a virtual reality icon in the environment, the icon including a plurality of icon points;
- determining, by the one or more processors, icon pixel coordinates of the virtual reality icon in the image frame based on a projection of the icon points into icon pixels in the image frame;
- determining, by the one or more processors, icon distances of the plurality of the icon points from the user;
- detecting, by the one or more processors, potential presence of an intervening object in the environment between the user and the icon location, based at least in part on the icon pixel coordinates, icon distances and the depth map;
- performing, using an image segmentation module executed by the one or more processors, image segmentation on an image frame from the plurality of image frames;
- labeling, by the one or more processors, each of the image pixels of the image frame with a corresponding segment label;
- identifying, by the one or more processors, image pixels of the image frame having same corresponding segment labels as same-segment-label pixels;
- determining, by the one or more processors, intervening object pixels, based at least in part on the depth map and the same-segment-label pixels overlapping the icon pixels and having pixel distances less than the icon distances;
- determining, by the one or more processors, icon pixels overlapping the intervening object pixels;
- generating, by the one or more processors, the icon with the overlapping icon pixels erased;
- generating, by the one or more processors, an output image frame based on the image frame, the icon, and the overlapping same-segment-label pixels;
- and
- outputting, by the one or more processors and for display, the output image frame.
12 more elements
A non-transitory computer storage that stores executable program instructions that, when executed by one or more computing devices, configure the one or more computing devices to:
- receive a video stream of an environment, the video stream including a plurality of image frames, each image frame comprising a projection of an environment point into one or more pixels in the image frame;
- generate a depth map of the image frame, the depth map including respective distances of environment points to a user for each of the image pixels of the image frame;
- determine icon location of a virtual reality icon in the environment, the icon including a plurality of icon points;
- determine icon pixel coordinates of the virtual reality icon in the image frame, based on a projection of the icon points into icon pixels in the image frame;
- determine icon distances of the plurality of the icon points from the user;
- detect potential presence of an intervening object in the environment between the user and the icon location, based at least in part on the icon pixel coordinates, icon distances and the depth map;
- perform image segmentation on an image frame from the plurality of image frames;
- label each of the image pixels of the image frame with a corresponding segment label;
- identify image pixels of the image frame having same corresponding segment labels as same-segment-label pixels;
- determine intervening object pixels, based at least in part on the depth map and the same-segment-label pixels overlapping the icon pixels and having pixel distances less than the icon distances;
- determine icon pixels overlapping the intervening object pixels;
- generate the icon with the overlapping icon pixels erased generate an output image frame based on the image frame, the icon, and the overlapping same-segment-label pixels;
- and
- output the output image frame.
11 more elements
A system comprising:
- a processor;
- and
- a storage device configured to store instructions that, when executed by the processor, cause the processor to:
- receive a video stream of an environment, the video stream including a plurality of image frames, each image frame comprising a projection of an environment point into one or more pixels in the image frame;
- generate a depth map of the image frame, the depth map including respective distances of environment points to a user for each of the image pixels of the image frame;
- determine icon location of a virtual reality icon in the environment, the icon including a plurality of icon points;
- determine icon pixel coordinates of the virtual reality icon in the image frame, based on a projection of the icon points into icon pixels in the image frame;
- determine icon distances of the plurality of the icon points from the user;
- detect potential presence of an intervening object in the environment between the user and the icon location, based at least in part on the icon pixel coordinates, icon distances and the depth map;
- perform image segmentation on an image frame from the plurality of image frames;
- label each of the image pixels of the image frame with a corresponding segment label;
- identify image pixels of the image frame having same corresponding segment labels as same-segment-label pixels;
- determine intervening object pixels, based at least in part on the depth map and the same-segment-label pixels overlapping the icon pixels and having pixel distances less than the icon distances;
- determine icon pixels overlapping the intervening object pixels;
- generate the icon with the overlapping icon pixels erased
- generate an output image frame based on the image frame, the icon, and the overlapping same-segment-label pixels;
- and
- output the output image frame.
15 more elements
Dependencies are inferred from the claim text.