High-quality rendering on resource-constrained devices based on view optimized RGBD mesh

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

• receive, from a head-mounted device, a first viewpoint associated with the head-mounted device;
• access a 3D mesh of a virtual scene;
• select a portion of the 3D mesh based on the first viewpoint;
• generate, based on the selected portion of the 3D mesh, an image and a corresponding depth map of the virtual scene that includes one or more depth cliff regions;
• applying depth dilation in each of the one or more depth cliff regions at least in part by identifying one or more nearby points around a depth cliff region that are within a first threshold distance from the depth cliff region and identifying one or more further-away points around the depth cliff region that are over a second threshold distance from the depth cliff region;
• generate a simplified 3D mesh based on the depth map, wherein the simplified 3D mesh has fewer primitives than the selected portion of 3D mesh of the virtual scene;
• generate a texture for the simplified 3D mesh based on the image;
• and
• send the simplified 3D mesh and the texture to the head-mounted device, wherein the simplified 3D mesh and the texture are configured to be used for rendering the virtual scene from one or more viewpoints different from the first viewpoint.

A system comprising:

• one or more processors;
• and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
• receive, from a head-mounted device, a first viewpoint associated with the head-mounted device;
• access a 3D mesh of a virtual scene that includes one or more depth cliff regions;
• applying depth dilation in each of the one or more depth cliff regions at least in part by identifying one or more nearby points around a depth cliff region that are within a first threshold distance from the depth cliff region and identifying one or more further-away points around the depth cliff region that are over a second threshold distance from the depth cliff region;
• select a portion of the 3D mesh based on the first viewpoint;
• generate, based on the selected portion of the 3D mesh, an image and a corresponding depth map of the virtual scene;
• generate a simplified 3D mesh based on the depth map, wherein the simplified 3D mesh has fewer primitives than the selected portion of 3D mesh of the virtual scene;
• generate a texture for the simplified 3D mesh based on the image;
• and
• send the simplified 3D mesh and the texture to the head-mounted device, wherein the simplified 3D mesh and the texture are configured to be used for rendering the virtual scene from one or more viewpoints different from the first viewpoint.