Rendering 360 depth content
AssigneeFacebook, Inc.
InventorsForrest Samuel Briggs, Michael John Toksvig
Life of this patent
Estimating the term…
Abstract
As user device can receive and display 360 panoramic content in a 360 depth format. 360 depth content can comprise 360 panoramic image data and corresponding depth information. To display 360 depth content, the user device can generate a 3D environment based on the 360 depth content and the current user viewpoint. A content display module on the user device can render 360 depth content using a standard 3D rendering pipeline modified to render 360 depth content. The content display module can use a vertex shader or fragment shader of the 3D rendering pipeline to interpret the depth information of the 360 depth content into the 3D environment as it is rendered.
Claims20
A method comprising:
- receiving, at a user device, one or more visual content frames each comprising, for each of a set of pixels, image pixel data for the pixel and a depth relative to an origin point of the one or more visual content frames for the pixel, wherein each visual content frame includes a range of viewing angles of a scene;
- rendering, at the user device, at least one output image for a viewpoint distinct from the origin point, each output image corresponding to a visual content frame of the one or more visual content frames by:
- generating at least a portion of a 3D sphere corresponding to the range of viewing angles of the visual content frame, the 3D sphere comprising a set of vertices, each vertex associated with a vertex position and a set of texture coordinates associated with one or more pixels of the image pixel data;
- applying a vertex shader to each vertex of the 3D sphere, the vertex shader configured to shift the vertex position of the vertex based on the texture coordinates associated with the vertex and the depth of the visual content frame for one or more pixels associated with the texture coordinates of the vertex;
- rasterizing the 3D sphere including the shifted vertices;
- and
- generating an output image based on the rasterized 3D sphere;
- and
- sending the at least one output image for display to a user of the user device.
A non-transitory computer readable storage medium comprising instructions which, when executed by a processor, cause the processor to perform the steps of:
- receiving, at a user device, one or more visual content frames each comprising, for each of a set of pixels, image pixel data for the pixel and a depth relative to an origin point of the one or more visual content frames for the pixel, wherein each visual content frame includes a range of viewing angles of a scene;
- rendering, at the user device, at least one output image for a viewpoint distinct from the origin point, each output image corresponding to a visual content frame of the one or more visual content frames by:
- generating at least a portion of a 3D sphere corresponding to the range of viewing angles of the visual content frame, the 3D sphere comprising a set of vertices, each vertex associated with a vertex position and a set of texture coordinates associated with one or more pixels of the image pixel data;
- applying a vertex shader to each vertex of the 3D sphere, the vertex shader configured to shift the vertex position of the vertex based on the texture coordinates associated with the vertex and the depth of the visual content frame for one or more pixels associated with the texture coordinates of the vertex;
- rasterizing the 3D sphere including the shifted vertices;
- and
- generating an output image based on the rasterized 3D sphere;
- and
- sending the at least one output image for display to a user of the user device.
Dependencies are inferred from the claim text.