3d ray-world intersection for xr devices

A method, comprising:

• obtaining video data comprising a video frame;
• obtaining depth map data comprising, for each pixel of a plurality of pixels of the video frame, a depth value of an object visible at a pixel location of the pixel;
• obtaining ray data representative of a ray in three-dimensional space;
• and
• processing the depth map data and the ray data to generate intersection data comprising a three dimensional location of an intersection of the ray with an object visible at a pixel location of a pixel of the plurality of pixels.

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor of a system, cause the system to perform operations comprising:

• obtaining video data comprising a video frame;
• obtaining depth map data comprising, for each pixel of a plurality of pixels of the video frame, a depth value of an object visible at a pixel location of the pixel;
• obtaining ray data representative of a ray in three-dimensional space;
• and
• processing the depth map data and the ray data to generate intersection data comprising a three dimensional location of an intersection of the ray with an object visible at a pixel location of a pixel of the plurality of pixels.