Reduction of the effects of latency for extended reality experiences by split rendering of imagery types

A non-transitory computer-readable medium tangibly embodying computer-executable instructions of a software program, the software program being executable by a processor to provide operations comprising:

• i) distinguishing between two types of imagery to be rendered;
• ii) differentially computing and communicating data, dependent upon the imagery type, such that when the imagery is of:
• a) a first type of the two types of imagery, which is background or distant imagery, it is rendered in a known frame of reference by a server and sent to a user's device from the server where it is transformed to a second frame of reference by the user's device and re-rendered;
• b) a second type of the two types of imagery, which is fast-moving imagery, fast-changing imagery, or imagery very close to a user's point of view relative to the first type imagery, it is rendered locally by the user's device;
• iii) compositing the first type imagery and the second type imagery, by the user device;
• wherein:
• i) a graphic processing unit (GPU) at the server is sent rendering commands, from the server, to generate an image-frame;
• ii) the GPU at the server is sent commands, from the server, to embed metadata providing information that facilitates the association of image-frame-specific information with an image-frame rendered locally by the user's device;
• iii) a user's GPU extracts the metadata to facilitate recompositing of the image by the user's device;
• and
• wherein where a rendered image is a spherical image or a partially updated spherical image that is not dependent upon the user's device's camera direction, the server sends the imagery to the user's device and the user's device re-renders the imagery, taking into account rotational and positional offset of the user's device, to transform it from the known frame of reference to camera space on the user's device.

A system, comprising:

• one or more XR devices, a server, a database and a network which contain software executable by a processor to provide operations comprising:
• i) distinguishing between two types of imagery to be rendered;
• ii) differentially computing and communicating data, dependent upon the imagery type, such that when the imagery is of:
• a) a first type of the two types of imagery, which is background or distant imagery, it is rendered in a known frame of reference by a server and sent to a user's device from the server where it is transformed to a second frame of reference by the user's device and re-rendered;
• b) a second type of the two types of imagery, which is fast-moving imagery, fast-changing imagery, or imagery very close to a user's point of view relative to the first type imagery, it is rendered locally by the user's device;
• iii) compositing the first type imagery and the second type imagery, by the user device;
• wherein:
• i) a graphic processing unit (GPU) at the server is sent rendering commands, from the server, to generate an image-frame;
• ii) the GPU at the server is sent commands, from the server, to embed metadata providing information that facilitates the association of image-frame-specific information with an image-frame rendered locally by the user's device;
• iii) a user's GPU extracts the metadata to facilitate recompositing of the image by the user's device;
• and
• wherein the imagery of the first type is stored on the user device, such that the frame of reference is consistent over time, with each image element being sent from the server to the user device where it is transformed from its own frame of reference to the storage frame and it updates the stored imagery for later transformation into camera space.