Video compression with adaptive view-dependent lighting removal

A non-transitory computer-readable medium for compressing a video stream for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

• causing a data store to retrieve, from a video stream of a scene captured by one or more image capture devices, base vantage data comprising base vantage color data depicting the scene from a base vantage location, and target vantage data comprising target vantage color data depicting the scene from a target vantage location;
• reprojecting the base vantage data to the target vantage location to obtain reprojected target vantage data;
• comparing the reprojected target vantage data with the target vantage data to obtain residual data;
• compressing the residual data to obtain compressed residual data by removing a subset of the residual data that is of a frequency that does not contribute to perceptual quality compared to a remainder of the residual data;
• and
• causing the data store to store a compressed video stream comprising the base vantage data and the compressed residual data, wherein removing the subset of the residual data comprises applying entropy encoding to the residual data.

A system for compressing a video stream for a virtual reality or augmented reality experience, the system comprising:

• one or more image capture devices configured to capture a video stream of a scene;
• a data store configured to retrieve, from the video stream, base vantage data comprising base vantage color data depicting the scene from a base vantage location, and target vantage data comprising target vantage color data depicting the scene from a target vantage location;
• and
• a processor configured to:
• reproject the base vantage data to the target vantage location to obtain reprojected target vantage data;
• compare the reprojected target vantage data with the target vantage data to obtain residual data;
• and
• compress the residual data to obtain compressed residual data by removing a subset of the residual data that is of a frequency that does not contribute to perceptual quality compared to a remainder of the residual data;
• wherein the data store is further configured to store a compressed video stream comprising the base vantage data and the compressed residual data, wherein removing the subset of the residual data comprises applying entropy encoding to the residual data.