Simultaneous localization and mapping for video coding

A method of encoding video data, the method comprising:

• generating a synthetic image based on a composite image and a scene structure map, wherein the composite image is constructed from one or more images that were previously encoded, wherein the scene structure map comprises a scene structure map of a current image of the video data or a scene structure map of an image of the video data that was previously encoded, wherein the scene structure map includes coordinate values for three-dimensional points, which indicate positions and relative depth of the points, within the current image or the image that was previously encoded, wherein generating the synthetic image comprises utilizing camera position and orientation information of the current image to render the synthetic image such that camera position and orientation for the synthetic image and the current image is the same, and wherein generating the synthetic image further comprises:
• interconnecting points of the scene structure map to form a proxy geometry;
• texture mapping the composite image to the proxy geometry to form an image-based model;
• and
• rendering the image-based model to generate the synthetic image;
• determining a residual image based on the synthetic image and the current image, wherein the residual image is indicative of a difference between the current image and the synthetic image, and wherein determining the residual image comprises determining the residual image based on one or more portions of the current image including a background static portion;
• and
• outputting information indicative of the residual image to encode the current image of the video data.

A device for coding video data, the device comprising:

• a video memory configured to store one or more images that were previously coded and that are used to construct a composite image;
• and
• a coder processor configured to:
• generate a synthetic image based on the composite image and a scene structure map, wherein the scene structure map comprises a scene structure map of a current image of the video data or a scene structure map of an image of the video data that was previously coded, wherein the scene structure map includes coordinate values for three-dimensional points, which indicates positions and relative depths of the points, within the current image or the image that was previously coded, wherein to generate the synthetic image, the coder processor is configured to utilize camera position and orientation information of the current image to render the synthetic image such that camera position and orientation for the synthetic image and the current image is the same, and wherein to generate the synthetic image, the coder processor is further configured to:
• interconnect points of the scene structure map to form a proxy geometry;
• texture map the composite image to the proxy geometry to form an image-based model;
• and
• render the image-based model to generate the synthetic image;
• and
• code the current image based on a residual image of the current image, wherein the residual image is indicative of a difference between the current image and the synthetic image, and wherein the residual image is based on one or more portions of the current image including a background static portion.

A non-transitory computer-readable storage medium having instructions stored thereon that when executed cause one or more processors for a device for coding video data to:

• generate a synthetic image based on a composite image and a scene structure map, wherein the composite image is constructed from one or more images that were previously coded, wherein the scene structure map comprises a scene structure map of a current image of the video data or a scene structure map of an image of the video data that was previously coded, wherein the scene structure map includes coordinate values, which indicates positions and relative depths of the points, for three-dimensional points within the current image or the image that was previously coded, wherein the instructions that cause the one or more processors to generate the synthetic image comprise instructions that cause the one or more processors to utilize camera position and orientation information of the current image to render the synthetic image such that camera position and orientation for the synthetic image and the current image is the same, and wherein the instructions that cause the one or more processors to generate the synthetic image comprise instructions that cause the one or more processors to:
• interconnect points of the scene structure map to form a proxy geometry;
• texture map the composite image to the proxy geometry to form an image-based model;
• and
• render the image-based model to generate the synthetic image;
• and
• code the current image based on a residual image of the current image, wherein the residual image is indicative of a difference between the current image and the synthetic image, and wherein the residual image is based on one or more portions of the current image including a background static portion.