Motion estimation using 3D auxiliary data

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• generating a segmentation of three-dimensional point cloud data of recorded media based on continuity data of the three-dimensional point cloud data;
• projecting a representation of the segmented three-dimensional point cloud data onto one or more sides of a three-dimensional bounding box, the representation of the segmented three-dimensional point cloud data being different based on a projected side of the three-dimensional bounding box;
• in response to projecting the representation of the segmented three-dimensional point cloud data onto the one or more sides of the three-dimensional bounding box, generating one or more patches based on the projected representation of the segmented three-dimensional point cloud data;
• generating a first frame of the one or more patches;
• generating first auxiliary information for the first frame;
• generating second auxiliary information for a reference frame, wherein the reference frame includes one or more second patches and the reference frame is a previously encoded and transmitted frame;
• identifying a first patch from the first frame that matches a second patch from the one or more second patches of the reference frame based on the first auxiliary information and the second auxiliary information;
• generating a motion vector candidate between the first patch and the second patch based on a difference between the first auxiliary information and the second auxiliary information;
• and
• performing motion compensation using the motion vector candidate.

One or more non-transitory computer-readable storage media comprising instructions stored thereon that are executable by one or more processing devices and upon such execution cause the one or more processing devices to perform operations comprising:

• generating a segmentation of three-dimensional point cloud data of recorded media based on continuity data of the three-dimensional point cloud data;
• projecting a representation of the segmented three-dimensional point cloud data onto one or more sides of a three-dimensional bounding box, the representation of the segmented three-dimensional point cloud data being different based on a projected side of the three-dimensional bounding box;
• in response to projecting the representation of the segmented three-dimensional point cloud data onto the one or more sides of the three-dimensional bounding box, generating one or more patches based on the projected representation of the segmented three-dimensional point cloud data;
• generating a first frame of the one or more patches;
• generating first auxiliary information for the first frame;
• generating second auxiliary information for a reference frame, wherein the reference frame includes one or more second patches and the reference frame is a previously encoded and transmitted frame;
• identifying a first patch from the first frame that matches a second patch from the one or more second patches of the reference frame based on the first auxiliary information and the second auxiliary information;
• generating a motion vector candidate between the first patch and the second patch based on a difference between the first auxiliary information and the second auxiliary information;
• and
• performing motion compensation using the motion vector candidate.