Effects for 3D data in a messaging system

A system comprising:

• a processor;
• and
• a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
• selecting a set of augmented reality content generators from a plurality of available augmented reality content generator based on metadata associated with each respective augmented reality content generator, the metadata including information indicating a corresponding augmented reality content generator includes at least a 3D effect, the set of augmented reality content generators including at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;
• receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;
• capturing image data and depth data using at least one camera of the client device;
• and
• applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:
• generating a depth map using at least the depth data,
• generating a segmentation mask based at least on the image data, and
• performing background inpainting and blurring of the image data using at least segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject, generating the depth map comprises converting a single channel floating point texture into a raw depth map, portions of the single channel floating point texture is sent into multiple lower precision channels, the raw depth map has a lower resolution than the image data;
• generating a 3D message based at least in part on the applied 3D effect;
• and
• rendering a view of the 3D message based at least in part on the applied 3D effect.

A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

• selecting a set of augmented reality content generators from a plurality of available augmented reality content generator based on metadata associated with each respective augmented reality content generator, the metadata including information indicating a corresponding augmented reality content generator includes at least a 3D effect, the set of augmented reality content generators including at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;
• receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect;
• capturing image data and depth data using at least one camera of the client device;
• and
• applying, to the image data and the depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect comprising:
• generating a depth map using at least the depth data,
• generating a segmentation mask based at least on the image data, and
• performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data, the performing the background inpainting comprising performing a diffusion based inpainting technique that fills in a missing region by propagating image content from a boundary between the missing region and a background region to an interior of the missing region, wherein the background region comprises a particular region of the image data without a foreground subject and the missing region includes the foreground subject, generating the depth map comprises converting a single channel floating point texture into a raw depth map, portions of the single channel floating point texture is sent into multiple lower precision channels, the raw depth map has a lower resolution than the image data;
• generating a 3D message based at least in part on the applied 3D effect;
• and
• rendering a view of the 3D message based at least in part on the applied 3D effect.