Object replacement system

A system comprising:

• at least one processor configured to perform operations comprising:
• receiving an image that includes a depiction of a real-world environment;
• processing the image to obtain data indicating presence of a real-world object in the real-world environment;
• receiving input that selects an augmented reality (AR) experience comprising an AR object;
• determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience;
• in response to determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience, applying a machine learning technique to the image to generate a new image that depicts the real-world environment without the real-world object, the machine learning technique trained by:
• receiving training data comprising a plurality of training images that depict real-world environments with a given real-world object and ground truth images that depict the real-world environments without the given real-world object;
• applying the machine learning technique to a first training image of the plurality of training images to estimate an image that depicts the real-world environment without the given real-world object;
• computing a deviation between the estimated image and a ground truth image associated with the first training image;
• and
• updating parameters of the machine learning technique based on the computed deviation;
• and
• applying the AR object to the new image to generate a modified new image that depicts the real-world environment including the AR object in place of the real-world object.

A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

• receiving an image that includes a depiction of a real-world environment;
• processing the image to obtain data indicating presence of a real-world object in the real-world environment;
• receiving input that selects an augmented reality (AR) experience comprising an AR object;
• determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience;
• in response to determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience, applying a machine learning technique to the image to generate a new image that depicts the real-world environment without the real-world object, the machine learning technique trained by:
• receiving training data comprising a plurality of training images that depict real-world environments with a given real-world object and ground truth images that depict the real-world environments without the given real-world object;
• applying the machine learning technique to a first training image of the plurality of training images to estimate an image that depicts the real-world environment without the given real-world object;
• computing a deviation between the estimated image and a ground truth image associated with the first training image;
• and
• updating parameters of the machine learning technique based on the computed deviation;
• and
• applying the AR object to the new image to generate a modified new image that depicts the real-world environment including the AR object in place of the real-world object.