# US 11,908,041 B2 · Object replacement system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,041 B2 - Document ID: US11908041B2 - Status: Granted - Kind: B2 - Application number: 17/648,363 - Filed: 2022-01-19 - Published: 2023-07-20 - Granted: 2024-02-20 - Earliest claimed: 2022-01-19 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Viacheslav Ivanov; Aleksei Zhuravlev - Web page: https://patentgrep.com/patent/US11908041B2 ## CPC - G06T 11/00: Two-dimensional [2D] image generation - G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 20/20: in augmented reality scenes - G06V 40/174: {Facial expression recognition} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving an image that includes a depiction of a real-world environment; processing the image to obtain data indicating presence of a real-world object in the real-world environment; receiving input that selects an AR experience comprising an AR object; determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object; applying a machine learning technique to the image to generate a new image that depicts the real-world environment without the real-world object; and applying the AR object to the new image to generate a modified new image that depicts the real-world environment including the AR object in place of the real-world object. ## Claims ### Claim 1 A method comprising: receiving, by one or more processors, an image that includes a depiction of a real-world environment; processing the image to obtain data indicating presence of a real-world object in the real-world environment; receiving input that selects an augmented reality (AR) experience comprising an AR object; determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience; in response to determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience, applying a machine learning technique to the image to generate a new image that depicts the real-world environment without the real-world object, the machine learning technique trained by: receiving training data comprising a plurality of training images that depict real-world environments with a given real-world object and ground truth images that depict the real-world environments without the given real-world object; applying the machine learning technique to a first training image of the plurality of training images to estimate an image that depicts the real-world environment without the given real-world object; computing a deviation between the estimated image and a ground truth image associated with the first training image; and updating parameters of the machine learning technique based on the computed deviation; and applying the AR object to the new image to generate a modified new image that depicts the real-world environment including the AR object in place of the real-world object. ### Claim 2 (depends on 1) The method of claim 1, further comprising: accessing a plurality of machine learning techniques, a first of the plurality of machine learning techniques being configured to remove a first type of object from a first image that depicts a first environment including the first type of object, a second of the plurality of machine learning techniques being configured to remove a second type of object from a second image that depicts a second environment including the second type of object. ### Claim 3 (depends on 2) The method of claim 2, further comprising: determining a type associated with the real-world object detected on the real-world environment depicted in the image indicated in the obtained data; and selecting the first machine learning technique from the plurality of machine learning techniques as the machine learning technique in response to determining that the type matches the first type of object. ### Claim 4 (depends on 1) The method of claim 1, wherein the real-world object comprises at least one of hardware on a cabinet, a door on a house, a faucet on a sink, a wheel on a vehicle, eyeglasses, headwear, one or more earrings, one or more piercings, or a necklace, and wherein the AR object comprises at least one of AR hardware on a cabinet, an AR door on a house, an AR faucet on a sink, an AR wheel on a vehicle, AR eyeglasses, AR headwear, one or more AR earrings, one or more AR piercings, or an AR necklace. ### Claim 5 (depends on 1) The method of claim 1, wherein applying the AR object to the new image comprises adding a depiction of the AR object to the modified new image. ### Claim 6 (depends on 1) The method of claim 1, wherein the machine learning technique comprises a first neural network. ### Claim 7 (depends on 6) The method of claim 6, further comprising: training a second neural network from the first neural network using a knowledge distillation process, the second neural network being a compact version of the first neural network and being configured to be implemented on a mobile device. ### Claim 8 (depends on 6) The method of claim 6, wherein the deviation is computed based on perceptual loss. ### Claim 9 (depends on 6) The method of claim 6, further comprising generating the training data by performing operations comprising: generating the first training image that depicts a first real-world environment; determining that the first training image includes a depiction of the given real-world object on the first real-world environment; applying a generative model comprising a latent code modification process to remove the given real-world object from the first real-world environment, an output of the generative model comprising an intermediate image that depicts the first real-world environment without the given real-world object; and generating the ground truth image corresponding to the first training image based on the intermediate image. ### Claim 10 (depends on 9) The method of claim 9, wherein the ground truth image corresponding to the first training image is generated by performing operations comprising: applying an environment segmentation model on the first training image to generate a binary mask corresponding to the given real-world object; and combining feature representations of a region of interest in the intermediate image, where the region of interest is given by the binary mask, with the feature representations of the first training image to generate a blended representation as a ground truth image corresponding to the first training image. ### Claim 11 (depends on 10) The method of claim 10, further comprising verifying integrity of the ground truth image corresponding to the first training image by performing operations comprising: applying an attribute extractor on the first training image to generate a first set of attributes; applying the attribute extractor on the ground truth image corresponding to the first training image to generate a second set of attributes; and determining that one or more verification criteria are satisfied by comparing the first and second sets of attributes. ### Claim 12 (depends on 11) The method of claim 11, wherein the one or more verification criteria comprises: an indication that the first set of attributes indicates presence of the given real-world object and that the second set of attributes indicates absence of the given real-world object; a demographic of the first set of attributes matching the demographic in the second set of attributes; a scene illumination of the first set of attributes matching the scene illumination in the second set of attributes; and a pose of the first set of attributes matching the pose in the second set of attributes. ### Claim 13 (depends on 12) The method of claim 12, wherein the one or more verification criteria further comprises a facial expression of the first set of attributes matching the facial expression in the second set of attributes. ### Claim 14 (depends on 9) The method of claim 9, further comprising: computing a first quantity of images in the training data that is associated with a first demographic; computing a second quantity of images in the training data that is associated with a second demographic; and selecting a new image associated with the first demographic for adding to the training data in response to determining that the second quantity exceeds the first quantity by more than a threshold value. ### Claim 15 (depends on 9) The method of claim 9, further comprising searching attributes associated with a collection of images to identify the first training image that is associated with an attribute indicating presence of the given real-world object. ### Claim 16 (depends on 1) The method of claim 1, wherein the image is a frame of a real-time video feed received from a client device of a person. ### Claim 17 (depends on 1) The method of claim 1, further comprising: receiving a video comprising the image depicting the real-world object being moved from a first position to a second position, the second position being closer in proximity to the real-world environment than the first position; and as the real-world object is being moved, applying the machine learning technique to gradually fade and remove the real-world object from the video. ### Claim 18 A system comprising: at least one processor configured to perform operations comprising: receiving an image that includes a depiction of a real-world environment; processing the image to obtain data indicating presence of a real-world object in the real-world environment; receiving input that selects an augmented reality (AR) experience comprising an AR object; determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience; in response to determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience, applying a machine learning technique to the image to generate a new image that depicts the real-world environment without the real-world object, the machine learning technique trained by: receiving training data comprising a plurality of training images that depict real-world environments with a given real-world object and ground truth images that depict the real-world environments without the given real-world object; applying the machine learning technique to a first training image of the plurality of training images to estimate an image that depicts the real-world environment without the given real-world object; computing a deviation between the estimated image and a ground truth image associated with the first training image; and updating parameters of the machine learning technique based on the computed deviation; and applying the AR object to the new image to generate a modified new image that depicts the real-world environment including the AR object in place of the real-world object. ### Claim 19 (depends on 18) The system of claim 18, wherein the operations further comprise accessing a plurality of machine learning techniques, a first of the plurality of machine learning techniques being configured to remove a first type of object from a first image that depicts a first environment including the first type of object, a second of the plurality of machine learning techniques being configured to remove a second type of object from a second image that depicts a second environment including the second type of object. ### Claim 20 A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: receiving an image that includes a depiction of a real-world environment; processing the image to obtain data indicating presence of a real-world object in the real-world environment; receiving input that selects an augmented reality (AR) experience comprising an AR object; determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience; in response to determining that the real-world object detected in the real-world environment depicted in the image indicated in the obtained data corresponds to the AR object of the AR experience, applying a machine learning technique to the image to generate a new image that depicts the real-world environment without the real-world object, the machine learning technique trained by: receiving training data comprising a plurality of training images that depict real-world environments with a given real-world object and ground truth images that depict the real-world environments without the given real-world object; applying the machine learning technique to a first training image of the plurality of training images to estimate an image that depicts the real-world environment without the given real-world object; computing a deviation between the estimated image and a ground truth image associated with the first training image; and updating parameters of the machine learning technique based on the computed deviation; and applying the AR object to the new image to generate a modified new image that depicts the real-world environment including the AR object in place of the real-world object.