US 12,749,198 B2Granted

Systems for determining image masks using multiple input images

AssigneeAmazon Technologies, Inc.

InventorsQianli Feng, Raghu Deep Gadde, Aleix Margarit Martinez

Life of this patent

Estimating the term…

Loading patent…

Abstract

To identify sets of pixels in a first image that correspond to different objects or a background, a first image is provided to a Generative Adversarial Network (GAN). The GAN determines alternate images that retain the structural characteristics of the first image, such as the locations and shapes of objects, while modifying style characteristics, such as the colors of pixels. The images generated by the GAN may then be analyzed, such as by using a k-means clustering algorithm, to determine sets of pixels at the same location that change color in a similar manner across the set of images. A set of pixels that changes in a similar manner across the images generated by the GAN may be used as a mask representing an object or background to enable modification of the image without interfering with other objects.

Claims20

Dependent claims

A system comprising:

one or more non-transitory memories storing computer-executable instructions;
and
one or more hardware processors to execute the computer-executable instructions to:
access a first image having a first set of structural characteristics and a first set of style characteristics, wherein the first set of structural characteristics is associated with one or more of locations or shapes of objects depicted within the first image, and wherein the first set of style characteristics is associated with colors of pixels within the first image;
provide the first image to a Generative Adversarial Network (GAN) that is trained to generate images having alternate style characteristics based on input images;
receive, from the GAN, a plurality of second images, wherein each second image of the plurality of second images includes:
the first set of structural characteristics;
and
a respective second set of style characteristics having at least one first style characteristic that differs from the first set of style characteristics, wherein each respective set of style characteristics further includes at least one second style characteristic that differs from at least one other respective set of style characteristics;
determine based on at least a subset of pixels of the plurality of second images, a tensor, wherein the tensor associates a location of a pixel within a respective image with a respective color value of the pixel for each second image of the plurality of second images;
use a clustering algorithm to determine, based on the tensor, a first set of pixels associated with a first set of changes in color values that occur within at least a threshold portion of images included in the plurality of second images;
use the clustering algorithm to determine, based on the tensor, a second set of pixels associated with a second set of changes in color values that occur within the at least a threshold portion of images included in the plurality of second images;
generate, based on the first set of pixels being associated with the first set of changes in color values, first mask data associated with the first image;
and
generate, based on the second set of pixels being associated with the second set of changes in color values, second mask data associated with the first image.

12 more elements

A system comprising:

one or more non-transitory memories storing computer-executable instructions;
and
one or more hardware processors to execute the computer-executable instructions to:
provide a first image to a machine learning system that is trained to generate images having alternate characteristics based on input images, wherein the first image includes a first set of characteristics associated with one or more of locations or shapes of objects depicted within the first image and a second set of characteristics;
receive, from the machine learning system, a plurality of second images, each second image having the first set of characteristics and a respective third set of characteristics that differs from the second set of characteristics and differs from at least one third set of characteristics of at least one other second image of the plurality of second images;
determine, based on the plurality of second images, at least a first set of pixels associated with a first set of changes in the respective third set of characteristics, wherein the first set of changes occurs within at least a threshold portion of images included in the plurality of second images, and a second set of pixels associated with a second set of changes in the respective third set of characteristics, wherein the second set of changes occurs within the at least a threshold portion of images included in the plurality of second images;
generate, based on the first set of pixels being associated with the first set of changes, first mask data associated with the first image;
and
generate, based on the second set of pixels being associated with the second set of changes, second mask data associated with the first image.

6 more elements

A system comprising:

one or more non-transitory memories storing computer-executable instructions;
and
one or more hardware processors to execute the computer-executable instructions to:
determine a plurality of first images, wherein:
each first image of the plurality of first images has a first set of characteristics, and
each first image of the plurality of first images has a respective second set of characteristics that differs from a respective second set of characteristics of at least one other first image of the plurality of first images;
determine, based on at least one region of the plurality of first images, a first set of pixels associated with a first set of changes in the respective second set of characteristics, wherein the first set of changes occurs within at least a threshold portion of images included in the plurality of first images;
determine, based on the at least one region of the plurality of first images, a second set of pixels associated with a second set of changes in the respective second set of characteristics, wherein the second set of changes occurs within at least a threshold portion of images included in the plurality of first images;
generate, based on the first set of pixels being associated with the first set of changes, first mask data associated with at least one first image of the plurality of first images, wherein the first mask data indicates the first set of pixels;
and
generate, based on the second set of pixels being associated with the second set of changes, second mask data associated with the at least one first image, wherein the second mask data indicates the second set of pixels.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/193,039Filed 30 Mar 2023
Published: US 2024/0331163 A13 Oct 2024
Granted: 29 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification12 codes

G06T 7/12
Segmentation; Edge detection: edge-based segmentation
G06V 20/70
Scenes; Scene-specific elements: labelling scene content, e.g. deriving syntactic or semantic representations
G06T 7/50
Image analysis: depth or shape recovery
G06T 7/70
Image analysis: determining position or orientation of objects or cameras