Back to results
US 11,238,596 B1Granted
Object co-segmentation in image data using semantic filter
AssigneeAmazon Technologies, Inc.
InventorsRaghu Deep Gadde, Peter Vincent Gehler
Life of this patent
Estimating the term…
Abstract
Devices and techniques are generally described for object co-segmentation in image data using a semantic filter. In various examples, first image data and second image data may be received. First semantic feature data may be determined for the first image data and second semantic feature data may be determined for the second image data. Filtered semantic feature data corresponding to the first semantic feature data may be generated by filtering the first semantic feature data and the second semantic feature data in a semantic feature space. A determination may be made that a first pixel corresponds to a first class of an object, where at least one object of the first class is represented in the first image data and the second image data.
Claims18
A method of co-segmenting objects in image data, the method comprising:
- receiving a first frame of image data and a second frame of image data, wherein the first frame, when rendered on a display, includes a first object of a first class, and wherein the second frame, when rendered on the display, includes a second object of the first class;
- sending the first frame and the second frame to a first convolutional neural network (CNN);
- determining, from a hidden layer of the first CNN, first semantic feature data representing the first frame of image data;
- determining, from the hidden layer of the first CNN, second semantic feature data representing the second frame of image data;
- determining, for the first frame of image data, a first super-pixel comprising a first plurality of contiguous pixels of the first frame of image data, wherein the first super-pixel represents a portion of the first object;
- determining, for the second frame of image data, a second super-pixel comprising a second plurality of contiguous pixels of the second frame of image data, wherein the second super-pixel represents a portion of the second object, wherein the portion of the first object corresponds to the portion of the second object;
- determining, using a Gaussian filter, a similarity in a semantic feature space between the first semantic feature data for the first super-pixel and the second semantic feature data for the second super-pixel;
- determining that the first super-pixel and the second super-pixel correspond to a common object of the first frame of image data and the second frame of image data;
- generating a first segmentation mask for the first frame, wherein the first plurality of contiguous pixels are labeled as foreground image data in the first segmentation mask;
- and
- generating a second segmentation mask for the second frame, wherein the second plurality of contiguous pixels are labeled as the foreground image data in the second segmentation mask.
8 more elements
A method of image segmentation, comprising:
- receiving first image data i and second image data j;
- determining first semantic feature data for the first image data i;
- determining second semantic feature data for the second image data j;
- determining filtered semantic feature data corresponding to the first semantic feature data by filtering the first semantic feature data and the second semantic feature data in a semantic feature space, wherein the filtered semantic feature data represents a semantic similarity between a first portion of the first image data i and other portions of the first image data i and between the first portion of the first image data i and portions of the second image data j, wherein the filtered semantic feature data is determined by computing ΣjΣtk(F(i, s), F(j, t))F(j, t) with a filter kernel k, where F(i, s) represents the first semantic feature data for a first super-pixel s in the first image data i, and where F(j, t) represents the second semantic feature data for a second super-pixel t in second image data j;
- determining that a first pixel corresponding to the filtered semantic feature data corresponds to a first class of object, wherein the first class of object is represented in the first image data i and the second image data j;
- and
- generating a first segmentation mask of the first image data i, wherein the first pixel is labeled as foreground image data in the first segmentation mask.
4 more elements
A system comprising:
- at least one processor;
- and
- a non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to program the processor to perform a method comprising:
- receiving first image data i and second image data j;
- determining first semantic feature data for the first image data i;
- determining second semantic feature data for the second image data j;
- determining filtered semantic feature data corresponding to the first semantic feature data by filtering the first semantic feature data and the second semantic feature data in a semantic feature space, wherein the filtered semantic feature data represents a semantic similarity between a first portion of the first image data i and other portions of the first image data i and between the first portion of the first image data i and portions of the second image data j, wherein the filtered semantic feature data is determined by computing ΣjΣtk(F(i, s), F(j, t))F(j, t) with a filter kernel k, where F(i, s) represents the first semantic feature data for a first super-pixel s in the first image data i, and where F(j, t) represents the second semantic feature data for a second super-pixel t in second image data j;
- determining that a first pixel corresponding to the filtered semantic feature data corresponds to a first class of object, wherein the first class of object is represented in the first image data i and the second image data j;
- generating a first segmentation mask of the first image data i, wherein the first pixel is labeled as foreground image data in the first segmentation mask;
- and
- storing the first segmentation mask in the non-transitory computer-readable memory.
8 more elements
Dependencies are inferred from the claim text.