US 9,965,865 B1Granted

Image data segmentation using depth data

AssigneeAmazon Technologies, Inc.

InventorsAmit Kumar Agrawal, Abhishek Singh, Visesh Uday Kumar Chari, Prakash Ramu, Lelin Zhang, Qiang Liu, Yadunandana Nagaraja Rao, Rohith Mysore Vijaya Kumar, Weijian Wang

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Devices and techniques are generally described for segmentation of image data using depth data. In various examples, color image data may be received from a digital camera. In some examples, depth image data may be received from a depth sensor. In various examples, the depth image data may be separated into a plurality of clusters of depth image data, wherein each cluster is associated with a respective range of depth values. In some further examples, a determination may be made that a first cluster of image data corresponds to an object of interest, such as a human subject, in the image data. In various examples, pixels of the first cluster may be encoded with foreground indicator data. In some further examples, segmented image data may be generated. The segmented image data may comprise pixels encoded with the foreground indicator data.

Claims20

Dependent claims

A method for segmenting human image data from background image data, the method comprising:

generating color image data representing a human in an environment using a digital camera;
generating depth data representing the human in the environment using a depth sensor;
separating the depth data into a plurality of clusters of depth data, wherein each cluster of the plurality of clusters is associated with a respective range of depth values;
determining a first cluster from the plurality of clusters, wherein the first cluster comprises depth data at least partially corresponding to the human;
encoding pixels in the first cluster with foreground indicator data to classify the pixels in the first cluster as foreground;
determining a first average three-dimensional position of the pixels of the first cluster;
determining a second average three-dimensional position of pixels of a second cluster from the plurality of clusters;
determining that the first average three-dimensional position corresponds more closely to the second average three-dimensional position relative to other clusters of the plurality of clusters;
determining that the second cluster at least partially corresponds to the human;
encoding the pixels of the second cluster with the foreground indicator data to classify the pixels of the second cluster as foreground;
associating a first pixel of the depth data with a second pixel of the color image data;
identifying a first portion of the color image data for which no corresponding depth information is available in the depth data;
determining a first average color value of the first portion of the color image data;
determining a second average color value of a second portion of pixels, wherein the second portion of pixels correspond to pixels of the first cluster in the depth data;
determining that the first average color value corresponds more closely to the second average color value relative to other portions of the color image data;
determining that the first portion of the color image data at least partially corresponds to the human;
encoding pixels of the first portion of the color image data for which no depth information is available with the foreground indicator data to classify the pixels of the first portion as foreground;
and
generating a segmentation mask, wherein the segmentation mask comprises one or more first pixels classified as foreground and one or more second pixels classified as background.

16 more elements

An image segmentation method comprising:

receiving color image data;
receiving depth image data;
separating the depth image data into a plurality of clusters of depth image data, wherein each cluster is associated with a respective range of depth values;
determining that a first cluster of depth image data corresponds to an object of interest;
encoding pixels of the first cluster with foreground indicator data;
associating a first pixel of the depth image data with a corresponding second pixel of the color image data;
determining that a third pixel of the depth image data corresponds to the object of interest based at least in part on the color image data;
encoding the third pixel of the depth image data with the foreground indicator data;
and
generating first image data, wherein the first image data comprises a first set of pixels of the color image data encoded with the foreground indicator data and a second set of pixels of the color image data encoded with background indicator data.

7 more elements

An image capture device comprising:

a digital camera effective to capture color image data;
a depth sensor effective to capture depth image data;
at least one processor in communication with the digital camera and the depth sensor, the at least one processor effective to execute instructions that when executed by the at least one processor cause the image capture device to perform a method comprising:
separating the depth image data into a plurality of clusters of depth image data, wherein each cluster is associated with a respective range of depth values;
associating a first pixel of the depth image data with a corresponding second pixel of the color image data;
determining that a first cluster of depth image data corresponds to an object of interest;
encoding pixels of the first cluster with foreground indicator data;
determining that a third pixel of the depth image data corresponds to the object of interest based at least in part on the color image data;
encoding the third pixel of the depth image data with the foreground indicator data;
and
generating first image data, wherein the first image data comprises a first set of pixels of the color image data encoded with the foreground indicator data and a second set of pixels of the color image data encoded with background indicator data.

8 more elements

Dependencies are inferred from the claim text.