Back to results
US 12,762,526 B2Granted
System for item recognition using computer vision
AssigneeMaplebear Inc.
InventorsShiyuan Yang, Shray Chandra
Life of this patent
Estimating the term…
Abstract
An item recognition system uses a top camera and one or more peripheral cameras to identify items. The item recognition system may use image embeddings generated based on images captured by the cameras to generate a concatenated embedding that describes an item depicted in the image. The item recognition system may compare the concatenated embedding to reference embeddings to identify the item. Furthermore, the item recognition system may detect when items are overlapping in an image. For example, the item recognition system may apply an overlap detection model to a top image and a pixel-wise mask for the top image to detect whether an item is overlapping with another in the top image. The item recognition system notifies a user of the overlap if detected.
Claims16
An item recognition system comprising:
- a receiving surface;
- a top camera coupled to a top portion of the item recognition system, wherein the top camera is configured to capture images of the receiving surface from a top-down view;
- one or more peripheral cameras coupled to one or more side portions of the item recognition system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views;
- a processor;
- and
- a non-transitory, computer-readable medium storing instructions that, when executed by the processor, cause the processor to:
- access a top image comprising an image captured by the top camera;
- access one or more peripheral images, each comprising an image captured by a peripheral camera of the one or more peripheral cameras;
- identify, for each of a plurality of items on the receiving surface, a region of the top image and a region of each of the one or more peripheral images that depicts the item on the receiving surface,
- wherein identifying the region of the top image and the region of each of the one or more peripheral images that depicts an item of the plurality of items comprises:
- generating a pixel-wise mask for the top image and a pixel-wise mask for each of the one or more peripheral images, wherein the pixel-wise masks identify pixel regions of the top image and the one or more peripheral images that depict the plurality of items;
- and
- identifying a pixel region in each of the top image and the one or more peripheral images that corresponds to the item by projecting the pixel regions to identify which pixel regions represent a common region of space;
- generate, for each of the plurality of items, an image embedding for each of the identified regions of the top image and the one or more peripheral images by applying an image embedding model to each of the identified regions, wherein the image embedding model is a machine- learning model trained to generate image embeddings for identified regions of images;
- concatenate, for each of the plurality of items, the image embeddings based on a pre-determined ordering of the top camera and the one or more peripheral cameras to form a concatenated embedding;
- and
- identify each of the plurality of items by comparing the concatenated embedding to one or more reference item embeddings, wherein each reference item embedding is associated with an item identifier, and wherein comparing the concatenated embedding to a reference item embedding of the one or more reference item embeddings comprises:
- computing a distance between the concatenated embedding and the reference item embedding.
15 more elements
A non-transitory, computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
- accessing a top image comprising an image captured by a top camera of an item recognition system, wherein the top camera is configured to capture images of a receiving surface of the item recognition system from a top-down view;
- accessing one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views;
- identifying, for each of a plurality of items on the receiving surface, a region of the top image and a region of each of the one or more peripheral images that depicts the item on the receiving surface, wherein identifying the region of the top image and the region of each of the one or more peripheral images that depicts an item of the plurality of items comprises:
- generating a pixel-wise mask for the top image and a pixel-wise mask for each of the one or more peripheral images, wherein the pixel-wise masks identify pixel regions of the top image and the one or more peripheral images that depict the plurality of items;
- and
- identifying a pixel region in each of the top image and the one or more peripheral images that corresponds to the item by projecting the pixel regions to identify which pixel regions represent a common region of space;
- generating, for each of the plurality of items, an image embedding for each of the identified regions of the top image and the one or more peripheral images by applying an image embedding model to each of the identified regions, wherein the image embedding model is a machine-learning model trained to generate image embeddings for identified regions of images;
- concatenating, for each of the plurality of items, the image embeddings based on a pre-determined ordering of the top camera and the one or more peripheral cameras to form a concatenated embedding;
- and
- identifying each of the plurality of items by comparing the concatenated embedding to one or more reference item embeddings, wherein each reference item embedding is associated with an item identifier, and wherein comparing the concatenated embedding to a reference item embedding of the one or more reference item embeddings comprises:
- computing a distance between the concatenated embedding and the reference item embedding.
8 more elements
A method comprising:
- accessing a top image comprising an image captured by a top camera of an item recognition system, wherein the top camera is configured to capture images of a receiving surface of the item recognition system from a top-down view;
- accessing one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views;
- identifying, for each of a plurality of items on the receiving surface, a region of the top image and a region of each of the one or more peripheral images that depicts the item on the receiving surface, wherein identifying the region of the top image and the region of each of the one or more peripheral images that depicts an item of the plurality of items comprises:
- generating a pixel-wise mask for the top image and a pixel-wise mask for each of the one or more peripheral images, wherein the pixel-wise masks identify pixel regions of the top image and the one or more peripheral images that depict the plurality of items;
- and
- identifying a pixel region in each of the top image and the one or more peripheral images that corresponds to the item by projecting the pixel regions to identify which pixel regions represent a common region of space;
- generating, for each of the plurality of items, an image embedding for each of the identified regions of the top image and the one or more peripheral images by applying an image embedding model to each of the identified regions, wherein the image embedding model is a machine-learning model trained to generate image embeddings for identified regions of images;
- concatenating, for each of the plurality of items, the image embeddings based on a pre- determined ordering of the top camera and the one or more peripheral cameras to form a concatenated embedding;
- and
- identifying each of the plurality of items by comparing the concatenated embedding to one or more reference item embeddings, wherein each reference item embedding is associated with an item identifier, and wherein comparing the concatenated embedding to a reference item embedding of the one or more reference item embeddings comprises:
- computing a distance between the concatenated embedding and the reference item embedding.
8 more elements
Dependencies are inferred from the claim text.