# US 12,217,236 B2 · Overlap detection for an item recognition system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,236 B2 - Document ID: US12217236B2 - Status: Granted - Kind: B2 - Application number: 17/726,389 - Filed: 2022-04-21 - Published: 2022-10-27 - Granted: 2025-02-04 - Earliest claimed: 2021-04-21 - Assignees: Maplebear Inc. - Applicants: Maplebear Inc. - Inventors: Shiyuan Yang; Shray Chandra - Web page: https://patentgrep.com/patent/US12217236B2 ## CPC - G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing} - G01G 19/4144: {for controlling weight of goods in commercial establishments, e.g. supermarket, P.O.S. systems} - G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/16: {using multiple overlapping images; Image stitching} - G06V 10/23: {based on positionally close patterns or neighbourhood relationships} - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06V 10/945: {User interactive design; Environments; Toolboxes} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 20/60: Type of objects - G06V 20/64: Three-dimensional [3D] objects - G07G 1/0036: {Checkout procedures} - G07G 1/0054: {with control of supplementary check-parameters, e.g. weight or number of articles} - G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration} - H04N 23/60: Control of cameras or camera modules - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G01G: WEIGHING (sorting by weighing B07C5/16) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An item recognition system uses a top camera and one or more peripheral cameras to identify items. The item recognition system may use image embeddings generated based on images captured by the cameras to generate a concatenated embedding that describes an item depicted in the image. The item recognition system may compare the concatenated embedding to reference embeddings to identify the item. Furthermore, the item recognition system may detect when items are overlapping in an image. For example, the item recognition system may apply an overlap detection model to a top image and a pixel-wise mask for the top image to detect whether an item is overlapping with another in the top image. The item recognition system notifies a user of the overlap if detected. ## Claims ### Claim 1 An item recognition system comprising: a receiving surface; a top camera coupled to a top portion of the item recognition system, wherein the top camera is configured to capture images of the receiving surface from a top-down view; one or more peripheral cameras coupled to one or more side portions of the item recognition system, wherein the one or more peripheral cameras are configured to capture images of the receiving surface from different peripheral views; a user interface; a processor; and a non-transitory, computer-readable medium storing instructions that, when executed by the processor, cause the processor to: access a top image comprising an image captured by the top camera, wherein the top image depicts a first item and a second item on the receiving surface; access one or more peripheral images, each comprising an image captured by a peripheral camera of the one or more peripheral cameras; generate a pixel-wise mask for the top image based on the top image, wherein pixel-wise mask indicates one or more portions of the top image where an item is depicted; apply an overlap detection model to the top image, the one or more peripheral images, and the pixel-wise mask to detect whether the first item overlaps with the second item, wherein the overlap detection model is a machine-learning model trained to detect overlapping items in top images based on the top images, peripheral images, and pixel-wise masks of the top images; and responsive to detecting that the first item overlaps with the second item, present a notification of the overlap through the user interface. ### Claim 2 (depends on 1) The item recognition system of claim 1, wherein the top camera and the one or more peripheral cameras are configured to capture 2D images of the receiving surface. ### Claim 3 (depends on 1) The item recognition system of claim 1, wherein the computer-readable medium further stores instructions that cause the processor to: generate a masked image for the top image based on the top image and the pixel-wise mask for the top image; and apply the overlap detection model to the masked image. ### Claim 4 (depends on 1) The item recognition system of claim 1, wherein the computer-readable medium further stores instructions that cause the processor to: generate a depth value for each pixel of the top image based on the top image and the one or more peripheral images; and apply the overlap detection model to the generated depth values of the top image. ### Claim 5 (depends on 1) The item recognition system of claim 1, wherein the computer-readable medium further stores instructions that cause the processor to: responsive to not detecting that the first item overlaps with the second item, identifying the first item and the second item based on the top image, the one or more peripheral images, and the pixel-wise mask. ### Claim 6 (depends on 1) The item recognition system of claim 1, wherein the computer-readable medium further stores instructions that cause the processor to: detect that the first item no longer overlaps with the second item based on a subsequent top image, a subsequent one or more peripheral images, and a subsequent pixel-wise mask. ### Claim 7 (depends on 1) The item recognition system of claim 1, wherein the instructions for notifying the user of the overlap further comprise instructions that cause the processor to: display a message on a display of the user interface notifying the user of the overlap. ### Claim 8 (depends on 1) The item recognition system of claim 1, wherein the pixel-wise mask comprises a set of binary values that indicate which pixels of the top image include the first item or the second item. ### Claim 9 (depends on 1) The item recognition system of claim 1, wherein the receiving surface comprises a weight sensor configured to capture weight sensor data describing a weight of items placed on the receiving surface, and wherein the computer-readable medium further stores instructions that, when executed by the processor, cause the processor to: access weight sensor data captured by the weight sensor, wherein the weight sensor data describes a weight of the first item and the second item; and apply the overlap detection model to the weight sensor data to detect whether the first item overlaps with the second item. ### Claim 10 (depends on 1) The item recognition system of claim 1, wherein the computer-readable medium further stores instructions that cause the processor to: apply the overlap detection model to the one or more peripheral images to detect whether the first item overlaps with the second item in the one or more peripheral images. ### Claim 11 A non-transitory, computer-readable medium storing instructions that, when executed by a processor, cause the processor to: access a top image comprising an image captured by a top camera of an item recognition system, wherein the top image depicts a first item and a second item on a receiving surface of the item recognition system; access one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system; generate a pixel-wise mask for the top image based on the top image, wherein pixel-wise mask indicates one or more portions of the top image where an item is depicted; apply an overlap detection model to the top image, the one or more peripheral images, and the pixel-wise mask to detect whether the first item overlaps with the second item, wherein the overlap detection model is a machine-learning model trained to detect overlapping items in top images based on the top images, peripheral images, and pixel-wise masks of the top images; and responsive to detecting that the first item overlaps with the second item, present a notification of the overlap through a user interface of the item recognition system. ### Claim 12 (depends on 11) The computer-readable medium of claim 11, wherein the top camera and the one or more peripheral cameras are configured to capture 2D images of the receiving surface. ### Claim 13 (depends on 11) The computer-readable medium of claim 11, further storing instructions that cause the processor to: generate a masked image for the top image based on the top image and the pixel-wise mask for the top image; and apply the overlap detection model to the masked image. ### Claim 14 (depends on 11) The computer-readable medium of claim 11, further storing instructions that cause the processor to: generate a depth value for each pixel of the top image based on the top image and the one or more peripheral images; and apply the overlap detection model to the generated depth values of the top image. ### Claim 15 (depends on 11) The computer-readable medium of claim 11, further storing instructions that cause the processor to: responsive to not detecting that the first item overlaps with the second item, identifying the first item and the second item based on the top image, the one or more peripheral images, and the pixel-wise mask. ### Claim 16 (depends on 11) The computer-readable medium of claim 11, further storing instructions that cause the processor to: detect that the first item no longer overlaps with the second item based on a subsequent top image, a subsequent one or more peripheral images, and a subsequent pixel-wise mask. ### Claim 17 (depends on 11) The computer-readable medium of claim 11, wherein the instructions for notifying the user of the overlap further comprise instructions that cause the processor to: display a message on a display of the user interface notifying the user of the overlap. ### Claim 18 (depends on 11) The computer-readable medium of claim 11, wherein the pixel-wise mask comprises a set of binary values that indicate which pixels of the top image include the first item or the second item. ### Claim 19 (depends on 11) The computer-readable medium of claim 11, further storing instructions that cause the processor to: access weight sensor data captured by a weight sensor coupled to the receiving surface, wherein the weight sensor is configured to capture weight sensor data describing a weight of items placed on the receiving surface; and apply the overlap detection model to the weight sensor data to detect whether the first item overlaps with the second item. ### Claim 20 A method comprising: accessing a top image comprising an image captured by a top camera of an item recognition system, wherein the top image depicts a first item and a second item on a receiving surface of the item recognition system; accessing one or more peripheral images, each comprising an image captured by a peripheral camera of one or more peripheral cameras of the item recognition system; generating a pixel-wise mask for the top image based on the top image, wherein pixel-wise mask indicates one or more portions of the top image where an item is depicted; applying an overlap detection model to the top image, the one or more peripheral images, and the pixel-wise mask to detect whether the first item overlaps with the second item, wherein the overlap detection model is a machine-learning model trained to detect overlapping items in top images based on the top images, peripheral images, and pixel-wise masks of the top images; and responsive to detecting that the first item overlaps with the second item, presenting a notification of the overlap through a user interface of the item recognition system. ### Claim 21 (depends on 5) The system of claim 5, wherein the computer-readable medium further stores instructions that cause the processor to: responsive to identifying the first item and the second item, completing a checkout process based on a first identifier associated with the first item and a second identifier associated with the second item.