# US 12,002,080 B2 · Systems and methods for checking and confirming the purchase of merchandise items > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,080 B2 - Document ID: US12002080B2 - Status: Granted - Kind: B2 - Application number: 17/775,414 - Filed: 2020-11-12 - Published: 2022-12-01 - Granted: 2024-06-04 - Earliest claimed: 2019-11-12 - Assignees: Walmart Apollo, LLC - Applicants: Walmart Apollo, LLC - Inventors: Zhichun Xiao; Lingfeng Zhang; Yao Liu; Jon Hammer; Yutao Tang; Sicong Fang; Haining Liu; Yijing Sun; Mingquan Yuan; Shouyi Zhang; Pingjian Yu; Ryan B. Reagan; Tianyi Mao; Shangeetha Ravichandran Susseelaa; Zhenyu Wang; Feiyun Zhu - Web page: https://patentgrep.com/patent/US12002080B2 ## CPC - G06Q 30/06: Buying, selling or leasing transactions - G06Q 30/0629: {by pre-processing results, e.g. ranking or ordering results} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06Q 20/045: {using payment protocols involving tickets} - G06Q 20/0457: {the tickets being sent electronically} - G06Q 20/047: {using payment protocols involving electronic receipts} - G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals - G06Q 20/202: {Interconnection or interaction of plural electronic cash registers [ECR] or to host computer, e.g. network details, transfer of information from host to ECR or from ECR to ECR} - G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing} - G06Q 20/209: {Specified transaction journal output feature, e.g. printed receipt or voice output} - G06Q 20/322: {Aspects of commerce using mobile devices [M-devices]} - G06Q 20/3267: {In-app payments} - G06Q 20/3276: {using a pictured code, e.g. barcode or QR-code, being read by the M-device} - G06Q 20/389: {Keeping log of transactions for guaranteeing non-repudiation of a transaction} - G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)} - G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration} - G07G 3/003: {Anti-theft control (theft alarms G08B13/00)} - G06N 3/045: Combinations of networks - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS ## Abstract There are provided systems and methods relating to checking and confirming merchandise purchased at shopping facilities. In one form, the system includes: a shopping cart containing merchandise items to be purchased; a sales transaction database; a point-of-sales system that creates transaction records identified by transaction identifiers; and a mobile device used by an employee that includes a sensor to scan a paper or digital receipt to obtain the transaction identifier and a camera to capture images of the items in the shopping cart. The system also includes a control circuit that receives the transaction identifier, accesses the database using the identifier to determine the purchased items; analyzes the images of the merchandise items in the shopping cart and creates a computer vision receipt listing detected items; compares the purchased items with the detected items; and takes an action if there is a discrepancy. ## Claims ### Claim 1 A system for checking and confirming the purchase of merchandise at shopping facilities, the system comprising: a shopping cart containing a plurality of merchandise items that are to be purchased by a customer at a shopping facility; a sales transaction database configured to store electronic transaction records involving the sales of one or more merchandise items at the shopping facility; a point-of-sales system that records a sales transaction of the plurality of merchandise items and that generates a barcode or QR code on a paper or digital receipt corresponding to the sales transaction, the point-of-sales system creating an electronic transaction record identified by the barcode or QR code, the electronic transaction record being stored in the sales transaction database; a mobile electronics device comprising: a barcode or QR code scanner; and a camera configured to capture an image sequence of the merchandise items in the shopping cart; a control circuit at the mobile electronics device configured to: receive the barcode or QR code; access the sales transaction database using the barcode or QR code to determine the purchased merchandise items; analyze the image sequence of the merchandise items in the shopping cart using a computer vision algorithm to identify the merchandise items based on the image sequence and create a computer vision receipt listing the merchandise items identified from the image sequence; compare the purchased merchandise items indicated by the barcode or QR code on the paper or digital receipt with the identified merchandise items from the computer vision receipt, and determine a discrepancy; and cause to display on the mobile electronics device the discrepancy, the display including one or both of a list of unpaid items or images of unpaid items; and an automated vehicle configured to receive a control signal for retrieving an unpaid merchandise item. ### Claim 2 (depends on 1) The system of claim 1, wherein the control circuit analyzes the image sequence using one or more convolutional neural networks (CNNs). ### Claim 3 (depends on 1) The system of claim 1, further comprising one or more checkout stations in the shopping facility, the one or more checkout stations configured for at least one of checkout by a cashier and self-checkout by the customer, the point-of-sales system comprising a point-of-sales device being located at the one or more checkout stations. ### Claim 4 (depends on 1) The system of claim 1, further comprising: a mobile electronics device of the customer used to purchase the merchandise items and configured to display a digital receipt at the mobile electronics device following purchase of the merchandise items. ### Claim 5 (depends on 4) The system of claim 4, further comprising an application provided by a retailer and downloaded by the customer to the customer's mobile electronics device for purchase of merchandise at shopping facilities using the customer's mobile electronics device. ### Claim 6 A method for checking and confirming the purchase of merchandise at shopping facilities, the method comprising: providing a shopping cart, the shopping cart containing a plurality of merchandise items that are to be purchased by a customer at a shopping facility; providing a sales transaction database configured to store electronic transaction records involving the sales of one or more merchandise items at the shopping facility; providing a point-of-sales system that records a sales transaction of the plurality of merchandise items and that generates a barcode or QR code on a paper or digital receipt corresponding to the sales transaction, the point-of-sales system creating an electronic transaction record identified by the barcode or QR code, the electronic transaction record being stored in the sales transaction database; providing a mobile electronics device including a barcode or QR code scanner and a camera configured to capture an image sequence of the merchandise items in the shopping cart; by a control circuit at the mobile electronics device: receiving the barcode or QR code; accessing the sales transaction database using the barcode or QR code to determine the purchased merchandise items; analyzing the image sequence of the merchandise items in the shopping cart using a computer vision algorithm to identify the merchandise items based on the image sequence and create a computer vision receipt listing the merchandise items identified from the image sequence; comparing the purchased merchandise items indicated by the barcode or QR code on the paper or digital receipt with the identified merchandise items from the computer vision receipt, and determine a discrepancy; and causing to display on the mobile electronics device the discrepancy, the display including one or both of a list of unpaid items or images of unpaid items; and receiving, by an automated vehicle, a control signal for retrieving an unpaid merchandise item. ### Claim 7 (depends on 6) The method of claim 6, wherein the control circuit analyzes the image sequence using one or more convolutional neural networks (CNNs). ### Claim 8 (depends on 6) The method of claim 6, further comprising providing one or more checkout stations in the shopping facility, the one or more checkout stations configured for at least one of checkout by a cashier and self-checkout by the customer, the point-of-sales system comprising a point-of-sales device being located at the one or more checkout stations. ### Claim 9 (depends on 6) The method of claim 6, further comprising: by a mobile electronics device of the customer, purchasing the merchandise items and displaying a digital receipt at the mobile electronics device following purchase of the merchandise items. ### Claim 10 (depends on 9) The method of claim 9, further comprising providing an application for download by the customer to the customer's mobile electronics device for purchase of merchandise at shopping facilities using the customer's mobile electronics device. ### Claim 11 A system for checking and confirming the purchase of merchandise at shopping facilities, the system comprising: a shopping cart containing a plurality of merchandise items that are to be purchased by a customer at a shopping facility; a sales transaction database configured to store electronic transaction records involving the sales of one or more merchandise items at the shopping facility; a point-of-sales system that records a sales transaction of the plurality of merchandise items and that generates a barcode or QR code on a paper or digital receipt corresponding to the sales transaction, the point-of-sales system creating an electronic transaction record identified by the barcode or QR code, the electronic transaction record being stored in the sales transaction database; a mobile electronics device comprising: a barcode or QR code scanner; a camera configured to capture an image sequence of the merchandise items in the shopping cart; and a local control circuit configured to: receive the barcode or QR code; analyze the image sequence of the merchandise items in the shopping cart using a computer vision algorithm to identify the merchandise items based on the image sequence and create a computer vision receipt listing the merchandise items identified from the image sequence; transmit the barcode or QR code and the computer vision receipt; a central control circuit configured to: access the sales transaction database using the barcode or QR code to determine the purchased merchandise items; compare the purchased merchandise items indicated by the barcode or QR code on the paper or digital receipt with the identified merchandise items from the computer vision receipt, and determine a discrepancy; and cause to display on the mobile electronics device the discrepancy, the display including one or both of a list of unpaid items or images of unpaid items; and an automated vehicle configured to receive a control signal for retrieving an unpaid merchandise item. ### Claim 12 (depends on 11) The system of claim 11, further comprising one or more checkout stations in the shopping facility, the one or more checkout stations configured for at least one of checkout by a cashier and self-checkout by the customer, the point-of-sales system comprising a point-of-sales device being located at the one or more checkout stations. ### Claim 13 (depends on 11) The system of claim 11, further comprising: a mobile electronics device of the customer used to purchase the merchandise items and configured to display a digital receipt at the mobile electronics device following purchase of the merchandise items. ### Claim 14 (depends on 1) The system of claim 1, wherein: analyzing the image sequence of the merchandise items in the shopping cart comprises comparing images of known products to the image sequence of the merchandise items in the shopping cart. ### Claim 15 (depends on 1) The system of claim 1, wherein: analyzing the image sequence of the merchandise items in the shopping cart comprises using identification information on each merchandise item including at least one of numbers and shapes to identify the merchandise item. ### Claim 16 (depends on 1) The system of claim 1, wherein the automated vehicle is configured to retrieve the unpaid merchandise item based on the control signal. ### Claim 17 (depends on 1) The system of claim 1, wherein the automated vehicle is an aerial drone or an automated ground vehicle.