# US 12,001,997 B2 · Systems and methods for machine vision based object recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,001,997 B2 - Document ID: US12001997B2 - Status: Granted - Kind: B2 - Application number: 18/350,053 - Filed: 2023-07-11 - Published: 2024-02-01 - Granted: 2024-06-04 - Earliest claimed: 2020-01-24 - Assignees: Synchrony Bank - Applicants: Synchrony Bank - Inventors: Ujjval Patel; Xiaodan Du; Lucas McDonald - Web page: https://patentgrep.com/patent/US12001997B2 ## CPC - G06Q 10/0833: Tracking - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06Q 20/12: specially adapted for electronic shopping systems - G06Q 20/206: {comprising security or operator identification provisions, e.g. password entry} - G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing} - G06Q 20/3224: {Transactions dependent on location of M-devices} - G06Q 30/0201: Market modelling; Market analysis; Collecting market data - G06Q 30/06: Buying, selling or leasing transactions - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/172: {Classification, e.g. identification} - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G07F 9/009: {User recognition or proximity detection} - G07F 9/026: {for alarm, monitoring and auditing in vending machines or means for indication, e.g. when empty} - G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration} - G07G 1/12: electronically operated - G07G 1/14: Systems including one or more distant stations co-operating with a central processing unit - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G07F: COIN-FREED OR LIKE APPARATUS (coin sorting G07D3/00; coin testing G07D5/00; {handling coins or paper currencies apart from payment activated apparatus G07D; payment architectures, schemes or protocols G06Q20/00}) - G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS ## Abstract The present disclosure is related to object recognition and tracking using multi-camera driven machine vision. In one aspect, a method includes capturing, via a multi-camera system, a plurality of images of a user, each of the plurality of images representing the user from a unique angle; identifying, using the plurality of images, the user; detecting, throughout a facility, an item selected by the user; creating a visual model of the item to track movement of the item throughout the facility; determining, using the visual model, whether the item is selected for purchase; and detecting that the user is leaving the facility; and processing a transaction for the item when the item is selected for purchase and when the user has left the facility. ## Claims ### Claim 1 A computer-implemented method comprising: receiving one or more images captured by cameras associated with a facility; applying a machine-learning model to the one or more images to generate an output indicative of whether an item located within the facility has been selected; detecting a selection of the item based on the generated output; associating the selected item with a user; generating a three-dimensional representation of the selected item, wherein the three-dimensional representation is identified from the one or more images; tracking in real-time a movement of the selected item throughout the facility, wherein tracking includes determining a location of the three-dimensional representation relative to the facility; determining that the user has left the facility based on the tracked movement of the selected item; and processing a transaction for the selected item after determining that that the user has left the facility. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein generating the three-dimensional representation includes modifying the three- dimensional representation into another representation associated with a different dimensional space. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the machine-learning model includes a convolutional neural network. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the transaction is processed when the tracked movement indicates that the selected item is exiting the facility. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: associating the selected item with a user; determining that the user has left the facility, wherein the tracked movement indicates that the selected item continues to remain within the facility; and determining a deselection of the selected item based on the indication. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the machine-learning model is trained using sample images that depict items associated with the facility. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein generating the three-dimensional representation includes calibrating the one or more images to create a consensus between images that represent the selected item. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the machine-learning model is trained based on a loss determined between the generated output and a label provided for the one or more images. ### Claim 9 A system, comprising: one or more processors; and memory storing thereon instructions that, as a result of being executed by the one or more processors, cause the system to perform operations comprising: receiving one or more images captured by cameras associated with a facility; applying a machine-learning model to the one or more images to generate an output indicative of whether an item located within the facility has been selected; detecting a selection of the item based on the generated output; associating the selected item with a user; generating a three-dimensional representation of the selected item, wherein the three-dimensional representation is identified from the one or more images; tracking in real-time a movement of the selected item throughout the facility, wherein tracking includes determining a location of the three-dimensional representation relative to the facility; determining that the user has left the facility based on the tracked movement of the selected item; and processing a transaction for the selected item after determining that that the user has left the facility. ### Claim 10 (depends on 9) The system of claim 9, wherein generating the three-dimensional representation includes modifying the three-dimensional representation into another representation associated with a different dimensional space. ### Claim 11 (depends on 9) The system of claim 9, wherein the machine-learning model includes a convolutional neural network. ### Claim 12 (depends on 9) The system of claim 9, wherein the transaction is processed when the tracked movement indicates that the selected item is exiting the facility. ### Claim 13 (depends on 9) The system of claim 9, wherein the instructions further cause the system to perform operations comprising: associating the selected item with a user; determining that the user has left the facility, wherein the tracked movement indicates that the selected item continues to remain within the facility; and determining a deselection of the selected item based on the indication. ### Claim 14 (depends on 9) The system of claim 9, wherein the machine-learning model is trained using sample images that depict items associated with the facility. ### Claim 15 (depends on 9) The system of claim 9, wherein generating the three-dimensional representation includes calibrating the one or more images to create a consensus between images that represent the selected item. ### Claim 16 (depends on 9) The system of claim 9, wherein the machine-learning model is trained based on a loss determined between the generated output and a label provided for the one or more images. ### Claim 17 A non-transitory, computer-readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to perform operations comprising: receiving one or more images captured by cameras associated with a facility; applying a machine-learning model to the one or more images to generate an output indicative of whether an item located within the facility has been selected; detecting a selection of the item based on the generated output; associating the selected item with a user; generating a three-dimensional representation of the selected item, wherein the three-dimensional representation is identified from the one or more images; tracking in real-time a movement of the selected item throughout the facility, wherein tracking includes determining a location of the three-dimensional representation relative to the facility; determining that the user has left the facility based on the tracked movement of the selected item; and processing a transaction for the selected item after determining that that the user has left the facility. ### Claim 18 (depends on 17) The non-transitory, computer-readable storage medium of claim 17, wherein generating the three-dimensional representation includes modifying the three-dimensional representation into another representation associated with a different dimensional space. ### Claim 19 (depends on 17) The non-transitory, computer-readable storage medium of claim 17, wherein the machine-learning model includes a convolutional neural network. ### Claim 20 (depends on 17) The non-transitory, computer-readable storage medium of claim 17, wherein the transaction is processed when the tracked movement indicates that the selected item is exiting the facility. ### Claim 21 (depends on 17) The non-transitory, computer-readable storage medium of claim 17, wherein the instructions further cause the computer system to perform operations comprising: associating the selected item with a user; determining that the user has left the facility, wherein the tracked movement indicates that the selected item continues to remain within the facility; and determining a deselection of the selected item based on the indication. ### Claim 22 (depends on 17) The non-transitory, computer-readable storage medium of claim 17, wherein the machine-learning model is trained using sample images that depict items associated with the facility. ### Claim 23 (depends on 17) The non-transitory, computer-readable storage medium of claim 17, wherein generating the three-dimensional representation includes calibrating the one or more images to create a consensus between images that represent the selected item. ### Claim 24 (depends on 17) The non-transitory, computer-readable storage medium of claim 17, wherein the machine-learning model is trained based on a loss determined between the generated output and a label provided for the one or more images.