# US 2019/0378205 A1 · Computer vision system and method for automatic checkout > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0378205 A1 - Document ID: US20190378205A1 - Status: Published application - Kind: A1 - Application number: 16/416,009 - Filed: 2019-05-17 - Published: 2019-12-12 - Earliest claimed: 2016-05-09 - Assignees: Grabango Co. - Applicants: Grabango Co. - Inventors: William Glaser; Brian Van Osdol - Web page: https://patentgrep.com/patent/US20190378205A1 ## CPC - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06Q 20/203: {Inventory monitoring} - G06Q 30/0635: {replenishment orders; recurring orders} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G07G 1/0036: {Checkout procedures} - H04N 7/181: {for receiving images from a plurality of remote sources} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system and method for computer vision driven applications in an environment that can include collecting image data across an environment; maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process that includes: classifying objects, tracking object locations, detecting interaction events, instantiating object associations in the environmental object graph, and updating the environmental object graph by propagating change in at least one object instance across object associations; and inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state. The system and method can be applied to automatic checkout, inventory management, and/or other system integrations. ## Claims ### Claim 1 A method for expediting a checkout process comprising: collecting image data across an environment; maintaining an environmental object graph from the image data wherein the environmental object graph is a data representation of computer vision classified objects in space and time across the environment, the environmental object graph comprising at least a subset of objects having probabilistic object associations; wherein maintaining the environmental object graph comprises at least one instance of: in a first region captured in the image data, classifying a first object and at least a shopper object, in the first region, detecting an interaction event between the first object and the shopper object, and updating the environmental object graph whereby the first object is probabilistically associated with the shopper object; inspecting objects that are probabilistically associated with the shopper object and thereby generating a checkout list. ### Claim 2 (depends on 1) The method of claim 1, further comprising, in a second region captured in the image data, detecting the shopper object and initiating a checkout process with the checkout list. ### Claim 3 (depends on 2) The method of claim 2, further comprising accessing an account associated with the shopper object; and wherein initiating the checkout process comprises charging the checkout list to the account after entering the second region. ### Claim 4 (depends on 3) The method of claim 3, wherein initiating the checkout process comprises communicating the checkout list to a checkout processing system in the second region. ### Claim 5 (depends on 4) The method of claim 4, further comprising, at the checkout processing system, populating the checkout processing system with items from the checkout list. ### Claim 6 (depends on 1) The method of claim 1, further comprising accessing an account associated with the shopper object; and in an application instance of the account, presenting the checkout list. ### Claim 7 A method comprising: collecting image data across an environment; maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process comprising: classifying objects and storing corresponding object instances in the environmental object graph, tracking object locations and establishing an association of object instances in an object path, detecting interaction events and, for event instances of a subset of detected interaction events, generating an object association of at least two object instances involved in the interaction event, and updating the environmental object graph comprising propagating change in at least one object instance across object associations; and inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state. ### Claim 8 (depends on 7) The method of claim 7, wherein detecting the interaction events comprises: for at least one interaction event, detecting an object proximity event, and, for at least a second interaction event, detecting an object transformation event. ### Claim 9 (depends on 8) The method of claim 8, wherein detecting the object transformation event can comprise detecting an object transformation selected from the set of: an object appearance event, object disappearance event, and object classification mutation event. ### Claim 10 (depends on 7) The method of claim 7, wherein classifying an object can include identifying an object as one of a set of possible object types, wherein possible object types comprise at least a product object, a compound object, and a person object. ### Claim 11 (depends on 7) The method of claim 7, wherein maintaining the environmental object graph further comprises instantiating an object instance in the environmental object graph as probabilistically having a hierarchical association with at least one object. ### Claim 12 (depends on 7) The method of claim 7, wherein classifying objects comprises retrieving supplementary inputs selected from the set of: purchase patterns, customer purchase history, and product pricing; and classifying an object at least partially based on one of the supplementary inputs. ### Claim 13 (depends on 7) The method of claim 7, wherein collecting image data includes collecting image data at a checkout processing system; and applying a product identifier generated by a POS input as supplemental input in updating the environmental object graph. ### Claim 14 (depends on 7) The method of claim 7, wherein inspecting object state and executing an action can comprise: detecting a low confidence scenario in the environmental object graph; communicating an environmental object graph calibration request specifying at least one region of the environment; receiving calibration data input from the region, and updating the environmental object graph with the calibration data input. ### Claim 15 (depends on 7) The method of claim 7, wherein executing an action comprises providing an inventory management system at least partially driven through the environmental object graph, wherein providing the inventory management system comprises: reporting shelf time of products, reporting product counts, reporting product locations, and reporting container contents. ### Claim 16 (depends on 7) The method of claim 7, wherein classifying objects comprises applying computer-vision driven processes, the computer-vision driven processes including at least image feature extraction and classification and convolution neural networks. ### Claim 17 A system comprising an imaging system configured to collect image data within an environment; a processing engine configured to maintain an environmental object graph from the image data wherein the environmental object graph is a data representation of classified objects in space and time across the environment, the environmental object graph comprising at least a subset of objects having probabilistic object associations; and the processing engine further configured to: classify objects in the image data and store corresponding object instances in the environmental object graph, track object locations and establish an association of object instances in an object path, detect interaction events and, for event instances of a subset of detected interaction events, generate an object association of at least two object instances involved in the interaction event, and update the environmental object graph and propagate change in at least one object instance across object associations. ### Claim 18 (depends on 17) The system of claim 17, wherein the processing engine is further configured to: instantiate a first set of object instances in the environmental object graph with compound object modeling, and instantiate a second set of object instances in the environmental object graph with probabilistic multi-state modeling. ### Claim 19 (depends on 17) The system of claim 17, further comprising a virtual cart management system configured to inspect object instances associated with a shopper object instance in the environmental object graph and generate a checkout list of products. ### Claim 20 (depends on 19) The system of claim 19, the virtual cart management system further configured to access an account associated with the shopper object instance and charge the checkout list to the account when the shopper associated object instance is detected in a checkout region. ### Claim 21 (depends on 19) The system of claim 19, further comprising a checkout processing system wherein the checkout list is communicated to the checkout processing system. ### Claim 22 (depends on 17) The system of claim 17, further comprising a calibration tool that is configured to provide calibration data input to the processing engine.