# US 12,602,676 B2 · Image-based user pose detection for user action prediction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,676 B2 - Document ID: US12602676B2 - Status: Granted - Kind: B2 - Application number: 18/499,154 - Filed: 2023-10-31 - Published: 2025-05-01 - Granted: 2026-04-14 - Earliest claimed: 2023-10-10 - Assignees: Maplebear Inc. - Applicants: Maplebear Inc. - Inventors: Ganglu Wu - Web page: https://patentgrep.com/patent/US12602676B2 ## CPC - G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals - G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)} - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system may access a first set of images captured by cameras coupled to a shopping cart, wherein each image depicts a user associated with the shopping cart. A system may apply a pose detection model to each of the images to predict a user's pose. A system may apply an action prediction model to the set of images and the predicted poses to predict whether the user performed an action to change the contents of a storage area of the shopping cart. A system may, responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action. A system may update an item list of the user based on the change action and the identified item. ## Claims ### Claim 1 A method comprising: accessing a first set of images captured by one or more cameras coupled to a shopping cart, wherein each image of the first set of images depicts a user associated with the shopping cart; applying a pose detection model to each of the set of images to predict a pose of the user depicted in each of the first set of images, wherein the pose detection model is a machine-learning model trained to predict the pose of a user depicted in an image; applying an action prediction model to the set of images and the predicted poses to predict whether the user performed a change action, wherein the change action is an action performed by the user whereby the user changes contents of a storage area of the shopping cart, and wherein the action prediction model is a machine-learning model trained to predict whether a user performed a change action to a shopping cart based on images depicting the user and the poses of the user in the images; responsive to predicting that the user performed a change action, applying an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action, wherein applying the item identification model to the second set of images of the storage area of the shopping cart to identify an item associated with the change action further comprises: responsive to detecting the change action, accessing a previous list of items that were in the cart before the action; and inputting the images, the previous list, and the change action into the item identification model to predict which item was associated with the change action; and updating, based on the change action and the identified item, an item list of the user. ### Claim 2 (depends on 1) The method of claim 1, wherein the first set of images comprises a subset of the second set of images. ### Claim 3 (depends on 1) The method of claim 1, wherein updating the item list of the user comprises displaying the updated item list on a user interface of the shopping cart. ### Claim 4 (depends on 1) The method of claim 1, wherein the action prediction model predicts a type of the change action, and wherein the method further comprises updating the item list of the user based on the type of the change action. ### Claim 5 (depends on 4) The method of claim 4, wherein predicting the type of the change action comprises predicting at least one of: an addition of a new item to the shopping cart or a removal of an item from the shopping cart. ### Claim 6 (depends on 1) The method of claim 1, further comprising: applying pose detection model to sensor data captured by sensors coupled to the shopping cart to predict the pose. ### Claim 7 A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to: access a first set of images captured by one or more cameras coupled to a shopping cart, wherein each image of the set of images depicts a user associated with the shopping cart; apply a pose detection model to each of the set of images to predict a pose of the user depicted in each of the set of images, wherein the pose detection model is a machine-learning model trained to predict the pose of a user depicted in an image; apply an action prediction model to the set of images and the predicted poses to predict whether the user performed a change action, wherein the change action is an action whereby the user changes contents of a storage area of the shopping cart, and wherein the action prediction model is a machine-learning model trained to predict whether a user performed a change action to a shopping cart based on images depicting the user and poses of the user in the images; responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action, wherein applying the item identification model to the second set of images of the storage area of the shopping cart to identify an item associated with the change action further comprises: responsive to detecting the change action, accessing a previous list of items that were in the cart before the action; and inputting the images, the previous list, and the change action into the item identification model to predict which item was associated with the change action; and update an item list of the user based on the change action and the identified item. ### Claim 8 (depends on 7) The non-transitory computer-readable storage medium of claim 7, wherein the first set of images comprises a subset of the second set of images. ### Claim 9 (depends on 7) The non-transitory computer-readable storage medium of claim 7, wherein the instructions to update the item list of the user based on the change action and the identified item further comprises instructions to display the updated item list on a user interface of the shopping cart. ### Claim 10 (depends on 7) The non-transitory computer-readable storage medium of claim 7, wherein the change action is one of a set of possible actions and wherein the set of possible actions includes at least addition of a new item to the shopping cart, and removal of an item from the shopping cart. ### Claim 11 (depends on 7) The non-transitory computer-readable storage medium of claim 7, wherein the change action is the user adding an item to the shopping cart, and wherein the instructions to update of the item list further comprises instructions to add the identified item to the item list. ### Claim 12 (depends on 7) The non-transitory computer-readable storage medium of claim 7, wherein the change action is the user removing an item from the shopping cart, and wherein the updating of the item list further comprises removing the identified item from the item list. ### Claim 13 (depends on 7) The non-transitory computer-readable storage medium of claim 7, wherein instructions to access a first set of images captured by one or more cameras coupled to a shopping cart further comprises instructions to access sensor data captured by sensors coupled to the shopping cart; and instructions to apply the pose detection model to each of the set of images to predict a pose of the user further comprises instructions to apply the pose detection model to the set of images and the sensor data. ### Claim 14 (depends on 7) The non-transitory computer-readable storage medium of claim 7, further comprising instructions to access a user history associated with the user of the shopping cart. ### Claim 15 A system comprising: at least one processor; and memory storing non-transitory computer-readable storage instructions, that when executed by at least one processor, cause the at least one processor to: access a first set of images captured by one or more cameras coupled to a shopping cart, wherein each image of the set of images depicts a user associated with the shopping cart; apply a pose detection model to each of the set of images to predict a pose of the user depicted in each of the set of images, wherein the pose detection model is a machine-learning model trained to predict the pose of a user depicted in an image; apply an action prediction model to the set of images and the predicted poses to predict whether the user performed a change action, wherein the change action is an action whereby the user changes contents of a storage area of the shopping cart, and wherein the action prediction model is a machine-learning model trained to predict whether a user performed a change action to a shopping cart based on images depicting the user and poses of the user in the images; responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action, wherein applying the item identification model to the second set of images of the storage area of the shopping cart to identify an item associated with the change action further comprises: responsive to detecting the change action, accessing a previous list of items that were in the cart before the action; and inputting the images, the previous list, and the change action into the item identification model to predict which item was associated with the change action; and update an item list of the user based on the change action and the identified item. ### Claim 16 (depends on 15) The system of claim 15, wherein the instructions that cause the at least one processor to update the item list of the user based on the change action and the identified item further comprises instructions to display the updated item list on a user interface of the shopping cart. ### Claim 17 (depends on 15) The system of claim 15, wherein the action prediction model predicts a type of the change action, and wherein the instructions further cause the processor to update the item list of the user based on the type of the change action. ### Claim 18 (depends on 17) The system of claim 17, wherein the instructions that cause the processor to predict the type of the change action comprises instructions to cause the processor to predict at least one of: an addition of a new item to the shopping cart, or a removal of an item from the shopping cart.