Image-based user pose detection for user action prediction

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to:

• access a first set of images captured by one or more cameras coupled to a shopping cart, wherein each image of the set of images depicts a user associated with the shopping cart;
• apply a pose detection model to each of the set of images to predict a pose of the user depicted in each of the set of images, wherein the pose detection model is a machine-learning model trained to predict the pose of a user depicted in an image;
• apply an action prediction model to the set of images and the predicted poses to predict whether the user performed a change action, wherein the change action is an action whereby the user changes contents of a storage area of the shopping cart, and wherein the action prediction model is a machine-learning model trained to predict whether a user performed a change action to a shopping cart based on images depicting the user and poses of the user in the images;
• responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action, wherein applying the item identification model to the second set of images of the storage area of the shopping cart to identify an item associated with the change action further comprises:
• responsive to detecting the change action, accessing a previous list of items that were in the cart before the action;
• and
• inputting the images, the previous list, and the change action into the item identification model to predict which item was associated with the change action;
• and
• update an item list of the user based on the change action and the identified item.

A system comprising:

• at least one processor;
• and
• memory storing non-transitory computer-readable storage instructions, that when executed by at least one processor, cause the at least one processor to:
• access a first set of images captured by one or more cameras coupled to a shopping cart, wherein each image of the set of images depicts a user associated with the shopping cart;
• apply a pose detection model to each of the set of images to predict a pose of the user depicted in each of the set of images, wherein the pose detection model is a machine-learning model trained to predict the pose of a user depicted in an image;
• apply an action prediction model to the set of images and the predicted poses to predict whether the user performed a change action, wherein the change action is an action whereby the user changes contents of a storage area of the shopping cart, and wherein the action prediction model is a machine-learning model trained to predict whether a user performed a change action to a shopping cart based on images depicting the user and poses of the user in the images;
• responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action, wherein applying the item identification model to the second set of images of the storage area of the shopping cart to identify an item associated with the change action further comprises:
• responsive to detecting the change action, accessing a previous list of items that were in the cart before the action;
• and
• inputting the images, the previous list, and the change action into the item identification model to predict which item was associated with the change action;
• and
• update an item list of the user based on the change action and the identified item.