Automated shopping experience using cashier-less systems

A system, comprising:

• a camera mounted onto a shelf unit;
• and
• one or more processors configured to:
• capture images from the camera;
• identify a user from the captured images, wherein the captured images comprises a representation of a body of the user;
• identify joints of the identified user by performing a deep neural network body joint detection on the captured images;
• detecting and tracking actions of the identified user over a first time period by:
• detect an affinity of joints for performing a deep learning identification on a first bounding box to identify a particular user and for associating results from the deep neural network body joint detection with the particular user to re-identify joints of the identified user, wherein the first bounding box comprises identified joints of the identified user,
• transform a 2D position information of the identified joints into a 3D position information of the identified joints by mapping 2D locations of the detected joints in the captured images to a corresponding point cloud, and
• track the 3D position information of the detected joints to determine when the detected joints have entered bins within the shelf unit, wherein the detected joints comprise at least a wrist;
• track an object from the bins over a second time period by associating the object with one or more joints among the identified joints that have entered the bins within the shelf unit, wherein the object is associated with the one or more joints based at least in part on a determination that the 3D position of the identified joints has entered a particular bin within the shelf unit, wherein the object was predetermined to be located in the particular bin using a virtual map of the location of the bins on shelf unit and corresponding objects located in the bins, wherein the virtual map is stored on a memory storage on the shelf unit;
• and
• determine an action inference of the identified user based at least in part on the associated object with the one or more joints and results from the deep learning identification on the bounding box.

An edge device comprising one or more processors;

• a non-transitory memory;
• and
• one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the edge device to be configured to perform:
• capturing images from a camera mounted on a shelf unit;
• identifying a user from the captured images, wherein the captured images comprise a representation of a body of the user;
• identifying joints of the identified user by performing a deep neural network body joint detection on the captured images;
• detecting and tracking actions of the identified user over a first time period by:
• detecting an affinity of joints for performing a deep learning identification on a first bounding box to identify a particular user and for associating results from the deep neural network body joint detection with the particular user to re-identify joints of the identified user, wherein the first bounding box comprises identified joints of the identified user,
• transforming a 2D position information of the identified joints into a 3D position information of the identified joints by mapping 2D locations of the detected joints in the captured images to a corresponding point cloud, and
• tracking the 3D position information of the detected joints to determine when the detected joints have entered bins within the shelf unit, wherein the detected joints comprise at least a wrist;
• tracking an object from the bins over a second time period by associating the object with one or more joints among the identified joints that have entered the bins within the shelf unit, wherein the object is associated with the one or more joints based at least in part on a determination that the 3D position of the identified joints has entered a particular bin within the shelf unit, wherein the object was predetermined to be located in the particular bin using a virtual map of the location of the bins on shelf unit and corresponding objects located in the bins, wherein the virtual map is stored on a memory storage on the shelf unit;
• and
• determining an action inference of the identified user based at least in part on the associated object with the one or more joints and results from the deep learning identification on the bounding box.