A system for multistage object detection and tracking for a cashierless checkout comprising:
- a mobile device comprising:
- a set of on-board sensors installed in the shopping container, wherein the on-board sensors comprise digital cameras viewing an interior of the shopping container from a set of different angles and positions, wherein the on-board sensors detect and provide a user activity data associated with respect to placement or removal of a set of shopping items into or from a shopping container, and wherein a digital camera obtains a set of video frames associated with these items entering and leaving the interior region of the shopping container and communicates these to a mobile device comprising an on-device machine learning (ML) detection;
- an on-device ML detection and tracking engine, wherein the on-device ML detection and tracking engine uses computer vision and specified ML algorithms to implement object detection of unique signatures associated with a shopping item of the set of shopping items and tracking the shopping item by interfacing with the set of on-board image sensors in the interior of the shopping container and determines a state of the item in the interior region of the shopping container;
- wherein the on-device ML detection and tracking engine detects the item in the interior of the shopping cart and determines the state of the item in the interior region of the shopping container based in two phases of analysis comprising:
- in a first phase, the on-device ML detection and tracking engine scans the set of video frames from different digital cameras of the interior region of the shopping container for a semantic change from a previous frame,
- in a second phase, the on-device ML detection and tracking engine implements an object detection process on the interior region, such that a presence or an absence of the item is detected;
- wherein the on-device ML detection and tracking engine comprising:
- detects an anomaly scenario;
- and
- flags the shopping container when the anomaly scenario is detected wherein the anomaly scenario comprises an occlusion scenario;
- wherein the occlusion scenario comprises a blocking of the view of the digital camera with a larger item,
- wherein the on-device ML detection and tracking engine persistently maintains the state of an occluded item in the shopping container, even as the occluded item is completely occluded,
- wherein the on-device ML detection and tracking engine utilizes a bounding box to track the occluded item and provide the occluded item with an identifier that is persistent while wherein the occluded item is detected as remaining in ROI-2,
- wherein an object detection algorithm uses the bounding box to describe a target location of occluded item,
- wherein when the bounding box lands in ROI-2, the tracking identifier is assigned and maintained in a list until the occluded item is later detected by the object detection algorithm and the tracking ID is removed from the shopping cart,
- wherein the shopping container is divided into multiple regions of interest (ROI) comprises a first ROI, a second ROI, and a third ROI,
- wherein the first ROI is defined across a top length and breadth of the shopping container,
- wherein the first ROI comprises a primary area of interest used to ascertain the unique signatures associated with a shopping item,
- wherein the first ROI comprises a primary area of interest used to ascertain a movement direction of shopping items, and
- wherein the second ROI is defined at the base of the shopping container.