Method, apparatus and robot for out-of-stock detection of shelf commodities

An apparatus for out-of-stock detection of shelf commodities, comprising:

• an acquisition unit configured to acquire a plurality of shelf RGB images and a plurality of shelf depth images;
• an out-of-stock area detection unit configured to execute out-of-stock area detection steps as follows for each shelf RGB image and each shelf depth image:
• inputting each shelf RGB image into an object detection model pre-trained by using a deep learning method, detecting all commodities to obtain commodity detection boxes;
• detecting all electronic shelf labels to obtain shelf label detection boxes;
• and detecting all shelf layer lattices to obtain shelf layer lattice detection boxes;
• obtaining a first vacant area corresponding to each shelf RGB image based on position information of the commodity detection boxes and the shelf layer lattice detection boxes both of which correspond to each shelf RGB image;
• determining a first average depth value of pixel in each shelf label detection box based on the shelf label detection boxes and the shelf depth image both of which correspond to each shelf RGB image;
• determining a second average depth value of pixel in each first vacant area based on the first vacant area and the shelf depth image both of which correspond to each shelf RGB image;
• obtaining a fourth vacant area corresponding to each shelf RGB image based on the pixel whose depth values are in a set range;
• wherein the set range is determined based on the first average depth value and the second average depth value;
• merging the first vacant area and the fourth vacant area both of which correspond to each shelf RGB image to obtain all vacant areas in the shelf, wherein each vacant area has position information and depth information;
• an image stitching unit configured to splice the shelf RGB images into a complete shelf image, perform a merging process and a de-duplicating process on the commodity detection boxes, the shelf label detection boxes and the layer lattice detection boxes and all vacant areas, and map them to the complete shelf image;
• a vacancy division unit for configured to delimit a placement area of each commodity for the complete shelf image to generate a shed grid area corresponding to each commodity;
• and divide the all vacant areas in the shelf based on the shed grid area corresponding to each commodity to obtain a vacant area corresponding to each commodity, wherein the vacant area and the shed grid area both of which correspond to each commodity both have position information and depth information;
• and
• an out-of-stock detection unit configured to determine an out-of-stock body volume in the vacant area corresponding to each commodity as an out-of-stock detection result based on the position information and the depth information of the vacant area corresponding to each commodity.