Method and system for assisting user with product handling in retail shopping

A product assistance system for assisting user with product handling in a retail shopping in real-time, the product assistance system comprising:

• a processor;
• and
• a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which on execution, cause the processor to:
• extract a list of products from a handheld device associated with a user and historic user details of the user from a database;
• receive a plurality of image frames of one or more products from the list of products and neighbouring products in a rack from the handheld device and Internet of Things (IoT) sensors attached to the rack;
• identify at least one of product type of the one or more products, neighbouring product type of the neighbouring products and orientations of the one or more products and the neighbouring products from the plurality of image frames and rack information corresponding to the rack stored in the database;
• compare the identified orientations of the one or more products and the neighbouring products with historic orientations of products and neighbouring products associated with the rack from the database;
• identify, using an artificial neural network, at least one product from the one or more products and the neighbouring products that is improperly oriented based on comparison;
• determine, using the artificial neural network, product safety handling instructions of the identified at least one product based on historic product details stored in the database and the identified orientations of the at least one product;
• receive at least one of historic images of products, historic images of neighbouring products and the historic orientations of products and neighbouring products associated with the rack from the IoT sensors;
• extract the historic product details of the products from the historic images of products and the historic product details of the neighbouring products from the historic images of neighbouring products;
• receive the historic user details from the handheld device associated with the user;
• store at least one of the historic images of products, the historic images of neighbouring products, the historic orientations of products and neighbouring products, the historic product details of the products and the neighbouring products and the historic user details in the database;
• and
• assist the user in real-time with handling of the identified at least one product based on the product safety handling instructions.

A non-transitory computer readable medium including instructions stored thereon that when processed by at least one processor cause a product assistance system to perform operations comprising:

• extracting a list of products from a handheld device associated with a user and historic user details of the user from a database;
• receiving a plurality of image frames of one or more products from the list of products and neighbouring products in a rack from the handheld device and Internet of Things (IoT) sensors attached to the rack;
• identifying at least one of product type of the one or more products, neighbouring product type of the neighbouring products and orientations of the one or more products and the neighbouring products from the plurality of image frames and rack information corresponding to the rack stored in the database;
• comparing the identified orientations of the one or more products and the neighbouring products with historic orientations of products and neighbouring products associated with the rack from the database;
• identifying, using an artificial neural network, at least one product from the one or more products and the neighbouring products that is improperly oriented based on comparison;
• determining, using the artificial neural network, product safety handling instructions of the identified at least one product based on historic product details stored in the database and the identified orientations of the at least one product;
• receiving at least one of historic images of products, historic images of neighbouring products and the historic orientations of products and neighbouring products associated with the rack from the IoT sensors;
• extracting the historic product details of the products from the historic images of products and the historic product details of the neighbouring products from the historic images of neighbouring products;
• receiving the historic user details from the handheld device associated with the user;
• and
• storing at least one of the historic images of products, the historic images of neighbouring products, the historic orientations of products and neighbouring products, the historic product details of the products and the neighbouring products and the historic user details in the database;
• and
• assisting the user in real-time with handling of the identified at least one product based on the product safety handling instructions.