System, device, and method of augmented reality based mapping of a venue and navigation within a venue

A non-transitory storage medium having stored thereon instructions that, when executed by a machine, cause the machine to perform a method comprising:

• performing a crowd-sourced mapping and localization process for a dynamically-changing indoor environment, that (i) maps a retail store, and (ii) maps particular products that are actually currently sold on shelves within said retail store, and (iii) localizes users within said retail store,
• based on computer-vision analysis of a plurality of images captured by a plurality of hand-held or wearable end-user devices of customers within said retail store, wherein each hand-held or wearable end-user device comprises a camera and a processor and a spatial orientation sensor, and is moved and rotated by a respective end-user with six degrees of freedom that are sensed by said spatial orientation sensor,
• by performing non-barcode computerized image recognition of external packages of retail products that are depicted in said plurality of images captured by the plurality of end-user devices of customers, and identifying said retail products based on visual analysis their external packages that are facing aisles of said retail store;
• generating an updated three-dimensional semantic representation of a store map reflecting actual and current real-time location of particular products within said retail store, with six degrees of freedom, based on identification of said retail products via visual analysis of their external packages;
• determining a precise spatial location of a particular end-user within said retail store, by performing non-barcode computerized image recognition of external packages of retail products that are depicted in a plurality of images captured by an end-user device of said particular end-user;
• creating an up-to-date three-dimensional semantic representation of said retail store and its current inventory, reflecting a current supply of products that are facing aisles of the retail store and reflecting their current and precise in-store spatial locations and arrangement, based on crowd-sourced images received from multiple end-users having multiple respective end-user devices;
• wherein creating said up-to-date three-dimensional semantic representation comprises creating an inventory map representation that represents therein not only aisles and corridors in which end-users walk, but also actual real-time inventory and spatial placement of items and products on shelves of said retail store;
• wherein the method comprises:
• firstly, obtaining images and spatial data and spatial orientation data captured by end-user devices within the retail store and recognizing their content via said computerized image process;
• secondly, inferring a precise indoor location and a precise spatial arrangement of items based on visual content analysis of said images;
• and thirdly, reconstructing and updating three-dimensional product inventory maps of said retail store based on real-time data that was learned from captured images in a dynamically-changing indoor environment of said retail store;
• wherein said localization process comprises:
• (I) capturing an image of a retail shelf of said retail store by a particular end-user device that is equipped with a camera,
• (II) performing non-barcode-based computerized image recognition of one or more retail products that are depicted in said image, based on recognition of visual elements of visible external packages,
• (III) determining which products are depicted in said image,
• (IV) querying a dynamically-changing inventory map of said retail store,
• (V) obtaining from said dynamically-changing inventory map a current location of one or more image-depicted products within said retail store, and determining that said particular end-user is located at said current location.

A system comprising:

• one or more processors, that are configured to execute code;
• wherein the one or more processors are operably associated with one or more memory units that are configured to store code;
• wherein the one or more processors are configured to perform a process comprising:
• performing a crowd-sourced mapping and localization process for a dynamically-changing indoor environment, that (i) maps a retail store, and (ii) maps particular products that are actually currently sold on shelves within said retail store, and (iii) localizes users within said retail store,
• based on computer-vision analysis of a plurality of images captured by a plurality of hand-held or wearable end-user devices of customers within said retail store, wherein each hand-held or wearable end-user device comprises a camera and a processor and a spatial orientation sensor, and is moved and rotated by a respective end-user with six degrees of freedom that are sensed by said spatial orientation sensor,
• by performing non-barcode computerized image recognition of external packages of retail products that are depicted in said plurality of images captured by the plurality of end-user devices of customers, and identifying said retail products based on visual analysis their external packages that are facing aisles of said retail store;
• generating an updated three-dimensional semantic representation of a store map reflecting actual and current real-time location of particular products within said retail store, with six degrees of freedom, based on identification of said retail products via visual analysis of their external packages;
• determining a precise spatial location of a particular end-user within said retail store, by performing non-barcode computerized image recognition of external packages of retail products that are depicted in a plurality of images captured by an end-user device of said particular end-user;
• creating an up-to-date three-dimensional semantic representation of said retail store and its current inventory, reflecting a current supply of products that are facing aisles of the retail store and reflecting their current and precise in-store spatial locations and arrangement, based on crowd-sourced images received from multiple end-users having multiple respective end-user devices;
• wherein creating said up-to-date three-dimensional semantic representation comprises creating an inventory map representation that represents therein not only aisles and corridors in which end-users walk, but also actual real-time inventory and spatial placement of items and products on shelves of said retail store;
• wherein the process comprises:
• firstly, obtaining images and spatial data and spatial orientation data captured by end-user devices within the retail store and recognizing their content via said computerized image process;
• secondly, inferring a precise indoor location and a precise spatial arrangement of items based on visual content analysis of said images;
• and thirdly, reconstructing and updating three-dimensional product inventory maps of said retail store based on real-time data that was learned from captured images in a dynamically-changing indoor environment of said retail store;
• wherein said localization process comprises:
• (I) capturing an image of a retail shelf of said retail store by a particular end-user device that is equipped with a camera,
• (II) performing non-barcode-based computerized image recognition of one or more retail products that are depicted in said image, based on recognition of visual elements of visible external packages,
• (III) determining which products are depicted in said image,
• (IV) querying a dynamically-changing inventory map of said retail store,
• (V) obtaining from said dynamically-changing inventory map a current location of one or more image-depicted products within said retail store, and determining that said particular end-user is located at said current location.