Augmented reality virtual order assistant

A computer program product for generating an interface, the computer program product comprising:

• one or more computer readable storage media having computer readable program code collectively stored on the one or more computer readable storage media, the computer readable program code being executed by a central processing unit (CPU) of a computer system to cause the computer system to perform a method comprising:
• using a natural language processing (NLP) module, a natural language understanding (NLU) module, and one or more microphones operatively coupled to the NLP and NLU modules, and using a database that associates ingredient names with pronunciations, keywords, and recipes, the computer system deriving ingredients of a first food item from first audio data captured by the one or more microphones and received from a first spoken order of a first customer who is ordering the first food item;
• the computer system determining that the first audio data includes a pronunciation of a first ingredient that is not included in the derived ingredients of the first food item and is not associated with the pronunciations associated with ingredient names by the database;
• using a machine learning component and a dialect of the first customer, the computer system learning that the pronunciation of the first ingredient is associated with the first ingredient, and the computer system associating, in the database, the pronunciation of the first ingredient with a name of the first ingredient;
• subsequent to the learning using the machine learning component, the computer system receiving second audio data from a second spoken order of a second customer who is ordering a second food item, the second audio data being captured by the one or more microphones;
• using the NLP module and the NLU module, the computer system determining that the second audio data includes the pronunciation of the first ingredient;
• using the NLP module, the NLU module, and the database, the computer system deriving ingredients of the second food item from the second audio data, wherein the deriving the ingredients includes deriving the first ingredient from the second audio data based on the database associating the pronunciation of the first ingredient with the name of the first ingredient;
• using a camera, the computer system identifying locations of the ingredients of the second food item in a preparation area, the preparation area including (i) the ingredients of the second food item and (ii) other ingredients, the camera being operatively coupled to an augmented reality (AR) headset worn by a second employee who is assigned to prepare the second food item;
• using image data captured by the camera, the computer system determining a current location of the second employee;
• the computer system calculating distances of the locations of the ingredients of the second food item to the current location of the second employee;
• based on the calculated distances, the computer system determining a sequence of selecting the ingredients of the second food item, so that the second employee selecting the ingredients of the second food item in the sequence maximizes a speed at which the second food item is prepared by the second employee;
• and
• the computer system generating and displaying an overlay to an interface for viewing in the AR headset worn by the second employee, the overlay including visual indicators overlaying an image of the preparation area, the visual indicators marking the ingredients of the second food item and the sequence of selecting the ingredients of the second food item, and distinguishing the ingredients of the second food item from the other ingredients in the preparation area.

A computer system comprising:

• a central processing unit (CPU);
• a memory coupled to the CPU;
• and
• one or more computer readable storage media coupled to the CPU, the one or more computer readable storage media collectively containing instructions that are executed by the CPU via the memory to implement a method of generating an interface, the method comprising:
• using a natural language processing (NLP) module, a natural language understanding (NLU) module, and one or more microphones operatively coupled to the NLP and NLU modules, and using a database that associates ingredient names with pronunciations, keywords, and recipes, the computer system deriving ingredients of a first food item from first audio data captured by the one or more microphones and received from a first spoken order of a first customer who is ordering the first food item;
• the computer system determining that the first audio data includes a pronunciation of a first ingredient that is not included in the derived ingredients of the first food item and is not associated with the pronunciations associated with ingredient names by the database;
• using a machine learning component and a dialect of the first customer, the computer system learning that the pronunciation of the first ingredient is associated with the first ingredient, and the computer system associating, in the database, the pronunciation of the first ingredient with a name of the first ingredient;
• subsequent to the learning using the machine learning component, the computer system receiving second audio data from a second spoken order of a second customer who is ordering a second food item, the second audio data being captured by the one or more microphones;
• using the NLP module and the NLU module, the computer system determining that the second audio data includes the pronunciation of the first ingredient;
• using the NLP module, the NLU module, and the database, the computer system deriving ingredients of the second food item from the second audio data, wherein the deriving the ingredients includes deriving the first ingredient from the second audio data based on the database associating the pronunciation of the first ingredient with the name of the first ingredient;
• using a camera, the computer system identifying locations of the ingredients of the second food item in a preparation area, the preparation area including (i) the ingredients of the second food item and (ii) other ingredients, the camera being operatively coupled to an augmented reality (AR) headset worn by a second employee who is assigned to prepare the second food item;
• using image data captured by the camera, the computer system determining a current location of the second employee;
• the computer system calculating distances of the locations of the ingredients of the second food item to the current location of the second employee;
• based on the calculated distances, the computer system determining a sequence of selecting the ingredients of the second food item, so that the second employee selecting the ingredients of the second food item in the sequence maximizes a speed at which the second food item is prepared by the second employee;
• and
• the computer system generating and displaying an overlay to an interface for viewing in the AR headset worn by the second employee, the overlay including visual indicators overlaying an image of the preparation area, the visual indicators marking the ingredients of the second food item and the sequence of selecting the ingredients of the second food item, and distinguishing the ingredients of the second food item from the other ingredients in the preparation area.