Smart glasses with multi-modal generative artifical intelligence assistant

A wearable system comprising:

• smart glasses configured to be worn by a user, the smart glasses comprising an outward side configured to be opposite an inward side facing the user;
• a light source comprising one or more light emitting diodes attached to the outward side of the smart glasses and configured to emit a light beam;
• one or more cameras that are CMOS back-illuminated sensors also attached to the outward side of the smart glasses and configured to capture a camera image or a camera video including an object;
• the smart glasses further comprising a processor, speakers, and one or more microphones configured to receive sound inputs including voice control or interactions;
• the smart glasses including the processor configured to be coupled to a smart phone or a tablet for further processing capabilities;
• the processor being configured to be coupled to a non-transitory computer readable medium;
• wherein smart glasses data comprises the camera image, the camera video, or the sound inputs, and the smart glasses data is configured to serve as at least a part of an input to an artificial intelligence assistant configured to communicate with or operate on the processor, the artificial intelligence assistant comprising a multi-modal generative artificial intelligence model comprising a transformer including self-attention and positional encoding layers and configured to perform computer vision processing and natural language processing;
• and
• wherein the multi-modal generative artificial intelligence model is configured to recognize at least some of the object captured in the camera image or the camera video;
• and
• wherein the multi-modal generative artificial intelligence model is configured to include memory that records at least a part of the smart glasses data.

A wearable system comprising:

• smart glasses configured to be worn by a user, the smart glasses comprising an outward side configured to be opposite an inward side facing the user;
• a light source comprising one or more light emitting diodes attached to the outward side of the smart glasses and configured to emit a light beam;
• one or more cameras that are CMOS back-illuminated sensors also attached to the outward side of the smart glasses and configured to capture a camera image or a camera video including an object, and wherein the one or more cameras are also configured to perform hand tracking for gesture analysis;
• the smart glasses further comprising a processor, speakers, and one or more microphones configured to receive sound inputs including voice control or interactions;
• the smart glasses including the processor configured to be coupled to a smart phone or a tablet for further processing capabilities;
• the processor being configured to be coupled to a non-transitory computer readable medium;
• wherein smart glasses data comprises the camera image, the camera video, the sound inputs, or the gesture analysis, and the smart glasses data is configured to serve as at least a part of an input to an artificial intelligence assistant configured to communicate with or operate on the processor, the artificial intelligence assistant comprising a multi-modal generative artificial intelligence model comprising a transformer including self-attention and positional encoding layers and configured to perform computer vision processing and natural language processing;
• and
• wherein the multi-modal generative artificial intelligence model is configured to recognize at least some of the object captured in the camera image or the camera video;
• and
• wherein the multi-modal generative artificial intelligence model is configured to include memory that records at least a part of the smart glasses data.