Advanced augmented assistive device for dementia, Alzheimer's disease, and visual impairment patients

A method for assisting a subject with an ailment, the method comprising:

• a) receiving input data about the subject and a surrounding area of the subject, the input data comprising real-time visual data received from at least one camera, data received from at least one gyroscope, physiological data received from at least one biometric sensor, and data received from at least one microphone;
• b) utilizing a machine learning (ML) algorithm to process the input data and generate a context-aware, predictive action plan to assist the subject, the ML algorithm being trained on sensor fusion data and the action plan dynamically adapting based on changes in the input data;
• and
• c) providing the action plan to the subject via:
• i) real-time AR visual cues provided to the subject by at least one augmented reality (AR) device;
• ii) audio feedback provided to the subject by at least one audio output device;
• and
• iii) haptic feedback provided to the subject by at least one haptic actuator, the haptic feedback comprising a vibration to alert the subject of an immediate risk in the surrounding area of the subject,
• the at least one camera being configured to capture the real-time visual data of the surrounding area of the subject;
• the at least one gyroscope being configured to track orientation and movement of the subject;
• the at least one AR device being configured to overlay customized real-time visual cues;
• the at least one biometric sensor being configured to continuously monitor the physiological data of the subject;
• the at least one microphone being configured to capture audio;
• the at least one haptic actuator being configured to provide adaptive tactile feedback to the subject;
• the at least one audio output device being configured to deliver real-time audio to the subject,
• the at least one audio output device comprising at least one of headphones, a headset, and a set of earbuds, and
• the utilizing of the ML algorithm in step b) comprising:
• detecting any deviation from an expected behavior pattern of the subject using a first long short-term memory (LSTM) recurrent neural network (RNN) trained on previous behavior data of the subject;
• using a first convolutional neural network (CNN) on the real-time visual data received from the at least one camera to assess the surrounding area of the subject;
• using a second RNN to manage time series data of the input data to assess the surrounding area of the subject;
• using a second CNN trained on a dataset of labeled facial images to perform facial recognition on any people within the surrounding area of the subject;
• using a time series analysis technique to detect patterns and trends in the data received from the at least one biometric sensor to keep track of the ailment of the subject;
• and
• adaptively adjusting the action plan using reinforcement learning (RL) based on preferences of the subject.

A system for assisting a subject with an ailment, the system comprising:

• at least one camera configured to capture real-time visual data of a surrounding area of the subject;
• at least one gyroscope configured to track orientation and movement of the subject;
• at least one augmented reality (AR) device configured to overlay customized real-time visual cues;
• at least one biometric sensor configured to continuously monitor physiological data of the subject;
• at least one microphone configured to capture audio;
• an accelerometer;
• a heart rate sensor;
• at least one audio output device configured to deliver real-time audio to the subject, the at least one audio output device comprising at least one of headphones, a headset, and a set of earbuds;
• at least one haptic actuator configured to provide adaptive tactile feedback to the subject;
• at least one global positioning system (GPS) sensor;
• at least one processor in operable communication with the at least one camera, the at least one gyroscope, the at least one AR device, the at least one biometric sensor, the at least one microphone, the accelerometer, the heart rate sensor, the at least one audio output device, the at least one haptic actuator, and the at least one GPS sensor;
• and
• a machine-readable medium in operable communication with the processor and having instructions stored thereon that, when executed by the processor, perform the following steps:
• a) receiving input data about the subject and the surrounding area of the subject, the input data comprising the real-time visual data received from the at least one camera, data received from the at least one gyroscope, the physiological data received from the at least one biometric sensor, data received from the at least one microphone, data received from the accelerometer, data received from the heart rate sensor, and data received from the at least one GPS sensor;
• b) utilizing a machine learning (ML) algorithm to process the input data and generate a context-aware, predictive action plan to assist the subject, the ML algorithm being trained on sensor fusion data, and the action plan dynamically adapts based on changes in the input data;
• c) providing the action plan to the subject via:
• i) real-time AR visual cues provided to the subject by the at least one AR device;
• ii) audio feedback provided to the by the at least one audio output device;
• and
• iii) haptic feedback provided to the subject by the at least one haptic actuator, the haptic feedback comprising a vibration to alert the subject of an immediate risk in the surrounding area of the subject,
• the action plan being an action plan for at least one of wayfinding, social interaction, situational awareness, health monitoring, emotional state management, and medication and appointment management, and
• the utilizing of the ML algorithm in step b) comprising:
• detecting any deviation from an expected behavior pattern of the subject using a first long short-term memory (LSTM) recurrent neural network (RNN) trained on previous behavior data of the subject;
• using a first convolutional neural network (CNN) on the real-time visual data received from the at least one camera to assess the surrounding area of the subject;
• using a second RNN to manage time series data of the input data to assess the surrounding area of the subject;
• using a second CNN trained on a dataset of labeled facial images to perform facial recognition on any people within the surrounding area of the subject;
• using a time series analysis technique to detect patterns and trends in the data received from the at least one biometric sensor to keep track of the ailment of the subject;
• and
• adaptively adjusting the action plan using reinforcement learning (RL) based on preferences of the subject.