Wearable assistive device
OwnerIsmat Saira Gillaniinventor
Life of this patent
Estimating the term…
Abstract
This disclosure describes a device and a method for detecting video images of hand gestures and the environment around a user. The device and method process the incoming video images and classify the detected images as belonging to an action. The device and method further may predict the likely next action to be taken by the user. The device and method may communicate with the user to describe the current action being undertaken as well as the predicted actions likely to take place next. In an embodiment, the device may be worn around a user's neck with a forward-facing camera.
Claims20
A device for use by a user suffering from a memory impairment disease, the device used by the user to assist the user in recognizing surroundings and actions of the user in order to remind the user of the task being performed by the suer, the device comprising:
- an image sensor;
- a speaker;
- a power source;
- and
- a computing device comprising memory and a processor, with computer instructions stored in the memory, which, when executed by the processor perform the steps of:
- receiving a series of images detected by the image sensor, wherein the images are from an area in front of the user;
- processing the series of images;
- applying a first algorithm to the processed series of images to classify the processed series of images as belonging to a selected action of a plurality of actions, wherein the first algorithm comprises at least the steps of;
- pre-processing the series of images, wherein the pre-processing includes:
- applying a transformation to the series of images;
- encoding, via an encoder, the pre-processed series of images to create an embedding;
- applying a decoder to the embedding to produce a plurality of labels on the series of images and a confidence score associated with each label of the plurality of labels, wherein the labels correspond with a prediction for human action of the user;
- and
- applying an action recognition model to select from the plurality of labels a most likely label, wherein the most likely label is based on the associated confidence score;
- communicating by the speaker to the user the selected action;
- applying a second algorithm to the processed series of images to predict the most likely following action of the plurality of actions relative to the selected action;
- and
- communicating by the speaker to the user the predicted following action.
A method for directing a user to complete a current action comprising:
- detecting a series of images from a camera worn by the user, wherein the camera takes images from an area in front of the user;
- processing the series of images, the processing comprising:
- pre-processing the series of images by applying a transformation to the series of images;
- encoding, via an encoder, the pre-processed series of images to create an embedding;
- and
- applying a decoder to the embedding to produce a plurality of labels on the series of images and a confidence score associated with each label of the plurality of labels, wherein the labels correspond with a prediction for human action of the user;
- classifying the processed series of images as belonging to a selected action of a plurality of actions by applying an action recognition model to select from the plurality of labels a most likely label, wherein the most likely label is based on the associated confidence score;
- communicating to the user the selected action;
- applying a second algorithm to the processed series of images to predict the most likely following action of the plurality of actions relative to the selected action;
- and
- communicating to the user the predicted following action.
Dependencies are inferred from the claim text.