Enabling the visually impaired with AR using force feedback

A method of providing feedback to a user to guide the user to an object, comprising:

• receiving at least one of a user voice input or a contextual trigger to identify a target object in a scene;
• identifying the target object using a camera mounted on a frame configured to be worn on a head of the user and a processor adapted to identify the target object and a hand of the user in a field of view of the camera;
• tracking, using at least the processor, relative positions of the hand relative to the target object;
• determining whether precise feedback is desired;
• when precise feedback is desired, determining a distance and orientation of hand segments of the hand relative to the target object;
• generating, using at least the processor, at least one control signal for guiding the hand of the user to the target object and, when precise feedback is desired, for guiding at least one fingertip of the hand of the user to the target object;
• communicating the at least one control signal to a user feedback device configured to be worn on or adjacent the hand of the user and adapted to receive the at least one control signal;
• and
• providing at least one of sound or haptic feedback, using the user feedback device, in response to the at least one control signal to guide at least one of the hand or the at least one fingertip to touch the target object.

A non-transitory computer readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:

• receiving at least one of a user voice input or a contextual trigger to identify a target object in a scene;
• capturing, using at least one camera mounted on a frame configured to be worn on a head of the user, a scene including a target object;
• identifying the target object and a hand of the user in a field of view of the at least one camera;
• tracking relative positions of the hand relative to the target object;
• determining whether precise feedback is desired;
• when precise feedback is desired, determining a distance and orientation of hand segments of the hand relative to the target object;
• generating at least one control signal for guiding the hand of the user to the target object and, when precise feedback is desired, for guiding at least one fingertip of the hand of the user to the target object;
• and
• communicating the at least one control signal to a user feedback device configured to be worn on or adjacent the hand of the user and adapted to receive the at least one control signal, wherein the user feedback device provides at least one of sound or haptic feedback in response to the at least one control signal to guide at least one of the hand or the at least one fingertip to touch the target object.