Blindness assist glasses
AssigneeSnap Inc.
InventorStephen Pomes
Life of this patent
Estimating the term…
Abstract
An eyewear device with camera-based compensation that improves the user experience for user's having partial blindness or complete blindness. The camera-based compensation determines objects, converts determined objects to text, and then converts the text to audio that is indicative of the objects and that is perceptible to the eyewear user. The camera-based compensation may use a region-based convolutional neural network (RCNN) to generate a feature map including text that is indicative of objects in images captured by a camera. Relevant text of the feature map is then processed through a text to speech algorithm featuring a natural language processor to generate audio indicative of the objects in the processed images.
Claims15
Eyewear, comprising:
- a frame having a first side and a second side;
- a first temple adjacent the first side of the frame;
- an optical element supported by the frame;
- a camera coupled to the frame and configured to generate an image having one or more features;
- and
- an electronic processor configured to:
- process the image and identify at least one relevant feature from the one or more features in the image;
- generate audio that is indicative of the at least one relevant feature in the image;
- use a convolutional neural network (CNN) to generate a convolutional feature map that is indicative of the image;
- and
- use a text to speech algorithm configured to convert the convolutional feature map to the audio, wherein the text to speech algorithm is configured to use a natural language processor and a digital signal processor to generate the audio that is indicative of the text in the convolutional feature map.
A method for use with eyewear, the eyewear having a frame, a temple adjacent the frame, an optical element supported by the frame, a camera coupled to the frame and configured to generate an image having one or more features, and an electronic processor, the processor configured to:
- process the image and identify at least one relevant feature from the one or more features in the image;
- generate audio that is indicative of the at least one relevant feature;
- use a convolutional neural network (CNN) to generate a convolutional feature map that is indicative of the image;
- and
- use a text to speech algorithm configured to convert the convolutional feature map to the audio, wherein the text to speech algorithm uses a natural language processor and a digital signal processor to generate the audio that is indicative of the text in the convolutional feature map.
A non-transitory computer-readable medium storing program code which, when executed, is operative to cause a computing device to perform the steps of:
- processing, by a processor of eyewear having a frame, a temple adjacent the frame, an optical element supported by the frame, and a camera coupled to the frame and configured to generate an image, the image to identify at least one relevant feature from one or more features in the image;
- generating, by the processor, audio that is indicative of the at least one relevant feature;
- using, by the processor, a convolutional neural network (CNN) to generate a convolutional feature map that is indicative of the image;
- and
- using, by the processor, a text to speech algorithm configured to convert the convolutional feature map to the audio, wherein the text to speech algorithm uses a natural language processor and a digital signal processor to generate the audio that is indicative of the text in the convolutional feature map.
Dependencies are inferred from the claim text.