# US 11,112,261 B2 · Navigation methods and apparatus for the visually impaired > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,112,261 B2 - Document ID: US11112261B2 - Status: Granted - Kind: B2 - Application number: 15/697,966 - Filed: 2017-09-07 - Published: 2018-03-08 - Granted: 2021-09-07 - Earliest claimed: 2014-05-08 - Assignees: Brown University - Applicants: Brown University - Inventors: Benjamin Kimia - Web page: https://patentgrep.com/patent/US11112261B2 ## CPC - G01C 21/3652: {Guidance using non-audiovisual output, e.g. tactile, haptic or electric stimuli} - A61H 3/061: {with electronic detecting or guiding means} - G01C 21/206: {specially adapted for indoor navigation} - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 21/3629: {Guidance using speech or audio output, e.g. text-to-speech (text to speech systems per se G10L13/00)} - G06F 3/005: {Input arrangements through a video camera} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06V 20/20: in augmented reality scenes - A61H 2003/063: {with tactile perception} - A61H 2201/1604: Head - A61H 2201/1619: Thorax - A61H 2201/1635: Hand or arm, e.g. handle - A61H 2201/165: Wearable interfaces - A61H 2201/5048: Audio interfaces, e.g. voice or music controlled - A61H 2201/5064: Position sensors - A61H 2201/5084: Acceleration sensors - A61H 2201/5092: Optical sensor - G06T 2207/10016: Video; Image sequence - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30244: Camera pose - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - A61H: PHYSICAL THERAPY APPARATUS, e.g. DEVICES FOR LOCATING OR STIMULATING REFLEX POINTS IN THE BODY; ARTIFICIAL RESPIRATION; MASSAGE; BATHING DEVICES FOR SPECIAL THERAPEUTIC OR HYGIENIC PURPOSES OR SPECIFIC PARTS OF THE BODY (electrotherapy, magnetotherapy, radiation therapy or ultrasound therapy A61N) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract According to some aspects, a wearable device for estimating a location of the device within a space is provided, the device comprising a plurality of cameras mounted to a structure, at least a portion of the structure being adapted to facilitate a user wearing the device, the plurality of cameras having substantially fixed positions and orientations on the structure relative to each other, and at least one processor configured to receive image data from the plurality of cameras, perform feature detection on the image data to obtain a first plurality of features from the image data, and determine an estimate of the location of the device in the space based, at least in part, on a location associated with a second plurality of features obtained from image data previously captured of the space that matches the first plurality of features. ## Claims ### Claim 1 A system for guiding a visually impaired user, the system comprising: an image acquisition device comprising multiple cameras mounted on a wearable head unit, the image acquisition device configured to capture image data; a plurality of path data stored in a database, the path data identifying traversable paths to and from selectable locations, wherein the path data is derived from previously captured images of the locations and the identified traversable paths include a map containing graph nodes representing defined locations, captured images, and annotations; an inertial odometry sensor for identifying motion based on inertial measurements and providing inertial data pertaining to a location and a trajectory of a user being tracked; a visual odometry processor for performing feature detection on the image data to obtain a plurality of features from the image data, wherein feature detection includes identifying curves, edges, and evaluating metadata of the image data; an analysis processor for comparing the plurality of features from the image data and the plurality of stored path data to determine a user's location; a map processor for automatically selecting the traversable path from the stored path data corresponding to the user's location and a user-selected destination location; a navigation processor for guiding the user along the traversable path to the user-selected destination; a haptic device configured to receive input from the navigation unit and render haptic gestures as non-audible guidance to the user on the traversable path to the destination; and an audio interface configured to receive spoken input from the user and produce audio feedback to the user. ### Claim 2 (depends on 1) The system of claim 1 wherein the image acquisition device captures image data of objects in multiple directions in synchronization with one another from the multiple cameras. ### Claim 3 (depends on 2) The system of claim 2 wherein the image analysis processor generates an N-best list of closest matches between previously acquired image data and newly acquired image data, along with the corresponding match scores, a closest matching image data used to estimate the location and/or orientation of the user. ### Claim 4 (depends on 2) The system of claim 2 wherein the visual odometry processor receives image data of the user's environment from the image data to determine incremental motion of the user. ### Claim 5 (depends on 2) The system of claim 2 wherein the inertial odometry sensor comprises any one or combination of accelerometers, magnetometers and/or gyroscopes that detect motion. ### Claim 6 (depends on 5) The system of claim 5 wherein the map comprises an undirected graph containing nodes that represent known locations, wherein each node includes any one or combination of location, references to neighboring or adjacent nodes, annotations, image data associated with the node, a timestamp indicating when the node was created, and/or a list of connecting links. ### Claim 7 (depends on 6) The system of claim 6 wherein the map further comprises edges indicating a walkable path between two nodes. ### Claim 8 (depends on 7) The system of claim 7 wherein the navigation processor publishes a sequence of commands indicating which direction the user needs to move to reach a next node in the walkable path from an origin node to a destination node, where the commands are dynamically updated based on the user's progress as the user follows the directions. ### Claim 9 (depends on 8) The system of claim 8 wherein the haptic device comprises multiple haptic feedback components. ### Claim 10 (depends on 1) A method comprising: in a wearable navigation system comprising the system of claim 1; determining a navigation instruction to be conveyed to a user, the navigation instruction comprising a direction to a target location identified by a vector in a direction of the target location from the user's current location and a difference in the direction between the vector and the user's current heading; identifying a haptic gesture adapted to convey the navigation instruction; and rendering the haptic gesture by activating one or more haptic feedback components of a haptic device according to the haptic gesture. ### Claim 11 (depends on 10) The method of claim 10 wherein the target location is one of a series of incremental target locations along a sequence of nodes forming a path to a destination. ### Claim 12 (depends on 10) The method of claim 10 wherein the target location is a destination location. ### Claim 13 (depends on 10) The method of claim 10 wherein the navigation system is a wearable device. ### Claim 14 (depends on 10) The method of claim 10 wherein the user's current location is determined by the navigation system. ### Claim 15 (depends on 14) The method of claim 14 wherein the navigation system receives image data from one or more image acquisition devices and matches features detected in the image data with features detected in previously acquired image data to facilitate tracking movements of the user in an environment.