# US 10,503,310 B2 · Assistive device for non-visually discerning a three-dimensional (3D) real-world area surrounding a user > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,310 B2 - Document ID: US10503310B2 - Status: Granted - Kind: B2 - Application number: 15/709,793 - Filed: 2017-09-20 - Published: 2019-03-21 - Granted: 2019-12-10 - Earliest claimed: 2017-09-20 - Applicants: Alex Hamid Mani; Ahmadreza Rofougaran - Inventors: Alex Hamid Mani; Ahmadreza Rofougaran - Web page: https://patentgrep.com/patent/US10503310B2 ## CPC - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/0416: {Control or interface arrangements specially adapted for digitisers} - G08B 6/00: Tactile signalling systems, e.g. tactile personal calling systems - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS ## Abstract An assistive device and method for non-visually discerning a three-dimensional (3D) real-world area surrounding a user, comprises a haptic feedback interface that includes a plurality of haptic elements. The assistive device receives sensor data of the 3D real-world area within a first proximity range of the assistive device from a plurality of different types of sensors that are communicatively coupled to the assistive device. The assistive device establishes a mapping of a plurality of objects within the first proximity range to the plurality of haptic elements in a defined region of the haptic feedback interface, based on the received sensor data. A haptic feedback generator generates a touch-discernible feedback on the haptic feedback interface based on the established mapping. The touch-discernible feedback comprises a plurality of differential touch-discernible cues to enable the user to non-visually discern the 3D real-world area surrounding the user. ## Claims ### Claim 1 An assistive device, comprising: a haptic feedback interface comprising a plurality of haptic elements; a first circuitry configured to receive sensor data of a three-dimensional (3D) real-world area from a plurality of different types of sensors that are communicatively coupled to the assistive device, wherein the sensor data is associated with a first proximity range of the assistive device; a second circuitry configured to: determine a scaling factor based on the first proximity range and a surface area of a defined region of the haptic feedback interface; and establish a mapping of a plurality of objects within the first proximity range to at least one haptic element of the plurality of haptic elements in the defined region of the haptic feedback interface; adjust sizes of the plurality of objects to map the plurality of objects to the at least one haptic element in the defined region of the haptic feedback interface, wherein the sizes of the plurality of objects are adjusted based on the scaling factor, and wherein the mapping is established based on the received sensor data; and a haptic feedback generator configured to generate a touch-discernible feedback on the haptic feedback interface based on the established mapping, wherein the touch-discernible feedback comprises a plurality of differential touch-discernible cues to discern a 3D arrangement of each object of the plurality of objects with respect to a position of a user of the assistive device. ### Claim 2 (depends on 1) The assistive device according to claim 1, wherein the touch-discernible feedback corresponds to at least one of a differential pressure-based, a differential temperature-based, a differential electric pulse-based, a differential raised shape pattern-based touch-discernible feedback, or a combination of different touch-discernible feedbacks. ### Claim 3 (depends on 1) The assistive device according to claim 1, wherein the second circuitry is further configured to identify an object-type of each object of the plurality of objects within the first proximity range of the assistive device, wherein the object-type of each object is identified based on the received sensor data. ### Claim 4 (depends on 3) The assistive device according to claim 3, wherein the haptic feedback generator is further configured to generate the plurality of differential touch-discernible cues to discern the object-type of each object of the plurality of objects within the first proximity range of the assistive device. ### Claim 5 (depends on 4) The assistive device according to claim 4, wherein the second circuitry is further configured to determine at least one of a relative position or a height of each object of the plurality of objects with respect to the position of the user of the assistive device for the 3D arrangement, and wherein the at least one of the relative position or the height of each object is determined based on the sensor data received in real time or near-real time from the plurality of different types of sensors. ### Claim 6 (depends on 4) The assistive device according to claim 4, wherein the second circuitry is further configured to generate a three-dimensional (3D) digital model of the 3D real-world area within the first proximity range, wherein the 3D digital model is generated based on the received sensor data, and wherein the generated 3D digital model is utilized for the mapping of the plurality of objects within the first proximity range to the at least one haptic element of the plurality of haptic elements. ### Claim 7 (depends on 4) The assistive device according to claim 4, wherein the second circuitry is further configured to: acquire a first template map of the 3D real-world area within the first proximity range of the assistive device from a server, wherein the first template is acquired based on the position of the user; and update the first template map with at least positional information of the plurality of objects based on the sensor data of the 3D real-world area within the first proximity range of the assistive device, wherein the at least positional information of the plurality of objects is received from the plurality of different types of sensors in real time or near-real time. ### Claim 8 (depends on 1) The assistive device according to claim 1, wherein the first proximity range is selected on the assistive device. ### Claim 9 (depends on 1) The assistive device according to claim 1, wherein the second circuitry is further configured to update the mapping of the plurality of objects within the first proximity range to the at least one haptic element of the plurality of haptic elements in the defined region of the haptic feedback interface, wherein the mapping is updated based on a change in position of at least one movable object of the plurality of objects or the user, and wherein the change in the position is detected in real time or near-real time from the received sensor data. ### Claim 10 (depends on 9) The assistive device according to claim 9, wherein the haptic feedback generator is further configured to dynamically update the touch-discernible feedback on the haptic feedback interface based on the update of the mapping, and wherein the updated touch-discernible feedback comprises a change in a relative position of at least one object of the plurality of objects with respect to the position of the user of the assistive device. ### Claim 11 (depends on 1) The assistive device according to claim 1, wherein the haptic feedback generator is further configured to output an audio feedback in combination with the touch-discernible feedback to enable non-visual discernment of the 3D real-world area within the first proximity range of the assistive device by the user as the user moves from a first location to a second location in the 3D real-world area within the first proximity range. ### Claim 12 (depends on 1) The assistive device according to claim 1, further comprises a learning engine in a memory of the assistive device, wherein the second circuitry is further configured to determine at least one pattern in a plurality of user interactions on the haptic feedback interface over a period of time, and wherein the at least one pattern is determined based on a tracking of a usage pattern of the assistive device by the learning engine. ### Claim 13 (depends on 12) The assistive device according to claim 12, wherein the second circuitry is further configured to adapt the mapping of the plurality of objects within the first proximity range to the at least one haptic element of the plurality of haptic elements in the defined region of the haptic feedback interface based on the determined at least one pattern. ### Claim 14 (depends on 1) The assistive device according to claim 1, wherein the haptic feedback interface is a haptic input/output interface. ### Claim 15 (depends on 14) The assistive device according to claim 14 further comprising a haptic feedback controller, wherein the haptic feedback controller is configured to detect a haptic input on the haptic feedback interface, and wherein the haptic input is detected based on a press on at least one of the plurality of differential touch-discernible cues generated in the touch-discernible haptic feedback. ### Claim 16 (depends on 1) The assistive device according to claim 1, wherein the scaling factor indicates a relative adjustment of at least one of a shape, a height, or a position of the plurality of objects for the mapping of the plurality of objects to the at least one haptic element of the plurality of haptic elements in the defined region of the haptic feedback interface. ### Claim 17 A method to provide assistance to a user, comprising: in an assistive device that comprises a first circuitry, a second circuitry, a haptic feedback generator, and a haptic feedback interface that includes a plurality of haptic elements: receiving, by the first circuitry, sensor data of a three-dimensional (3D) real-world area from a plurality of different types of sensors that are communicatively coupled to the assistive device, wherein the sensor data is associated with a first proximity range of the assistive device; determining a scaling factor based on the first proximity range and a surface area of a defined region of the haptic feedback interface; establishing, by the second circuitry, a mapping of a plurality of objects within the first proximity range to at least one haptic element of the plurality of haptic elements in the defined region of the haptic feedback interface; adjusting sizes of the plurality of objects to map the plurality of objects to the at least one haptic element in the defined region of the haptic feedback interface, wherein the sizes of the plurality of objects are adjusted based on the scaling factor, and wherein the mapping is established based on the received sensor data; and generating, by the haptic feedback generator, a touch-discernible feedback on the haptic feedback interface based on the established mapping, wherein the touch-discernible feedback comprises a plurality of differential touch-discernible cues to discern a 3D arrangement of each objects of the plurality of objects with respect to a position of the user of the assistive device, and wherein the touch-discernible feedback is for non-visual discernment of the 3D real-world area surrounding the user. ### Claim 18 (depends on 17) The method according to claim 17, further comprising identifying, by the second circuitry, an object-type of each object of the plurality of objects within the first proximity range of the assistive device, wherein the object-type of each object is identified based on the received sensor data. ### Claim 19 (depends on 18) The method according to claim 18, further comprising generating, by the haptic feedback generator, the plurality of differential touch-discernible cues to discern the object-type of each object of the plurality of objects within the first proximity range of the assistive device. ### Claim 20 (depends on 17) The method according to claim 17, further comprising determining, by the second circuitry, at least one of a relative position or a height of each object of the plurality of objects with respect to the position of the user of the assistive device for the 3D arrangement, wherein the at least one of the relative position or the height of each object is determined based on the sensor data received in real time or near-real time from the plurality of different types of sensors, and wherein the plurality of different types of sensors are worn by the user. ### Claim 21 (depends on 17) The method according to claim 17, further comprising generating, by the second circuitry, a three-dimensional (3D) digital model of the 3D real-world area within the first proximity range, wherein the 3D digital model is generated based on the received sensor data, and wherein the generated 3D digital model is utilized for the mapping of the plurality of objects within the first proximity range to the at least one haptic element of the plurality of haptic elements.