# US 2016/0260353 A1 · Object recognition for the visually impaired

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0260353 A1
- Document ID: US20160260353A1
- Status: Published application
- Kind: A1
- Application number: 14/637,495
- Filed: 2015-03-04
- Published: 2016-09-08
- Earliest claimed: 2015-03-04
- Inventors: Arjun Kundan Dhawan
- Web page: https://patentgrep.com/patent/US20160260353A1

## CPC

- G09B 21/001: {Teaching or communicating with blind persons (G09B21/02 - G09B21/06 take precedence)}
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A three-dimensional sensor device is shown that may include a sensor network comprising one or more sensors configured to detect conditions proximate to the three-dimensional sensor device, a speech recognition module configured to detect and respond to operator speech patterns, an object detection module configured to detect objects proximate to the three-dimensional sensor device using contact-less detection, and a classification module configured to compare images received from the sensor network with known images stored in the image library or features of images that the sensor device has been trained to recognize. The three-dimensional sensor device may have improved qualities of object recognition for the visually impaired.

## Claims

### Claim 1

A method of object recognition, comprising:
(a) detecting operator speech instructing a three-dimensional sensor device to find an object;
(b) capturing an image;
(c) classifying the object in the image;
(d) detecting location information of the object relative to the operator; and
(e) conveying the location information to the operator until the object is found.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the three-dimensional sensor device comprises processing circuitry configured for:
(a) detecting operator speech instructing a three-dimensional sensor device to find an object;
(b) capturing an image;
(c) classifying the object in the image;
(d) detecting location information of the object relative to the operator; and
(e) conveying the location information to the operator until the object is found.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the three-dimensional sensor device comprises:
(a) a sensor network comprising one or more sensors configured to detect conditions proximate to the three-dimensional sensor device;
(b) a speech recognition module configured to detect and respond to operator speech patterns;
(c) an object detection module configured to detect objects proximate to the three-dimensional sensor device using contact-less detection; and
(d) a classification module configured to compare images received from the sensor network with known images stored in the image library or features of images that the sensor device has been trained to recognize.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the sensor network comprises at least one of a camera, an infrared (IR) depth sensor, an IR emitter, a leveling sensor, a voice sensor, or any combination thereof.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the speech recognition module is configured to receive speech information from at least one of the voice sensor or the camera.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the classification module is configured to receive object detection information from at least one of the camera or the IR depth sensor.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the camera provides images of objects to the classification module to compare the images with images of known objects from the image library or features of images that the sensor device has been trained to recognize

### Claim 8 (depends on 4)

The method according to claim 4, wherein the object detection module is configured to receive distance information from IR dot patterns from at least one of the camera, the IR emitter, the IR depth sensor, or any combination thereof.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising generating an auditory map of an area in which the three-dimensional sensor operates.

### Claim 10 (depends on 9)

The method according to claim 9, wherein generating the auditory map of the area comprises incorporating input from multiple sensors of the sensor network into a mapping module to determine current position of objects in the area.

### Claim 11

A three-dimensional sensor device, comprising:
(a) a sensor network comprising one or more sensors configured to detect conditions proximate to the three-dimensional sensor device;
(b) a speech recognition module configured to detect and respond to operator speech patterns;
(c) an object detection module configured to detect objects proximate to the three-dimensional sensor device using contact-less detection; and
(d) a classification module configured to compare images received from the sensor network with known images stored in the image library or features of images that the sensor device has been trained to recognize.

### Claim 12 (depends on 11)

The three-dimensional sensor device according to claim 11, further comprising processing circuitry configured for:
(a) detecting speech from an operator instructing a three-dimensional sensor device to find an object;
(b) capturing an image;
(c) classifying the object in the image;
(d) detecting location information of the object relative to the operator; and
(e) conveying the location information to the operator.

### Claim 13 (depends on 11)

The three-dimensional sensor device according to claim 11, wherein the sensor network comprises at least one of a color sensor, an infrared (IR) depth sensor, an IR emitter, a leveling sensor, a voice sensor, or any combination thereof.

### Claim 14 (depends on 13)

The three-dimensional sensor device according to claim 13, wherein the speech recognition module is configured to receive speech information from at least one of the voice sensor or the camera.

### Claim 15 (depends on 13)

The three-dimensional sensor device according to claim 13, wherein the object detection module is configured to receive distance information from IR dot patterns from at least one of the camera, the IR emitter, the IR depth sensor, or any combination thereof.

### Claim 16 (depends on 13)

The three-dimensional sensor device according to claim 13, wherein the classification module is configured to receive object detection information from at least one of the camera or the IR depth sensor.

### Claim 17 (depends on 16)

The three-dimensional sensor device according to claim 16, wherein the camera provides images of objects to the classification module to compare the images with images of known objects from the image library or features of images that the sensor device has been trained to recognize.

### Claim 18 (depends on 11)

The three-dimensional sensor device according to claim 11, further comprising a mapping module configured to generate an auditory map of an area in which the three-dimensional sensor device operates.

### Claim 19 (depends on 18)

The three-dimensional sensor device according to claim 18, wherein the mapping module is configured to incorporate input from multiple sensors of the sensor network to determine current positions of multiple objects in the area.

### Claim 20 (depends on 11)

The three-dimensional sensor device according to claim 11, wherein the three-dimensional sensor device is positioned in a wearable item.

### Claim 21 (depends on 1)

The method according to claim 1, wherein the operator is visually impaired.

### Claim 22 (depends on 1)

The method according to claim 1, wherein the location information is distance information between the object and the operator.

### Claim 23 (depends on 2)

The method according to claim 2, wherein the operator is visually impaired.

### Claim 24 (depends on 2)

The method according to claim 2, wherein the location information is distance information between the object and the operator.

### Claim 25 (depends on 12)

The method according to claim 12, wherein the operator is visually impaired.

### Claim 26 (depends on 12)

The method according to claim 12, wherein the location information is distance information between the object and the operator.
