# US 2019/0317594 A1 · System and method for detecting human gaze and gesture in unconstrained environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0317594 A1
- Document ID: US20190317594A1
- Status: Published application
- Kind: A1
- Application number: 15/955,333
- Filed: 2018-04-17
- Published: 2019-10-17
- Earliest claimed: 2018-04-17
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Simon A. I. STENT
- Web page: https://patentgrep.com/patent/US20190317594A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G05D 1/0221: {involving a learning process}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/55: from multiple images
- G06V 10/143: Sensing or illuminating at different wavelengths
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 40/19: Sensors therefor
- B25J 9/1697: {Vision controlled systems}
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

A system for gaze and gesture detection in unconstrained environments includes a 360-degree (omnidirectional) camera system, one or more depth sensors, and associated memory, processors and programming instructions to determine an object of a human user's attention in the unconstrained environment. The illustrative system may identify the object using eye gaze, gesture detection, and/or speech recognition. The system may generate a saliency map and identify areas of interest. A directionality vector may be projected on the saliency map to find intersecting areas of interest. The system may identify the object of attention once the object of attention is located.

## Claims

### Claim 1

A system for locating and identifying an object, the system comprising:
an image capturing device configured to generate a 360-degree image of an unconstrained environment comprising an individual and an object of interest;
a depth sensor configured to generate a depth map of the unconstrained environment;
a knowledge base; and
a processor configured to:
generate an environment map from the 360-degree image and the depth map;
determine a direction of attention from the individual in the 360-degree image;
generate a directionality vector representative of the direction of attention;
project the directionality vector from the individual on the environment map;
detect an intersection of the directionality vector with the object of interest; and
search the knowledge base for the identity of the object of interest using data relating to the object of interest.

### Claim 2 (depends on 1)

The system of claim 1, wherein the image capturing device is a panoramic camera.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processor is further configured to determine the direction of attention from at least one of an eye gaze, gesturing direction of the individual, or a combination thereof.

### Claim 4 (depends on 1)

The system of claim 1, wherein the depth sensor is one of an infrared sensor, a neural network trained for depth estimation, or a simultaneous localization and mapping sensor.

### Claim 5

A method for locating and identifying an object, the method comprising:
capturing a 360-degree image of an unconstrained environment comprising an individual and an object of interest;
generating an environment map;
determining a direction of attention from the individual in the captured 360-degree image;
generating a directionality vector representative of the direction of attention;
projecting the directionality vector from the individual on the environment map; and
detecting an intersection of the directionality vector with the object of interest.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
searching a knowledge base for an identity of the object of interest; and
initiating a learning process when the identity of the object of interest is not found in the knowledge base.

### Claim 7 (depends on 6)

The method of claim 6, wherein the learning process comprises a neural network.

### Claim 8 (depends on 5)

The method of claim 5, wherein the 360-degree image is captured by a monocular panoramic camera.

### Claim 9 (depends on 5)

The method of claim 5, further comprising:
capturing a plurality of images, wherein the 360-degree image comprises the plurality of images stitched together.

### Claim 10 (depends on 5)

The method of claim 5, wherein generating the environment map comprises:
obtaining environment depth information; and
combining the environment depth information with the 360-degree image.

### Claim 11 (depends on 10)

The method of claim 10, wherein the environment depth information is obtained by an infrared sensor.

### Claim 12 (depends on 10)

The method of claim 10, wherein the environment depth information is obtained from a simultaneous localization and mapping process.

### Claim 13 (depends on 10)

The method of claim 10, wherein the environment depth information is obtained from a neural network.

### Claim 14 (depends on 5)

The method of claim 5, wherein the environment map comprises a saliency map.

### Claim 15 (depends on 5)

The method of claim 5, wherein the environment map comprises a semantic parsing of the environment.

### Claim 16 (depends on 5)

The method of claim 5, further comprising estimating an eye gaze of the individual, wherein the direction of attention is determined from the eye gaze.

### Claim 17 (depends on 16)

The method of claim 16, wherein the eye gaze is estimated using infrared sensors.

### Claim 18 (depends on 5)

The method of claim 5, further comprising estimating a gesturing direction of the individual, wherein the direction of attention is determined from the gesturing direction.

### Claim 19 (depends on 5)

The method of claim 5, further comprising estimating an eye gaze and a gesturing direction of the individual, wherein the direction of attention is determined from the eye gaze and the gesturing direction.

### Claim 20

A system for locating and identifying an object, the system comprising:
a monocular camera configured to capture a 360-degree image of an unconstrained environment comprising an individual and an object of interest;
an infrared depth sensor configured to generate a three-dimensional depth map of the unconstrained environment;
a knowledge base; and
a processor configured to:
generate a saliency map from the 360-degree image and the depth map;
determine a direction of attention from the individual;
generate a three-dimensional directionality vector representative of the direction of attention;
project the three-dimensional directionality vector from the individual on the saliency map;
detect an intersection of the three-dimensional directionality vector with the object of interest; and
search the knowledge base for an identity of the object of interest using captured image data relating to the object of interest.
