# US 2022/0036077 A1 · Communication support device, communication support method, computer-readable storage medium including program, and server

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036077 A1
- Document ID: US20220036077A1
- Status: Published application
- Kind: A1
- Application number: 17/391,615
- Filed: 2021-08-02
- Published: 2022-02-03
- Earliest claimed: 2020-08-03
- Assignees: OMRON Corporation
- Applicants: OMRON Corporation
- Inventors: Endri RAMA; Kazuo YAMAMOTO; Tomohiro YABUUCHI
- Web page: https://patentgrep.com/patent/US20220036077A1

## CPC

- G06V 20/20: in augmented reality scenes
- A61F 9/08: Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
- G06T 7/248: {involving reference images or patches}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/82: using neural networks
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/174: {Facial expression recognition}
- G08B 6/00: Tactile signalling systems, e.g. tactile personal calling systems
- G08B 21/22: responsive to presence or absence of persons
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20064: Wavelet transform [DWT]
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30201: Face
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

The communication support device includes a position acquisition unit, an imaging unit, a storage, a category ranking setting unit, a counterpart detector, and a notification unit. The position acquisition unit acquires position information indicating a position of a user. The imaging unit captures an image of a surrounding environment of the user to acquire a captured image. The storage stores the counterpart database. In the counterpart database, an image of a counterpart and a category indicating a property of the counterpart are associated with the counterpart. The category ranking setting unit sets a priority to the category according to the position information acquired by the position acquisition unit. The counterpart detector detects a counterpart belonging to the category in the captured image in order of the priority set by the category ranking setting unit. The notification unit notifies the user of information regarding the counterpart detected by the counterpart detector.

## Claims

### Claim 1

A communication support device comprising:
a position acquisition unit that acquires position information indicating a position of a user;
an imaging unit that captures an image of a surrounding environment of the user to acquire a captured image;
a storage that stores a counterpart database in which an image of a counterpart and a category indicating a property of the counterpart are associated with the counterpart;
a category ranking setting unit that sets a priority to the category according to the position information;
a counterpart detector that detects a counterpart belonging to the category in the captured image in order of the priority set by the category ranking setting unit; and
a notification unit that notifies the user of information regarding the counterpart detected by the counterpart detector.

### Claim 2 (depends on 1)

The communication support device according to claim 1, further comprising:
a distance measuring unit that measures a distance between the counterpart and the imaging unit based on the captured image;
an expression determination unit that determines a facial expression of the counterpart based on the captured image; and
a motion determination unit that determines a motion of the counterpart based on the captured image; and
wherein the notification unit notifies the user of identification information for identifying the counterpart by a voice when the distance measured by the distance measuring unit is an interaction distance of a first threshold or less, and notifies the user of the identification information and at least one of facial expression information related to the facial expression determined by the expression determination unit and motion information related to the motion determined by the motion determination unit by a voice when the distance measured by the distance measuring unit is longer than the first threshold.

### Claim 3 (depends on 2)

The communication support device according to claim 2, wherein
the notification unit
notifies the user of the identification information and one of the facial expression information and the motion information when the distance measured by the distance measuring unit is a perceptual distance longer than the first threshold and equal to or less than a second threshold, and
notifies the user of the facial expression information, the motion information, and the identification information when the distance measured by the distance measuring unit is an observation distance longer than the second threshold.

### Claim 4 (depends on 3)

The communication support device according to claim 3, wherein
the notification unit
notifies the user of one of the facial expression information and the motion information by a voice after notifying the user of the identification information by a voice when the distance between the counterpart and the imaging unit, measured by the distance measuring unit at a timepoint when detection has been performed by the counterpart detector, is the perceptual distance, and
then, notifies the user of the other of the facial expression information and the motion information by a voice when the distance between the counterpart and the imaging unit measured by the distance measuring unit is the perceptual distance.

### Claim 5 (depends on 2)

The communication support device according to claim 2, further comprising a vibration unit that notifies the user of a detection result obtained by the distance measuring unit by a vibration,
wherein the vibration unit changes at least one of a pattern and a magnitude of the vibration according to the distance between the counterpart and the imaging unit detected by the distance measuring unit.

### Claim 6 (depends on 5)

The communication support device according to claim 5, wherein
the imaging unit acquires a plurality of captured images by capturing an image of the surrounding environment of the user in a time-series manner,
the communication support device further comprising a counterpart tracker that tracks the counterpart detected by the counterpart detector in the plurality of captured images,
the vibration unit includes two or more vibrating elements capable of vibrating, and
the vibration unit
causes the respective vibrating elements to sequentially vibrate in a predetermined order when the distance between the counterpart and the imaging unit detected by the distance measuring unit is a predetermined feedback distance in a case where the counterpart tracker tracks the counterpart, and
causes the respective vibrating elements to sequentially vibrate in an order different from the predetermined order in a case where it is difficult for the counterpart tracker to track the counterpart that has been tracked.

### Claim 7 (depends on 6)

The communication support device according to claim 6, wherein
the vibrating elements are three or more vibrating elements which are regularly arranged, and
the vibration unit
causes the respective vibrating elements to sequentially vibrate in an arranged order when the distance between the counterpart and the imaging unit detected by the distance measuring unit is the feedback distance in the case where the counterpart tracker tracks the counterpart, and
causes the respective vibrating elements to sequentially vibrate in a reverse order of the arranged order in the case where it is difficult for the counterpart tracker to track the counterpart that has been tracked.

### Claim 8 (depends on 6)

The communication support device according to claim 6, further comprising a counterpart position detector that detects a position of the counterpart with respect to the imaging unit based on the captured image,
wherein the vibrating elements include a right vibrating element and a left vibrating element, and
the vibration unit causes the right vibrating element to vibrate when the position of the counterpart with respect to the imaging unit detected by the counterpart position detector is a right side of an optical axis of the imaging unit, and causes the left vibrating element to vibrate when the position of the counterpart is a left side of the optical axis of the imaging unit.

### Claim 9

A server comprising:
a position acquisition unit that acquires position information indicating a position of a user;
a counterpart database in which an image of a counterpart and a category indicating a property of the counterpart are associated with the counterpart;
a category ranking setting unit that sets a priority to the category according to the position information; and
a transmission unit that transmits information in the counterpart database and information indicating the priority set by the category ranking setting unit to a communication support device that detects the counterpart based on the information in the counterpart database and the information indicating the priority, and notifies the user of the detected counterpart.

### Claim 10 (depends on 9)

A communication support device, which receives the information in the counterpart database and the information indicating the priority set by the category ranking setting unit from the server according to claim 9, the communication support device comprising:
an imaging unit that captures an image of a surrounding environment of the user to acquire a captured image;
a counterpart detector that detects a counterpart belonging to the category in the captured image in order of the priority received from the server; and
a notification unit that notifies the user of information regarding the counterpart detected by the counterpart detector.

### Claim 11

A communication support method comprising:
causing a position acquisition unit to acquire position information indicating a position of a user;
causing an imaging unit to capture an image of a surrounding environment of the user to acquire a captured image;
storing a counterpart database in which an image of a counterpart and a category indicating a property of the counterpart are associated with the counterpart, in a storage;
causing a category ranking setting unit to set a priority to the category according to the position information;
causing a counterpart detector to detect a counterpart belonging to the category in the captured image in order of the priority set by the category ranking setting unit; and
causing a notification unit to notify the user of information regarding the counterpart detected by the counterpart detector.

### Claim 12 (depends on 11)

A computer-readable storage medium including a program for causing a computer to execute the communication support method according to claim 11.
