# US 2021/0064334 A1 · Device operation apparatus, device operation system and device operation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0064334 A1
- Document ID: US20210064334A1
- Status: Published application
- Kind: A1
- Application number: 16/960,198
- Filed: 2018-01-18
- Published: 2021-03-04
- Earliest claimed: 2018-01-18
- Assignees: Mitsubishi Electric Corporation
- Applicants: Mitsubishi Electric Corporation
- Inventors: Masashi HIDAI
- Web page: https://patentgrep.com/patent/US20210064334A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Included are: an operation information acquirer for acquiring information indicating a function of an operation target device being an operation target; an image recognition unit for calculating, from image information on an image of a user who operates the operation target device, the user's line-of-sight information; a position calculator for calculating the position of an operation target device using information transmitted from the operation target device; a voice signal acquirer for acquiring a voice signal indicating an operation instruction for operating the operation target device; an operation target device specifying unit for specifying an operation target device to be a target of the operation instruction based on the calculated line-of-sight information and the calculated position of the operation target device; and a remote controller for generating an operation command for controlling the specified operation target device based on text information corresponding to the acquired operation instruction.

## Claims

### Claim 1

A device operation apparatus comprising:
processing circuitry to
acquire, as operation information, information indicating a function of at least one operation target device including a plurality of operation target devices that are operation targets;
calculate, from image information on an image of a user who operates the plurality of operation target devices, the user's line-of-sight information;
calculate a position of each of the plurality of operation target devices using information transmitted from the plurality of operation target devices;
store position information of each of the calculated operation target devices;
acquire a voice signal indicating an operation instruction for operating each of the plurality of operation target devices;
specify, when the processing circuitry acquires the voice signal, the at least one operation target device to be a target of the operation instruction on a basis of the calculated line-of-sight information and position information of each of the operation target devices, the position information being calculated and stored; and
generate an operation command for controlling the specified at least one operation target device on a basis of text information corresponding to the acquired operation instruction.

### Claim 2 (depends on 1)

The device operation apparatus according to claim 1,
wherein the processing circuitry calculates a position of the at least one operation target device on a basis of a light emission signal transmitted from a light emitting apparatus associated with the at least one operation target device.

### Claim 3 (depends on 1)

The device operation apparatus according to claim 1,
wherein the processing circuitry further estimates, when the processing circuitry fails to calculate a first position of one of the plurality of operation target devices, the first position of one operation target device that the processing circuitry fails to calculate, on a basis of second positions of the plurality of operation target devices, the second positions being successfully calculated.

### Claim 4 (depends on 1)

The device operation apparatus according to claim 1,
wherein the processing circuitry further stores the user's line-of-sight information calculated for a preset period,
wherein the processing circuitry refers to the stored line-of-sight information and specifies the at least one operation target device located in a direction of the user's line-of-sight vector as an operation target device to be a target of the operation instruction.

### Claim 5 (depends on 4)

The device operation apparatus according to claim 4,
wherein the processing circuitry refers to the stored line-of-sight information and specifies the at least one operation target device located in a direction of the user's line-of-sight vector during a period obtained by going back a certain period from a time at which the processing circuitry has acquired the voice signal as an operation target device to be a target of the operation instruction.

### Claim 6 (depends on 5)

The device operation apparatus according to claim 5,
wherein the processing circuitry specifies, when the plurality of operation target devices is located in a direction of the user's line-of-sight vector, the at least one operation target device located in the direction of the user's line-of-sight vector in a time zone closest to a time at which the operation instruction has been input as an operation target device to be a target of the operation instruction.

### Claim 7 (depends on 1)

The device operation apparatus according to claim 1,
wherein the text information is information for operation of the at least one operation target device, obtained by executing a voice recognition process and an interactive process on a voice stream corresponding to the acquired operation instruction.

### Claim 8 (depends on 1)

A device operation system comprising:
the device operation apparatus according to claim 1;
the at least one operation target device for performing control of a function corresponding to the operation command transmitted from the device operation apparatus; and
a light emitting apparatus for transmitting a light emission signal to the device operation apparatus, the light emitting apparatus being disposed while being associated with the at least one operation target device,
wherein the processing circuitry of the device operation apparatus calculates a position of the at least one operation target device on a basis of the light emission signal transmitted by the light emitting apparatus.

### Claim 9

A device operation method comprising:
acquiring information indicating a function of at least one operation target device including a plurality of operation target devices that are operation targets, as operation information;
calculating from image information on an image of a user who operates the plurality of operation target devices, the user's line-of-sight information;
calculating a position of each of the plurality of operation target devices using information transmitted from the plurality of operation target devices;
storing calculated position information of each of the operation target devices;
acquiring a voice signal indicating an operation instruction for operating each of the plurality of operation target devices;
specifying when the voice signal is acquired, the at least one operation target device to be a target of the operation instruction on a basis of the calculated line-of-sight information and the calculated and stored position information of each of the operation target devices; and
generating an operation command for controlling the at least one operation target device having been specified, on a basis of text information corresponding to the acquired operation instruction.
