# US 2021/0342401 A1 · Method, System, and Computer Program Product for Communication with an Intelligent Industrial Assistant and Industrial Machine

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342401 A1
- Document ID: US20210342401A1
- Status: Published application
- Kind: A1
- Application number: 17/376,365
- Filed: 2021-07-15
- Published: 2021-11-04
- Earliest claimed: 2018-07-17
- Assignees: iT SpeeX LLC
- Applicants: iT SpeeX LLC
- Inventors: Ronald D. Bagley, JR.
- Web page: https://patentgrep.com/patent/US20210342401A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G05B 19/41815: {characterised by the cooperation between machine tools, manipulators and conveyor or other workpiece supply system, workcell}
- G05B 19/41865: {characterised by job scheduling, process planning, material flow}
- G06F 9/451: Execution arrangements for user interfaces
- Y02P 90/80: Management or planning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS

## Abstract

Provided is a method for communication with an intelligent industrial assistant and industrial machine. The method may include receiving a first natural language input from a user. The first natural language input may be associated with a first command for an industrial machine to perform a first process. The industrial machine may be instructed to perform the first process based on the first natural language input. A second natural language input may be received from the user while the industrial machine is performing the first process. A first response may be determined based on the second natural language input. Communication of the first response to the user may be initiated while the industrial machine is performing the first process. A system and computer program product are also disclosed.

## Claims

### Claim 1

A method for communication with an intelligent industrial assistant and industrial machine, comprising:
receiving, by an intelligent industrial assistant, a first natural language input from a user, the first natural language input associated with a first command for an industrial machine to perform a first process;
mapping, by the intelligent industrial assistant, at least a portion of the first natural language input to at least a portion of a first dialogue template of a plurality of dialogue templates that is associated with the first command, wherein mapping the at least the portion of the first natural language input to the at least the portion of the first dialogue template comprises generating a first confidence score and a second confidence score for controlling selection of the first dialogue template of the plurality of dialogue templates based on comparing the first confidence score and the second confidence score to a plurality of thresholds;
instructing, by the intelligent industrial assistant, the industrial machine to perform the first process based on the first natural language input by determining the first command based on the first dialogue template and communicating command data associated with the first command to the industrial machine, wherein determining the first command comprises controlling selection of a first action associated with the first command from a plurality of actions based on at least one first prerequisite criterion;
receiving, by the intelligent industrial assistant, a second natural language input from the user while the industrial machine is performing the first process;
mapping, by the intelligent industrial assistant, at least a portion of the second natural language input to at least a portion of a second dialogue template of the plurality of dialogue templates;
determining, by the intelligent industrial assistant, a first response based on the second natural language input, wherein determining the first response is based on determining a second command based on the second dialogue template, communicating the second command, and receiving response data based on the second command, wherein determining the second command comprises controlling selection of a second action associated with the second command from the plurality of actions based on at least one second prerequisite criterion; and
initiating, by the intelligent industrial assistant, communication of the first response to the user while the industrial machine is performing the first process.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first response is based on a first context, and wherein the first context is based on at least one of the first process running on the industrial machine, one or more processes of a work cell that includes the industrial machine, a type of the industrial machine, or a role of the user.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving, by the intelligent industrial assistant, event data associated with a first event from the industrial machine; and
interrupting, by the intelligent industrial assistant, the communication of the response to the user.

### Claim 4 (depends on 3)

The method of claim 3, further comprising, before interrupting the communication of the response, determining, by the intelligent industrial assistant, whether to interrupt the communication of the first response based on the event data.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
determining, by the intelligent industrial assistant, a second response based on the event data; and
initiating, by the intelligent industrial assistant, communication of the second response to the user.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
upon completion of the communication of the second response to the user, continuing, by the intelligent industrial assistant, communication of the first response to the user.

### Claim 7 (depends on 1)

The method of claim 1, wherein the second natural language input is associated with at least one of a query of a database connected to the intelligent industrial assistant, a query regarding a status of the industrial machine, a query regarding a status of the first process, a request to set up a second process of the industrial machine, a request to generate a report associated with the industrial machine, a request to perform a third process by a second industrial machine, a request to coach the user how to perform a fourth process, or a request to teach the user a new skill.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first response comprises at least one of coaching the user how to perform a second process or teaching the user a new skill.

### Claim 9 (depends on 3)

The method of claim 3, wherein the first event comprises an alarm of the industrial machine.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
instructing, by the intelligent industrial assistant, the industrial machine to stop the first process based on the alarm.

### Claim 11 (depends on 1)

The method of claim 1, wherein the first natural language input comprises a first voice input, wherein the first confidence score is associated with a first probability that a first text representation accurately reflects the first voice input, and wherein the second confidence score is associated with mapping the first text representation to the first dialogue template.

### Claim 12

A system for communication with an intelligent industrial assistant and industrial machine, comprising:
at least one processor; and
at least one non-transitory computer-readable medium comprising instructions to direct the at least one processor to:
receive, via an intelligent industrial assistant, a first natural language input from a user, the first natural language input associated with a first command for an industrial machine to perform a first process;
map, via the intelligent industrial assistant, at least a portion of the first natural language input to at least a portion of a first dialogue template of a plurality of dialogue templates that is associated with the first command, wherein mapping the at least the portion of the first natural language input to the at least the portion of the first dialogue template comprises generating a first confidence score and a second confidence score for controlling selection of the first dialogue template of the plurality of dialogue templates based on comparing the first confidence score and the second confidence score to a plurality of thresholds;
instruct, via the intelligent industrial assistant, the industrial machine to perform the first process based on the first natural language input by determining the first command based on the first dialogue template and communicating command data associated with the first command to the industrial machine, wherein determining the first command comprises controlling selection of a first action associated with the first command from a plurality of actions based on at least one first prerequisite criterion;
receive, via the intelligent industrial assistant, a second natural language input from the user while the industrial machine is performing the first process;
map, via the intelligent industrial assistant, at least a portion of the second natural language input to at least a portion of a second dialogue template of the plurality of dialogue templates;
determine, via the intelligent industrial assistant, a first response based on the second natural language input, wherein determining the first response is based on determining a second command based on the second dialogue template, communicating the second command, and receiving response data based on the second command, wherein determining the second command comprises controlling selection of a second action associated with the second command from the plurality of actions based on at least one second prerequisite criterion; and
initiate, via the intelligent industrial assistant, communication of the first response to the user while the industrial machine is performing the first process.

### Claim 13 (depends on 12)

The system of claim 12, wherein the instructions further direct the at least one processor to:
receive, by the intelligent industrial assistant, event data associated with a first event from the industrial machine; and
interrupt, by the intelligent industrial assistant, the communication of the response to the user.

### Claim 14 (depends on 13)

The system of claim 13, wherein the instructions further direct the at least one processor to, before interrupting the communication of the response, determine, by the intelligent industrial assistant, whether to interrupt the communication of the first response based on the event data.

### Claim 15 (depends on 13)

The system of claim 13, wherein the instructions further direct the at least one processor to:
determine, by the intelligent industrial assistant, a second response based on the event data; and
initiate, by the intelligent industrial assistant, communication of the second response to the user.

### Claim 16 (depends on 15)

The system of claim 15, wherein the instructions further direct the at least one processor to:
upon completion of the communication of the second response to the user, continue, by the intelligent industrial assistant, communication of the first response to the user.

### Claim 17 (depends on 13)

The system of claim 13, wherein the first event comprises an alarm of the industrial machine, wherein the instructions further direct the at least one processor to:
instruct, by the intelligent industrial assistant, the industrial machine to stop the first process based on the alarm.

### Claim 18 (depends on 12)

The system of claim 12, wherein the first natural language input comprises a first voice input, wherein the first confidence score is associated with a first probability that a first text representation accurately reflects the first voice input, and wherein the second confidence score is associated with mapping the first text representation to the first dialogue template.

### Claim 19

A computer program product for communication with an intelligent industrial assistant and industrial machine, the computer program product comprising at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to:
receive, via an intelligent industrial assistant, a first natural language input from a user, the first natural language input associated with a first command for an industrial machine to perform a first process;
map, via the intelligent industrial assistant, at least a portion of the first natural language input to at least a portion of a first dialogue template of a plurality of dialogue templates that is associated with the first command, wherein mapping the at least the portion of the first natural language input to the at least the portion of the first dialogue template comprises generating a first confidence score and a second confidence score for controlling selection of the first dialogue template of the plurality of dialogue templates based on comparing the first confidence score and the second confidence score to a plurality of thresholds;
instruct, via the intelligent industrial assistant, the industrial machine to perform the first process based on the first natural language input by determining the first command based on the first dialogue template and communicating command data associated with the first command to the industrial machine, wherein determining the first command comprises controlling selection of a first action associated with the first command from a plurality of actions based on at least one first prerequisite criterion;
receive, via the intelligent industrial assistant, a second natural language input from the user while the industrial machine is performing the first process;
map, via the intelligent industrial assistant, at least a portion of the second natural language input to at least a portion of a second dialogue template of the plurality of dialogue templates;
determine, via the intelligent industrial assistant, a first response based on the second natural language input, wherein determining the first response is based on determining a second command based on the second dialogue template, communicating the second command, and receiving response data based on the second command, wherein determining the second command comprises controlling selection of a second action associated with the second command from the plurality of actions based on at least one second prerequisite criterion; and
initiate, via the intelligent industrial assistant, communication of the first response to the user while the industrial machine is performing the first process.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein the first natural language input comprises a first voice input, wherein the first confidence score is associated with a first probability that a first text representation accurately reflects the first voice input, and wherein the second confidence score is associated with mapping the first text representation to the first dialogue template.
