# US 2017/0255445 A1 · Translation of natural language into user interface actions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0255445 A1
- Document ID: US20170255445A1
- Status: Published application
- Kind: A1
- Application number: 14/912,011
- Filed: 2015-11-03
- Published: 2017-09-07
- Earliest claimed: 2015-11-03
- Assignees: Ovservepoint, Inc.
- Applicants: Observepoint, Inc.
- Inventors: Robert K. Seolas; John Raymond Pestana; Tyler Broadbent; Gregory Larson; Alan Martin Feurelein
- Web page: https://patentgrep.com/patent/US20170255445A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 8/38: for implementing user interfaces
- G06F 9/451: Execution arrangements for user interfaces
- G06F 40/35: Discourse or dialogue representation
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments are directed to methods and systems for deriving automation instructions. In one scenario, a computer system derives automation instructions by performing the following: rendering a user interface (UI) based on information from an information source and receiving natural language inputs from a user, where the natural language inputs specify an element description and an action type for UI elements rendered in the UI. The method also includes identifying UI elements in the UI that match the element descriptions in the natural language input and whose actions are performable according to their specified action type, and mapping the natural language inputs into executable code using information that corresponds to the identified UI elements.

## Claims

### Claim 1

A computerized method, performed at a computer system comprising at least one processor, for deriving automation instructions, the method comprising:
rendering a user interface (UI) based on information from an information source;
receiving one or more natural language inputs from a user, wherein the natural language inputs specify an element description and an action type for one or more UI elements rendered in the UI;
identifying one or more UI elements in the UI that match the element descriptions in the natural language input and whose actions are performable according to their specified action type; and
mapping the natural language inputs into executable code using information that corresponds to the one or more identified UI elements.

### Claim 2 (depends on 1)

The method as recited in claim 1, further comprising executing the executable code generated from the mapped natural language inputs.

### Claim 3 (depends on 2)

The method as recited in claim 2, further comprising:
determining that one or more actions performed as a result of executing the executable code did not function as expected;
implementing at least one secondary, different action; and
determining that the secondary, different action functioned as expected.

### Claim 4 (depends on 3)

The method of claim 3, further comprising using heuristics to determine which actions are most likely to be performed for each natural language input.

### Claim 5 (depends on 1)

The method of claim 1, wherein the information source comprises at least one of a software application or a web page.

### Claim 6 (depends on 5)

The method of claim 5, wherein the natural language inputs specify one or more actions that are to be taken on a software application UI or on a web page interface.

### Claim 7 (depends on 6)

The method of claim 6, wherein the executable code is repeatedly executed for a given natural language input until the natural language input matches a specified expected output.

### Claim 8 (depends on 7)

The method of claim 7, further comprising generating one or more portions of analytics data that indicates how accurately a given set of natural language inputs matched an given set of expected outputs.

### Claim 9 (depends on 1)

The method of claim 1, further comprising identifying, on the information source, which UI elements are actionable.

### Claim 10 (depends on 9)

The method of claim 9, wherein actionable UI elements are identified per action type, and are recorded in an element tree.

### Claim 11 (depends on 10)

The method of claim 10, wherein the element tree is constructed based on the UI elements in the rendered UI.

### Claim 12 (depends on 11)

The method of claim 11, wherein the construction of the element tree further comprises:
navigating through the information source by providing a plurality of navigation inputs to the information source;
mapping one or more UI views that are generated in response to the navigation inputs; and
constructing the element tree with the one or more UI elements based on the mapped UI views.

### Claim 13

A computerized method, performed at a computer system comprising at least one processor, for deriving automation instructions, the method comprising:
rendering a user interface (UI) based on information from an information source;
receiving one or more natural language inputs from a user, wherein the natural language inputs include an action type and an element description for one or more UI elements rendered in the UI;
identifying one or more UI elements in the UI that match the element descriptions in the natural language input and are performable according to their specified action type;
mapping the natural language inputs into executable code using information corresponding to the one or more identified UI elements;
executing the executable code generated from the mapped natural language inputs;
determining that one or more actions performed as a result of executing the executable code executed actions did not function as expected;
implementing at least one secondary, different action; and
determining that the secondary, different action functioned as expected.

### Claim 14 (depends on 13)

The method as recited in claim 13, wherein actionable UI elements are identified per action type, and are recorded in an element tree, the element tree being constructed based on the UI elements in the rendered UI.

### Claim 15 (depends on 14)

The method as recited in claim 14, wherein an element tree is constructed for each UI view, and wherein a plurality of constructed element trees are linked together.

### Claim 16 (depends on 13)

The method of claim 13, wherein the natural language inputs are translated into UI commands that apply to UI views of the information source.

### Claim 17 (depends on 16)

The method of claim 16, wherein the translation of natural language inputs into UI commands occurs in an emulator.

### Claim 18

A computer system comprising:
one or more processors;
a renderer configured to render a user interface (UI) based on information from an information source;
a natural language input receiver configured to receive one or more natural language inputs from a user, wherein the natural language inputs include an action type and an element description for one or more UI elements rendered in the UI;
an identifying module configured to identify one or more UI elements in the UI that match the element descriptions in the natural language input and are performable according to their specified action type; and
a natural language mapping module configured to map the natural language inputs into executable code using information corresponding to the one or more identified UI elements.

### Claim 19 (depends on 18)

The computer system of claim 18, wherein the one or more processors execute the executable code generated from the mapped natural language inputs.

### Claim 20 (depends on 18)

The computer system of claim 18, further comprising:
a determining module configured to determine that one or more actions performed as a result of executing the executable code did not function as expected; and
an implementing module configured to implement at least one secondary, different action, wherein the determining module determines that that the secondary, different action functioned as expected.
