# US 12,217,000 B1 · Optimizing natural language analytical conversations using platform-specific input and output interface functionality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,000 B1
- Document ID: US12217000B1
- Status: Granted
- Kind: B1
- Application number: 17/589,813
- Filed: 2022-01-31
- Granted: 2025-02-04
- Earliest claimed: 2021-09-10
- Assignees: Tableau Software, Inc.
- Applicants: Tableau Software, LLC
- Inventors: Vidya Raghavan Setlur; Melanie K. Tory
- Web page: https://patentgrep.com/patent/US12217000B1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 16/3329: Natural language query formulation
- G06F 16/34: Browsing; Visualisation therefor (browsing or visualisation for clustering or classification G06F16/358)
- G06F 40/186: Templates
- G06F 40/205: Parsing
- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06F 40/253: Grammatical analysis; Style critique
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method uses natural language processing for visual analysis of a dataset. The method includes receiving a first natural language (NL) input directed to a data source, from a first client. The method also includes parsing the first NL input into tokens based on a grammar and the data source. The method also includes generating and outputting an intermediate NL response, to a second client, based on the tokens and output modality of the second client. In response to receiving a user input to provide missing information: the method also includes generating an input query based on the user input; and querying the data source using the input query, to obtain a result set. The method also includes generating and outputting, to the second client, a first NL output and a snapshot of a data visualization, based on the result set and the output modality of the second client.

## Claims

### Claim 1

A method of using natural language for analysis of a dataset, comprising:
receiving a first natural language (NL) input directed to a data source, from a first client;
parsing the first NL input into tokens based on a grammar and the data source;
generating and outputting an intermediate NL response, to a second client, based on the tokens and output modality of the second client;
in response to receiving, from the second client, a user input to provide missing information in the first NL input:
generating an input query based on the user input; and
querying the data source using the input query, to obtain a result set;
generating and outputting, to the second client, a first NL output and a snapshot of a data visualization, based on the result set and the output modality of the second client;
in accordance with a determination that the intermediate NL response requires refinement or repair:
generating alternatives and interpretations for terms in the first NL input, wherein the alternatives include one or more alternative analytical functions, updates to one or more attributes, and/or value filters;
in accordance with a determination that the output modality of the second client includes a display:
providing, in an interface on the display of the second client, one or more affordances to refine and/or repair the alternatives and interpretations; and
generating the input query based on a selection of the one or more affordances on the second client; and
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and outputting, on the second client, speech output, based on alternatives and interpretations, for eliciting clarification through a series of verbal actions; and
generating the input query based on a clarifying user input on the second client.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
in accordance with a determination that the first NL input includes at least one ambiguous utterance:
in accordance with a determination that the output modality of the second client includes a display:
providing, in an interface on the display of the second client, one or more affordances, that correspond to alternatives for resolving ambiguity in the at least one ambiguous utterance; and
receiving, from the second client, a selection of the one or more affordances corresponding to an alternative that resolves ambiguity in the at least one ambiguous utterance; and
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the second client, a speech output of a textual explanation relevant to context of the data source; and
receiving, from the second client, a voice input that resolves ambiguity in the at least one ambiguous utterance.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
in accordance with a determination that the first NL input includes at least one underspecified utterance:
generating missing information in the at least one underspecified utterance, to create a valid query against the data source;
in accordance with a determination that the output modality of the second client includes a display:
providing, in an interface on the display of the second client, the missing information;
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the second client, a speech output corresponding to the missing information; and
receiving the user input comprises receiving, from the second client, a confirmation that the missing information fills in for underspecified information in the at least one underspecified utterance.

### Claim 4 (depends on 1)

The method of claim 1, further comprising, prior to receiving the first NL input:
in accordance with a determination that the output modality of the first client includes a display:
generating and displaying, on the first client, a brief summary of the data source on the display of the first client; and
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the first client, a speech output corresponding to a brief textual summary of the data source.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
selecting an expression type for the first NL input from amongst aggregation, grouping, filtering, limiting, and sorting;
selecting a response from a pre-defined template of responses based on the expression type;
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the second client, (i) a speech output corresponding to the response and (ii) a follow-up question; and
in accordance with a determination that the output modality of the second client includes a display:
generating and displaying, on the second client, a screenshot of a corresponding visualization for the response.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating and outputting the snapshot of the data visualization comprises:
in accordance with a determination that (i) a response to the input query requires a single answer and (ii) the second client is a messaging application:
displaying the single answer in the messaging application.

### Claim 7 (depends on 1)

The method of claim 1, wherein the second client is a messaging application, the method further comprising:
subsequently receiving a second NL input in the messaging application;
detecting an anaphora in an utterance in the second NL input; and
in accordance with a determination that the anaphora corresponds to a follow-up utterance:
generating and displaying a second NL output in a same thread as the first NL output.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first client and the second client are different applications.

### Claim 9 (depends on 1)

The method of claim 1, wherein the first client and the second client are a same application executing on different devices.

### Claim 10 (depends on 1)

The method of claim 1, wherein an input modality of the first client is different from the output modality of the second client.

### Claim 11 (depends on 1)

The method of claim 1, wherein generating and outputting the intermediate NL response and/or the first NL output to the second client comprises:
storing context and/or session information for user interactions on the first client; and
retrieving the context and/or session information for the user interactions on the first client and generating the intermediate NL response and/or the first NL output based on the context and/or session information.

### Claim 12

An electronic device, comprising:
a display;
one or more processors;
memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a first natural language (NL) input directed to a data source, from a first client;
parsing the first NL input into tokens based on a grammar and the data source;
generating and outputting an intermediate NL response, to a second client, based on the tokens and output modality of the second client;
in response to receiving a user input to provide missing information in the first NL input:
generating an input query based on the user input; and
querying the data source using the input query, to obtain a result set;
generating and outputting, to the second client, a first NL output and a snapshot of a data visualization, based on the result set and the output modality of the second client;
selecting an expression type for the first NL input from amongst aggregation, grouping, filtering, limiting, and sorting;
selecting a response from a pre-defined template of responses based on the expression type;
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the second client, (i) a speech output corresponding to the response and (ii) a follow-up question; and
in accordance with a determination that the output modality of the second client includes a display:
generating and displaying, on the second client, a screenshot of a corresponding visualization for the response.

### Claim 13 (depends on 12)

The electronic device of claim 12, wherein the one or more programs further comprise instructions for:
in accordance with a determination that the first NL input includes at least one ambiguous utterance:
in accordance with a determination that the output modality of the second client includes a display:
providing, in an interface on the display of the second client, one or more affordances, that correspond to alternatives for resolving ambiguity in the at least one ambiguous utterance; and
receiving, from the second client, a selection of the one or more affordances corresponding to an alternative that resolves ambiguity in the at least one ambiguous utterance; and
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the second client, a speech output of a textual explanation relevant to context of the data source; and
receiving, from the second client, a voice input that resolves ambiguity in the at least one ambiguous utterance.

### Claim 14 (depends on 12)

The electronic device of claim 12, wherein the one or more programs further comprise instructions for:
in accordance with a determination that the first NL input includes at least one underspecified utterance:
generating missing information in the at least one underspecified utterance, to create a valid query against the data source;
in accordance with a determination that the output modality of the second client includes a display:
providing, in an interface on the display of the second client, the missing information;
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the second client, a speech output corresponding to the missing information; and
receiving the user input comprises receiving, from the second client, a confirmation that the missing information fills in for underspecified information in the at least one underspecified utterance.

### Claim 15 (depends on 12)

The electronic device of claim 12, wherein the one or more programs further comprise instructions for:
in accordance with a determination that the intermediate NL response requires refinement or repair:
generating alternatives and interpretations for terms in the first NL input, wherein the alternatives include one or more alternative analytical functions, updates to one or more attributes, and/or value filters;
in accordance with a determination that the output modality of the second client includes a display:
providing, in an interface on the display of the second client, one or more affordances to refine and/or repair the alternatives and interpretations; and
generating the input query based on a selection of the one or more affordances on the second client; and
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and outputting, on the second client, speech output, based on alternatives and interpretations, for eliciting clarification through a series of verbal actions; and
generating the input query based on a clarifying user input on the second client.

### Claim 16 (depends on 12)

The electronic device of claim 12, wherein the one or more programs further comprise instructions for, prior to receiving the first NL input:
in accordance with a determination that the output modality of the first client includes a display:
generating and displaying, on the first client, a brief summary of the data source on the display of the first client; and
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the first client, a speech output corresponding to a brief textual summary of the data source.

### Claim 17 (depends on 12)

The electronic device of claim 12, wherein the second client is a messaging application, and the one or more programs further comprise instructions for:
subsequently receiving a second NL input in the messaging application;
detecting an anaphora in an utterance in the second NL input; and
in accordance with a determination that the anaphora corresponds to a follow-up utterance:
generating and displaying a second NL output in a same thread as the first NL output.

### Claim 18

A non-transitory computer readable storage medium storing one or more programs configured for execution by an electronic device with a display, the one or more programs comprising instructions for:
receiving a first natural language (NL) input directed to a data source, from a first client;
parsing the first NL input into tokens based on a grammar and the data source;
generating and outputting an intermediate NL response, to a second client, based on the tokens and output modality of the second client;
in response to receiving a user input to provide missing information in the first NL input:
generating an input query based on the user input; and
querying the data source using the input query, to obtain a result set;
generating and outputting, to the second client, a first NL output and a snapshot of a data visualization, based on the result set and the output modality of the second client;
selecting an expression type for the first NL input from amongst aggregation, grouping, filtering, limiting, and sorting;
selecting a response from a pre-defined template of responses based on the expression type;
in accordance with a determination that the output modality of the second client includes voice-only responses:
generating and playing, on the second client, (i) a speech output corresponding to the response and (ii) a follow-up question; and
in accordance with a determination that the output modality of the second client includes a display:
generating and displaying, on the second client, a screenshot of a corresponding visualization for the response.