US 11,455,339 B1Granted

Incremental updates to natural language expressions in a data visualization user interface

AssigneeTableau Software, Inc.

InventorsJeffrey Ericson, Vidya Raghavan Setlur, Alex Djalali, Sanaz Golbabaei, Theodore Kornish, Xin Zhao, An Shi, Christoph Sachse, Eliana Leite Goldner, Suyang Duan

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A method is performed at a computing device that includes a display, one or more processors, and memory. The device displays a data visualization based on a dataset retrieved from a database. It also displays one or more phrases in a first region. The phrases define the data visualization. The device receives user input in a second region to specify a natural language command related to the displayed data visualization. In response to the user input, the device extracts independent analytic phrases from the command, determines one or more proposed actions based on the extracted phrases, and displays the proposed actions. The device receives user selection of a first proposed action. In response, the device generates an updated data visualization. It displays the updated data visualization and displays one or more updated phrases in the first region. The updated phrases define the updated data visualization.

Claims20

Dependent claims

A method of using natural language for updating visual analysis of datasets, comprising:

at a computing device having a display, one or more processors, and memory storing one or more programs configured for execution by the one or more processors:
displaying a data visualization based on data fields of a dataset retrieved from a database;
displaying, concurrently with the data visualization, a plurality of phrases in a first region in proximity to the data visualization, wherein the plurality of phrases collectively define the data visualization and each of the displayed phrases includes a respective one or more first field names of the data fields in the dataset;
receiving user input in a natural language command input region to specify a natural language command related to the displayed data visualization, wherein the natural language command input region is distinct from the first region;
in response to the user input:
determining that the natural language command includes a metacommand to:
(1) add a data field to the data visualization, or (2) remove a data field from the data visualization, or (3) replace a data field in the data visualization with another data field;
determining one or more proposed actions in accordance with the determined metacommand;
for each of the one or more proposed actions, generating one or more respective phrases corresponding to the respective proposed action;
ranking the one or more proposed actions based on a saliency or weight of a missing term that is inferred from the natural language command according to one or more inferencing rules based on syntactic constraints imposed by a context-free grammar and semantic constraints imposed by attributes of the data fields of the dataset;
and
displaying the ranked one or more proposed actions in a dropdown menu adjacent to the natural language command input region;
receiving user selection of a first proposed action of the ranked one or more proposed actions;
and
in response to the user selection of the first proposed action:
generating an updated data visualization in accordance with the first proposed action;
determining one or more updated phrases corresponding to the first proposed action, wherein the one or more updated phrases collectively define the updated data visualization and include one or more second field names of data fields in the dataset;
displaying the updated data visualization;
and
displaying the one or more updated phrases concurrently with the updated data visualization, in the first region.

17 more elements

A computing device, comprising:

one or more processors;
memory coupled to the one or more processors;
a display;
and
one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs comprising instructions for:
displaying a data visualization based on data fields of a dataset retrieved from a database;
displaying, concurrently with the data visualization, a plurality of phrases in a first region in proximity to the data visualization, wherein the plurality of phrases collectively define the data visualization and each of the displayed phrases includes a respective one or more first field names of the data fields in the dataset;
receiving user input in a natural language command input region to specify a natural language command related to the displayed data visualization, wherein the natural language command input region is distinct from the first region;
in response to the user input:
determining that the natural language command includes a metacommand to:
(1) add a data field to the data visualization, or (2) remove a data field from the data visualization, or (3) replace a data field in the data visualization with another data field;
determining one or more proposed actions in accordance with the determined metacommand;
for each of the one or more proposed actions, generating one or more respective phrases corresponding to the respective proposed action;
ranking the one or more proposed actions based on a saliency or weight of a missing term that is inferred from the natural language command according to one or more inferencing rules based on syntactic constraints imposed by a context-free grammar and semantic constraints imposed by attributes of the data fields of the dataset, and
displaying the ranked one or more proposed actions in a dropdown menu adjacent to the natural language command input region;
receiving user selection of a first proposed action of the ranked one or more proposed actions;
and
in response to the user selection of the first proposed action:
generating an updated data visualization in accordance with the first proposed action;
determining one or more updated phrases corresponding to the first proposed action, wherein the one or more updated phrases collectively define the updated data visualization and include one or more second field names of data fields in the dataset;
displaying the updated data visualization;
and
displaying the one or more updated phrases concurrently with the updated data visualization, in the first region.

20 more elements

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the device to perform the method of:

displaying a data visualization based on data fields of a dataset retrieved from a database;
displaying, concurrently with the data visualization, a plurality of phrases in a first region in proximity to the data visualization, wherein the plurality of phrases collectively define the data visualization and each of the displayed phrases includes a respective one or more first field names of the data fields in the dataset;
receiving user input in a natural language command input region to specify a natural language command related to the displayed data visualization, wherein the natural language command input region is distinct from the first region;
in response to the user input:
determining that the natural language command includes a metacommand to:
(1) add a data field to the data visualization, or (2) remove a data field from the data visualization, or (3) replace a data field in the data visualization with another data field;
determining one or more proposed actions in accordance with the determined metacommand;
for each of the one or more proposed actions, generating one or more respective phrases corresponding to the respective proposed action;
ranking the one or more proposed actions based on a saliency or weight of a missing term that is inferred from the natural language command according to one or more inferencing rules based on syntactic constraints imposed by a context-free grammar and semantic constraints imposed by attributes of the data fields of the dataset, and
displaying the one or more proposed actions in a dropdown menu adjacent to the natural language command input region;
receiving user selection of a first proposed action of the one or more proposed actions;
and
in response to the user selection of the first proposed action:
generating an updated data visualization in accordance with the first proposed action;
determining one or more updated phrases corresponding to the first proposed action, wherein the one or more updated phrases collectively define the updated data visualization and include one or more second field names of data fields in the dataset;
displaying the updated data visualization;
and
displaying the one or more updated phrases concurrently with the updated data visualization, in the first region.

15 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/601,437Filed 14 Oct 2019 · us provisional application
Priority: 6 Sep 2019
Granted: 27 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification19 codes

G06F 16/904
Details of database functions independent of the retrieved data types: browsing; Visualisation therefor
G06F 40/18
Of tables; using ruled lines: of spreadsheets
G06F 3/0482
Based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like…:…
G06F 16/243
Query formulation: natural language query formulation