US 12,718,004 B2Granted

Authoring data stories with generative AI

AssigneeSalesforce, Inc.

InventorsNicole Barbosa Sultanum, Arjun Srinivasan

Life of this patent

Estimating the term…

Loading patent…

Abstract

A computing device receives user interaction with a portion of a data visualization that is displayed on the computing device. In response to receiving the user interaction, the computing device generates a first prompt that includes parameters specifying a visualization type of the data visualization, a data array corresponding to data marks of the data visualization, and a title of the data visualization. The computing device inputs the first prompt into a large language model (LLM), and obtains from the LLM a text narrative for the portion of the data visualization. The device generates a second prompt in accordance with the text narrative, inputs the second prompt into the LLM, and obtains from the LLM a title for the text narrative. The device generates a self-contained story comprising the text narrative and the title.

Claims20

Dependent claims

A method of authoring data stories with large language models (LLMs), performed at a computing device having one or more processors and memory storing one or more programs configured for execution by the one or more processors, the method comprising:

receiving a first user interaction with one or more portions of a data visualization that is displayed on the computing device;
in response to receiving the first user interaction, generating a first prompt that includes a first plurality of parameters, the first plurality of parameters including:
a first parameter specifying a visualization type of the displayed data visualization;
a second parameter specifying a data array corresponding to data marks of the displayed data visualization;
and
a third parameter specifying a title of the displayed data visualization;
inputting the first prompt into a large language model (LLM) and obtaining, from the LLM, a text narrative for the one or more portions of the displayed data visualization, the text narrative including one or more sentences;
determining, using a combination of dependency parsing and keyword matching to map phrases in the one or more sentences to attributes and values in the displayed data visualization, that a first sentence of the one or more sentences of the text narrative includes a first keyword that references a first data element in the displayed data visualization;
in accordance with the determination that the first sentence includes the first keyword that references the first data element, generating a first correlation that links the first keyword in the text narrative to the first data element in the displayed data visualization;
after obtaining the text narrative, generating a second prompt in accordance with the text narrative;
inputting the second prompt into the LLM and obtaining, from the LLM, a title for the text narrative;
generating a self-contained story comprising the text narrative and the title;
concurrently displaying the story and the data visualization;
while concurrently displaying the story and the data visualization, receiving a second user interaction with a displayed portion of the text narrative of the story;
in response to receiving the second user interaction:
automatically visually emphasizing an appearance of the displayed portion of the text narrative;
determining that the displayed portion of the text narrative is correlated to a second data element in the displayed data visualization;
and
in accordance with the determination that the displayed portion of the text narrative is correlated to the second data element:
determining a portion of the displayed data visualization corresponding to the second data element;
and
visually emphasizing the portion of the displayed data visualization simultaneously with the visually emphasized appearance of the displayed portion of the text narrative.

19 more elements

A computing device, comprising:

one or more processors;
and
memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a first user interaction with one or more portions of a data visualization that is displayed on the computing device;
in response to receiving the first user interaction, generating a first prompt that includes a first plurality of parameters, the first plurality of parameters including:
a first parameter specifying a visualization type of the displayed data visualization;
a second parameter specifying a data array corresponding to data marks of the displayed data visualization;
and
a third parameter specifying a title of the displayed data visualization;
inputting the first prompt into a large language model (LLM) and obtaining, from the LLM, a text narrative for the one or more portions of the displayed data visualization, the text narrative including one or more sentences;
determining, using a combination of dependency parsing and keyword matching to map phrases in the one or more sentences to attributes and values in the displayed data visualization, that a first sentence of the one or more sentences of the text narrative includes a first keyword that references a first data element in the displayed data visualization;
in accordance with the determination that the first sentence includes the first keyword that references the first data element, generating a first correlation that links the first keyword in the text narrative to the first data element in the displayed data visualization;
after obtaining the text narrative, generating a second prompt in accordance with the text narrative;
inputting the second prompt into the LLM and obtaining, from the LLM, a title for the text narrative;
generating a self-contained story comprising the text narrative and the title;
concurrently displaying the story and the data visualization;
while concurrently displaying the story and the data visualization, receiving a second user interaction with a displayed portion of the text narrative of the story;
in response to receiving the second user interaction:
automatically visually emphasizing an appearance of the displayed portion of the text narrative;
determining that the displayed portion of the text narrative is correlated to a second data element in the displayed data visualization;
and
in accordance with the determination that the displayed portion of the text narrative is correlated to the second data element:
determining a portion of the displayed data visualization corresponding to the second data element;
and
visually emphasizing the portion of the displayed data visualization simultaneously with the visually emphasized appearance of the displayed portion of the text narrative.

22 more elements

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device having one or more processors and memory, cause the computing device to perform operations comprising:

receiving a first user interaction with one or more portions of a data visualization that is displayed on the computing device;
in response to receiving the first user interaction, generating a first prompt that includes a first plurality of parameters, the first plurality of parameters including:
a first parameter specifying a visualization type of the displayed data visualization;
a second parameter specifying a data array corresponding to data marks of the displayed data visualization;
and
a third parameter specifying a title of the displayed data visualization;
inputting the first prompt into a large language model (LLM) and obtaining, from the LLM, a text narrative for the one or more portions of the displayed data visualization, the text narrative including one or more sentences;
determining, using a combination of dependency parsing and keyword matching to map phrases in the one or more sentences to attributes and values in the displayed data visualization, that a first sentence of the one or more sentences of the text narrative includes a first keyword that references a first data element in the displayed data visualization;
in accordance with the determination that the first sentence includes the first keyword that references the first data element, generating a first correlation that links the first keyword in the text narrative to the first data element in the displayed data visualization;
after obtaining the text narrative, generating a second prompt in accordance with the text narrative;
inputting the second prompt into the LLM and obtaining, from the LLM, a title for the text narrative;
generating a self-contained story comprising the text narrative and the title;
concurrently displaying the story and the data visualization;
while concurrently displaying the story and the data visualization, receiving a second user interaction with a displayed portion of the text narrative of the story;
in response to receiving the second user interaction:
automatically visually emphasizing an appearance of the displayed portion of the text narrative;
determining that the displayed portion of the text narrative is correlated to a second data element in the displayed data visualization;
and
in accordance with the determination that the displayed portion of the text narrative is correlated to the second data element:
determining a portion of the displayed data visualization corresponding to the second data element;
and
visually emphasizing the portion of the displayed data visualization simultaneously with the visually emphasized appearance of the displayed portion of the text narrative.

19 more elements

Dependencies are inferred from the claim text.