# US 12,762,163 B2 · Bimodal data exploration tool for interactive text and visual analysis > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,163 B2 - Document ID: US12762163B2 - Status: Granted - Kind: B2 - Application number: 18/802,611 - Filed: 2024-08-13 - Published: 2025-10-30 - Granted: 2026-10-13 - Earliest claimed: 2024-04-30 - Assignees: Salesforce, Inc. - Applicants: Salesforce, Inc. - Inventors: Dennis Nathan Bromley; Vidya Raghavan Setlur - Web page: https://patentgrep.com/patent/US12762163B2 ## CPC - G06F 16/3328: {using graphical result space presentation or visualisation} - G06F 16/383: using metadata automatically derived from the content - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computing device displays first data describing a dataset. At least a portion of the first data is encoded with metadata that links the first data to data values and/or data fields of the dataset. The computing device receives a user interaction with a first affordance. The user interaction specifies a first portion of the first data, which includes at least a first data field of the dataset. In response to receiving the user interaction, the computing device retrieves metadata corresponding to the first portion of the first data, and generates second data describing the dataset according to (i) the at least the first data field and (ii) data values of the at least the first data field specified in the metadata, corresponding to the first portion of the first data. The computing device concurrently displays the first data and the second data describing the dataset. ## Claims ### Claim 1 A method of analyzing data, performed at a computing device that includes one or more processors and memory, the method comprising: displaying, via a user interface, first data describing a dataset, wherein the first data has a first modality and a first portion of the first data is encoded as a data object that includes metadata information having (i) a semantic level of the first portion of the first data, (ii) a data value of the dataset, and (iii) a data field of the dataset; receiving a first user interaction with a first affordance that is displayed on the user interface, the first user interaction further specifying the data object; and in response to receiving the first user interaction with the first affordance: retrieving the metadata information corresponding to the data object; generating second data describing the dataset according to the retrieved metadata information, the generating includes assigning a semantic level to the second data according to a modality of the second data, wherein the second data has a second modality and the semantic level of the second data is different from the semantic level of the first portion of the first data; and concurrently displaying, on the user interface, the first data and the second data describing the dataset. ### Claim 2 (depends on 1) The method of claim 1, wherein the first modality or the second modality is one of: a text modality, a chart modality, an audio modality, a video modality, an augmented reality (AR) modality, or a virtual reality (VR) modality. ### Claim 3 (depends on 1) The method of claim 1, further comprising: after concurrently displaying, on the user interface, the first data and the second data describing the dataset: receiving a second user interaction with the first affordance, the second user interaction specifying a second portion of the second data that is displayed via the user interface; and in response to receiving the second user interaction with the first affordance: retrieving metadata corresponding to the second portion of the second data; generating third data describing the dataset according to a data field and/or data value of the dataset specified in the metadata corresponding to the second portion of the second data, wherein the third data has the same modality as the second data; and displaying the first data, the second data, and the third data concurrently on the user interface. ### Claim 4 (depends on 3) The method of claim 3, wherein: the second data comprises chart data that is displayed as a chart in the user interface; and the second portion of the second data comprises one of: a data mark of the chart, a cluster of data points of the chart, a label of the chart, an axis of the chart, or an axis value of the chart. ### Claim 5 (depends on 3) The method of claim 3, wherein: the second data comprises text data that is displayed text in the user interface; and the second portion of the second data comprises one of: a string of letters or numbers, a phrase, a sentence, or paragraph of text. ### Claim 6 (depends on 1) The method of claim 1, wherein: the first data describing the dataset comprises text data; and displaying the first data and the second data concurrently on the user interface includes the second data as additional text in the user interface. ### Claim 7 (depends on 1) The method of claim 1, further comprising: after concurrently displaying, on the user interface, the first data and the second data describing the dataset: receiving a second user interaction with a second affordance of the user interface, different from the first affordance of the user interface, the second user interaction specifying a second portion of the second data that is displayed via the user interface; and in response to receiving the second user interaction with the first affordance: retrieving metadata corresponding to the second portion of the second data; generating third data describing the dataset according to a data field and/or data value of the dataset specified in the metadata corresponding to the second portion of the second data, wherein the third data has a different modality from the second data; and displaying the first data, the second data, and the third data concurrently on the user interface. ### Claim 8 (depends on 1) The method of claim 1, wherein: the first data is text data and the second data is chart data. ### Claim 9 (depends on 8) The method of claim 8, wherein generating the second data describing the dataset includes determining a chart type for the second data according to at least one of: a number of data fields specified in the first portion of the first data, a data type of the data fields specified in the first portion of the first data, and semantics of the data fields specified in the first portion of the first data. ### Claim 10 (depends on 9) The method of claim 9, further comprising: when the number of data fields specified in the first portion of the first data is one, determining that the chart type for the second data is a bar chart; or when the number of data fields specified in the first portion of the first data is one, determining that the chart type for the second data is a scatter plot. ### Claim 11 (depends on 1) The method of claim 1, wherein: the first data is chart data and the second data is text data. ### Claim 12 (depends on 1) The method of claim 1, wherein the first data and the second data are chart data. ### Claim 13 (depends on 12) The method of claim 12, wherein the first data and the second data are displayed as different chart types. ### Claim 14 (depends on 1) The method of claim 1, wherein the first data and the second data are text data. ### Claim 15 (depends on 1) The method of claim 1, wherein: the second data is text data; and generating the second data describing the dataset includes generating the second data using a language model application. ### Claim 16 (depends on 1) The method of claim 1, further comprising: displaying a plurality of options, each of the options corresponding to a respective affordance, wherein receiving the first user interaction with the first affordance includes receiving user selection of a first option, of the plurality of options, corresponding to the first affordance. ### Claim 17 (depends on 1) The method of claim 1, wherein the first user interaction with the first affordance includes placement of the data object over the first affordance. ### Claim 18 A computing device, comprising: one or more processors; and memory coupled to the one or more processors, the memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for: displaying, via a user interface, first data describing a dataset, wherein the first data has a first modality and a first portion of the first data is encoded as a data object that includes metadata information having (i) a semantic level of the first portion of the first data, (ii) a data value of the dataset, and (iii) a data field of the dataset; receiving a first user interaction with a first affordance that is displayed on the user interface, the first user interaction further specifying the data object; and in response to receiving the first user interaction with the first affordance: retrieving the metadata information corresponding to the data object; generating second data describing the dataset according to the retrieved metadata information, the generating includes assigning a semantic level to the second data according to a modality of the second data, wherein the second data has a second modality and the semantic level of the second data is different from the semantic level of the first portion of the first data; and concurrently displaying, on the user interface, the first data and the second data describing the dataset. ### Claim 19 A non-transitory computer-readable medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations comprising: displaying, via a user interface, first data describing a dataset, wherein the first data has a first modality and a first portion of the first data is encoded as a data object that includes metadata information having (i) a semantic level of the first portion of the first data, (ii) a data value of the dataset, and (iii) a data field of the dataset; receiving a first user interaction with a first affordance that is displayed on the user interface, the first user interaction further specifying the data object; and in response to receiving the first user interaction with the first affordance: retrieving the metadata information corresponding to the data object; generating second data describing the dataset according to the retrieved metadata information, the generating includes assigning a semantic level to the second data according to a modality of the second data, wherein the second data has a second modality and the semantic level of the second data is different from the semantic level of the first portion of the first data; and concurrently displaying, on the user interface, the first data and the second data describing the dataset.