# US 12,007,988 B2 · Interactive assistance for executing natural language queries to data sets > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,007,988 B2 - Document ID: US12007988B2 - Status: Granted - Kind: B2 - Application number: 18/182,303 - Filed: 2023-03-10 - Published: 2023-10-12 - Granted: 2024-06-11 - Earliest claimed: 2021-03-31 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Ramesh M Nallapati; Zhiguo Wang; Bing Xiang; Patrick Ng; Yung Haw Wang; Mukul Karnik; Nanyan Li; Sharanabasappa Parashuram Revadigar; Timothy Jones; Stephen Michael Ash; Sudipta Sengupta; Gregory David Adams; Deepak Shantha Murthy; Douglas Scott Cerny; Stephanie Weeks; Hanbo Li - Web page: https://patentgrep.com/patent/US12007988B2 ## CPC - G06F 16/24522: {Translation of natural language queries to structured queries} - G06F 16/2423: {Interactive query statement specification based on a database schema} - G06F 16/243: {Natural language query formulation} - G06F 40/295: Named entity recognition - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/08: Learning methods - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Interactive assistances for executing natural language queries to data sets may be performed. A natural language query may be received. Candidate entity linkages may be determined between an entity recognized in the natural language query and columns in data sets. The candidate linkages may be ranked according to confidence scores which may be evaluated to detect ambiguity for an entity linkage. Candidate entity linkages may be provided to a user via an interface to select an entity linkage to use as part of completing the natural language query. ## Claims ### Claim 1 A system, comprising: one or more processors; and a memory, that stores program instructions that, when executed by the at least one processor, cause the one or more processors to implement a business intelligence service, the business intelligence service configured to: receive a natural language query via an graphical interface for the business intelligence service that provides access to a plurality of different data sets; identify, using one or more machine learning models trained for natural language processing, a plurality of candidate entity linkages between an entity recognized in the natural language query and respective columns from one or more of the plurality of different data sets; responsive to a determination that respective confidence scores determined by the one or more machine learning models for the candidate entity linkages identified for the entity are not above a minimum confidence threshold: underline, in the graphical interface, the entity in the natural language query to indicate that the entity is ambiguous; and provide, in the graphical interface, a prompt to resolve the ambiguity of the entity before a result of the natural language query is provided. ### Claim 2 (depends on 1) The system of claim 1, wherein the prompt comprises one or more of the plurality of candidate entity linkages as selectable suggestions. ### Claim 3 (depends on 1) The system of claim 1, wherein the business intelligence service is further configured to: execute the natural language query to return a first result via the graphical interface before the prompt is provided; and execute the natural language query again to return a second result via the graphical interface based on a response to the prompt received via the graphical interface. ### Claim 4 (depends on 1) The system of claim 1, wherein the business intelligence service is further configured to execute the natural language query to return a result via the graphical interface based on a response to the prompt received via the graphical interface. ### Claim 5 (depends on 4) The system of claim 4, wherein the result comprises a visualization. ### Claim 6 (depends on 4) The system of claim 4, wherein the result comprises a restatement of the natural language query. ### Claim 7 (depends on 1) The system of claim 1, wherein the business intelligence service is further configured to provide via the interface one or more names for one or more data sets as part of an auto completion of the natural language query. ### Claim 8 A method, comprising: receiving a natural language query via an graphical interface for a business intelligence service that provides access to a plurality of different data sets; identifying, by the business intelligence service using one or more machine learning models trained for natural language processing, a plurality of candidate entity linkages between an entity recognized in the natural language query and respective columns from one or more of the plurality of different data sets; responsive to a determination that respective confidence scores determined by the one or more machine learning models for the candidate entity linkages identified for the entity are not above a minimum confidence threshold: underlining, in the graphical interface, the entity in the natural language query to indicate that the entity is ambiguous; and providing, in the graphical interface, a prompt to resolve the ambiguity of the entity before a result of the natural language query is provided. ### Claim 9 (depends on 8) The method of claim 8, wherein the prompt comprises one or more of the plurality of candidate entity linkages as selectable suggestions. ### Claim 10 (depends on 8) The method of claim 8, further comprising: executing, by the business intelligence service, the natural language query to return a first result via the graphical interface before the prompt is provided; and executing, by the business intelligence service, the natural language query again to return a second result via the graphical interface based on a response to the prompt received via the graphical interface. ### Claim 11 (depends on 8) The method of claim 8, further comprising executing, by the business intelligence service, the natural language query to return a result via the graphical interface based on a response to the prompt received via the graphical interface. ### Claim 12 (depends on 11) The method of claim 11, wherein the result comprises a visualization. ### Claim 13 (depends on 11) The method of claim 11, wherein the result comprises a restatement of the natural language query. ### Claim 14 (depends on 8) The method of claim 8, further comprising providing via the interface one or more names for one or more data sets as part of an auto completion of the natural language query. ### Claim 15 One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement: receiving a natural language query via an graphical interface for a business intelligence service that provides access to a plurality of different data sets; identifying, using one or more machine learning models trained for natural language processing, a plurality of candidate entity linkages between an entity recognized in the natural language query and respective columns from one or more of the plurality of different data sets; responsive to a determination that respective confidence scores determined by the one or more machine learning models for the candidate entity linkages identified for the entity are not above a minimum confidence threshold: underlining, in the graphical interface, the entity in the natural language query to indicate that the entity is ambiguous; and providing, in the graphical interface, a prompt to resolve the ambiguity of the entity before a result of the natural language query is provided. ### Claim 16 (depends on 15) The one or more non-transitory computer-readable storage media of claim 15, wherein the prompt comprises one or more of the plurality of candidate entity linkages as selectable suggestions. ### Claim 17 (depends on 15) The one or more non-transitory computer-readable storage media of claim 15, storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement: executing the natural language query to return a first result via the graphical interface before the prompt is provided; and executing the natural language query again to return a second result via the graphical interface based on a response to the prompt received via the graphical interface. ### Claim 18 (depends on 15) The one or more non-transitory computer-readable storage media of claim 15, storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement executing, by the business intelligence service, the natural language query to return a result via the graphical interface based on a response to the prompt received via the graphical interface. ### Claim 19 (depends on 18) The one or more non-transitory computer-readable storage media of claim 18, wherein the result comprises a visualization. ### Claim 20 (depends on 15) The one or more non-transitory computer-readable storage media of claim 15, storing further program instructions that when executed on or across the one or more computing devices, cause the one or more computing devices to further implement providing via the interface one or more names for one or more data sets as part of an auto completion of the natural language query.