# US 2020/0401590 A1 · Translating a natural language query into a formal data query > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401590 A1 - Document ID: US20200401590A1 - Status: Published application - Kind: A1 - Application number: 16/446,809 - Filed: 2019-06-20 - Published: 2020-12-24 - Earliest claimed: 2019-06-20 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Peter Willem Jan Staar; Michele Dolfi; Christoph Auer; Leonidas Georgopoulos; Aleksandros Sobczyk; Tim Jan Baccaert; Konstantinos Bekas - Web page: https://patentgrep.com/patent/US20200401590A1 ## CPC - G06F 16/24522: {Translation of natural language queries to structured queries} - G06F 40/30: Semantic analysis - G06F 40/35: Discourse or dialogue representation - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer-implemented method for generating ground-truth for natural language querying may include providing a knowledge graph as data model, receiving a natural language query from a user and translating the natural language query into a formal data query. The method can also include visualizing the formal data query to the user and receiving a feedback response from the user. The feedback response can include a verified and/or edited formal data query. The method can also include storing the natural language query and the corresponding feedback response as ground-truth pair. Corresponding system and a related computer program product may be provided. ## Claims ### Claim 1 A computer-implemented method for generating ground-truth for natural language querying, the method comprising: providing a knowledge graph as data model; receiving a natural language query from a user; translating the natural language query into a formal data query; visualizing the formal data query to the user; receiving a feedback response from the user, the feedback response comprising a verified and/or edited formal data query; and storing the natural language query and the corresponding feedback response as ground-truth pair. ### Claim 2 (depends on 1) A computer-implemented method according to claim 1, wherein translating the natural language query into a formal data query comprises: providing the natural language query to a workflow model; and computing, by the workflow model, a workflow graph as formal data query. ### Claim 3 (depends on 2) A computer-implemented method according to claim 2, further comprising: using the ground-truth pair in a machine learning algorithm to train the workflow model. ### Claim 4 (depends on 2) A computer-implemented method according to claim 2, wherein the workflow graph is a directed acyclical graph. ### Claim 5 (depends on 2) A computer-implemented method according to claim 2, wherein the workflow graph comprises a plurality of operators. ### Claim 6 (depends on 5) A computer-implemented method according to claim 5, wherein the plurality of operators are selected from the group consisting of: an input operator; an edge traversal operator; a node filtering operator; a node ranking operator; logical operators; and an output operator. ### Claim 7 (depends on 1) A computer-implemented method according to claim 1, wherein translating the natural language query into a formal data query comprises: identifying one or more input node types of the knowledge graph in the natural language query; and identifying one or more output node types of the knowledge graph in the natural language query. ### Claim 8 (depends on 7) A computer-implemented method according to claim 7, wherein translating the natural language query into a formal data query comprises: computing one or more search paths through the knowledge graph between the one or more input node types and the one or more output node types. ### Claim 9 (depends on 8) A computer-implemented method according to claim 8, wherein computing the one or more search paths comprises performing a breadth-first-search algorithm. ### Claim 10 (depends on 7) A computer-implemented method according to claim 7, wherein computing the one or more search paths comprises: providing the natural language query as string input to a natural language classifier; and computing, by the natural language classifier, the one or more input node types and the one or more output node types. ### Claim 11 (depends on 1) A computer-implemented method according to claim 1, wherein visualizing the formal data query comprises: visualizing the formal data query in a visual programming tool. ### Claim 12 (depends on 11) A computer-implemented method according to claim 11, wherein the visual programming tool is configured to provide a test functionality to test the natural language query. ### Claim 13 (depends on 11) A computer-implemented method according to claim 11, wherein the visual programming tool is configured to provide an edit functionality to visually edit the formal data query. ### Claim 14 A computer system for performing a computer-implemented method for generating ground-truth for natural language querying, the system comprising a processing unit, a storage system and a user interface comprising an input unit and a display, the system being configured to: store a knowledge graph as data model in the storage system; receive, via the user interface, a natural language query; translate the natural language query into a formal data query; visualize the formal data query on the display; receive, via the input unit, a feedback response from a user, the feedback response comprising a verified and/or edited formal data query; and store the natural language query and the corresponding feedback response as ground-truth pair in the storage system. ### Claim 15 (depends on 14) A computer system according to claim 14, the system comprising: a translation module configured to translate the natural language query into the formal data query, the translation module comprising a workflow model; a training module configured to: receive the ground-truth pair; perform a training of the workflow model by means of a machine learning algorithm; and update the workflow model. ### Claim 16 (depends on 15) A computer system according to claim 15, wherein the translation module is configured to: identify input node types of the knowledge graph in the natural language query; identify output node types of the knowledge graph in the natural language query; and compute a workflow graph as formal data query. ### Claim 17 (depends on 16) A computer system according to claim 16, wherein the workflow graph is a directed acyclical graph. ### Claim 18 A computer program product for performing a computer-implemented method for generating ground-truth for natural language querying on a computing system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by the system to cause the system to perform a method comprising: providing a knowledge graph as data model; receiving a natural language query from a user; translating the natural language query into a formal data query; visualizing the formal data query to the user; receiving a feedback response from the user, the feedback response comprising a verified and/or edited formal data query; and providing the natural language query and the corresponding feedback response as ground-truth pair. ### Claim 19 (depends on 18) A computer program product according to claim 18, wherein translating the natural language query into a formal data query comprises: providing the natural language query to a workflow model; and computing, by the workflow model, a workflow graph as formal data query. ### Claim 20 (depends on 18) A computer program product according to claim 18, further comprising: using the ground-truth pair in a machine learning algorithm to train the workflow model. ### Claim 21 (depends on 19) A computer program product according to claim 19, wherein the workflow graph is a directed acyclical graph. ### Claim 22 A user interface comprising a display and an input unit, the user interface being configured to: receive a natural language query from a user; provide the natural language query to a translation module; receive a formal data query from the translation module, the formal data query being a translation of the natural language query in a formal language; visualize the formal data query on the display; receive a feedback response from the user, the feedback response comprising a verified and/or edited formal data query; and provide the natural language query and the corresponding feedback response as ground-truth pair. ### Claim 23 (depends on 22) A user interface according to claim 22, wherein visualizing the formal data query comprises: displaying a workflow graph as formal data query. ### Claim 24 (depends on 22) A user interface according to claim 22, wherein the user interface is configured to provide a test functionality to test the natural language query. ### Claim 25 (depends on 22) A user interface according to claim 22, wherein the user interface is configured to provide an edit functionality to visually edit the workflow graph.