# US 12,112,276 B2 · Training conversational computing interfaces with traced programs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,276 B2 - Document ID: US12112276B2 - Status: Granted - Kind: B2 - Application number: 18/064,475 - Filed: 2022-12-12 - Published: 2023-04-06 - Granted: 2024-10-08 - Earliest claimed: 2019-08-05 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: David Leo Wright Hall - Web page: https://patentgrep.com/patent/US12112276B2 ## CPC - G06F 8/51: Source to source - G06N 5/04: Inference or reasoning models - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 40/30: Semantic analysis - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06N 20/00: Machine learning - G06F 2009/45591: {Monitoring or debugging support} - G06N 5/041: {Abduction} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/223: {Execution procedure of a spoken command} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method of extending a conversational computing interface comprises obtaining data-flow traces indicating computational steps that occur based on execution of a nonnative skill implemented in a nonnative programming language of the conversational computing interface. A native computer-executable plan is automatically generated. The generated native computer-executable plan represents the data-flow traces in a native programming language of the conversational computing interface. The conversational computing interface is automatically trained based on the native computer-executable plan and a conversational event associated with the native computer-executable plan. ## Claims ### Claim 1 A method of extending a conversational computing interface, comprising: obtaining data-flow traces indicating computational steps occurring based on execution of a nonnative skill implemented in a nonnative programming language of the conversational computing interface, wherein obtaining the data-flow traces comprises tracing execution of the nonnative skill in response to an exemplary event; automatically generating a native computer-executable plan representing the data-flow traces in a native programming language of the conversational computing interface; and automatically training the conversational computing interface based on the native computer-executable plan and a conversational event associated with the native computer-executable plan. ### Claim 2 (depends on 1) The method of claim 1, further comprising: associating the native computer-executable plan with the exemplary event to generate an annotated dialogue; and automatically performing additional training of the conversational computing interface based at least on the annotated dialogue. ### Claim 3 (depends on 2) The method of claim 2, wherein automatically performing additional training of the conversational computing interface based at least on the annotated dialogue includes using the annotated dialogue as a labeled example for supervised training of the conversational computing interface. ### Claim 4 (depends on 2) The method of claim 2, wherein the annotated dialogue is associated with a data-flow trace representing a suitable action for responding to the exemplary event. ### Claim 5 (depends on 2) The method of claim 2, further comprising: querying a human user to provide the exemplary event. ### Claim 6 (depends on 5) The method of claim 5, further comprising: presenting the exemplary event to a different human user in order to prompt the different human user to provide input to guide generation of the native computer-executable plan. ### Claim 7 (depends on 2) The method of claim 2, wherein the data-flow traces are obtained by executing the nonnative skill and observing results of executing the nonnative skill. ### Claim 8 (depends on 7) The method of claim 7, wherein the exemplary event is automatically determined based at least on inputs received during execution of the nonnative skill. ### Claim 9 A method of training a conversational computing interface, comprising: obtaining data-flow traces indicating computational steps occurring based on execution of a skill that is nonnative to the conversational computing interface, wherein obtaining the data-flow traces comprises tracing execution of the nonnative skill in response to an exemplary event; automatically training the conversational computing interface based at least on the data-flow traces to generate a trained conversational computing interface; and using the trained conversational computing interface, automatically computer-generating a computer-executable plan in a programming language that is native to the trained conversational computing interface. ### Claim 10 (depends on 9) The method of claim 9, further comprising: associating the computer-executable plan with the exemplary event to generate an annotated dialogue; and automatically performing additional training of the trained conversational computing interface based at least on the annotated dialogue. ### Claim 11 (depends on 10) The method of claim 10, wherein automatically performing additional training of the trained conversational computing interface based at least on the annotated dialogue includes using the annotated dialogue as a labeled example for supervised training of the trained conversational computing interface. ### Claim 12 (depends on 10) The method of claim 10, wherein the data-flow traces are obtained from a human user operating the nonnative skill. ### Claim 13 (depends on 12) The method of claim 12, further comprising: prompting the human user to execute the nonnative skill in response to the exemplary event. ### Claim 14 (depends on 10) The method of claim 10, further comprising: querying a human user to provide the exemplary event. ### Claim 15 (depends on 10) The method of claim 10, wherein the data-flow traces are obtained by executing the nonnative skill and observing results of executing the nonnative skill. ### Claim 16 (depends on 15) The method of claim 15, wherein the exemplary event is automatically determined based at least on inputs received during execution of the nonnative skill. ### Claim 17 A computer system, comprising: a logic subsystem; and a storage subsystem holding instructions executable by one or more processors of the logic subsystem to: obtain data-flow traces indicating computational steps occurring based on execution of a nonnative skill implemented in a nonnative programming language of a conversational computing interface, wherein obtaining the data-flow traces comprises tracing execution of the nonnative skill in response to an exemplary event; automatically generate a native computer-executable plan representing the data-flow traces in a native programming language of the conversational computing interface; and automatically train the conversational computing interface based on the native computer-executable plan and a conversational event associated with the native computer-executable plan. ### Claim 18 (depends on 17) The computer system of claim 17, wherein the storage subsystem further holds instructions executable by the logic subsystem to: associate the native computer-executable plan with the exemplary event to generate an annotated dialogue; and automatically perform additional training of the conversational computing interface based at least on the annotated dialogue. ### Claim 19 (depends on 18) The computer system of claim 18, wherein automatically performing additional training of the conversational computing interface based at least on the annotated dialogue includes using the annotated dialogue as a labeled example for supervised training of the conversational computing interface. ### Claim 20 (depends on 17) The computer system of claim 17, wherein the data-flow traces are obtained by executing the nonnative skill and observing results of executing the nonnative skill.