# US 11,908,181 B2 · Generating multi-perspective responses by assistant systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,181 B2 - Document ID: US11908181B2 - Status: Granted - Kind: B2 - Application number: 18/164,480 - Filed: 2023-02-03 - Published: 2023-06-15 - Granted: 2024-02-20 - Earliest claimed: 2018-04-20 - Assignees: Meta Platforms, Inc. - Applicants: Meta Platforms, Inc. - Inventors: Emmanouil Koukoumidis; Michael Robert Hanson; Rajen Subba; Heidi Young; Rushin Shah; Jinsong Yu; Benoit F. Dumoulin; Jeremy Gillmor Kahn; Chandrasekhar Iyer - Web page: https://patentgrep.com/patent/US11908181B2 ## CPC - G06V 10/82: using neural networks - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 7/14: Merging, i.e. combining at least two sets of record carriers each arranged in the same ordered sequence to produce a single set having the same ordered sequence - G06F 9/453: {Help systems} - G06F 16/176: Support for shared access to files; File sharing support - G06F 16/2255: {Hash tables} - G06F 16/2365: {Ensuring data consistency and integrity} - G06F 16/243: {Natural language query formulation} - G06F 16/248: Presentation of query results - G06F 16/24552: {Database cache management} - G06F 16/24575: {using context} - G06F 16/24578: {using ranking} - G06F 16/338: Presentation of query results - G06F 16/3323: {using document space presentation or visualization, e.g. category, hierarchy or range presentation and selection} - G06F 16/3329: Natural language query formulation - G06F 16/3344: {using natural language analysis} - G06F 16/904: Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957) - G06F 16/9038: Presentation of query results - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 16/90335: {Query processing} - G06F 16/951: Indexing; Web crawling techniques - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 40/205: Parsing - G06F 40/295: Named entity recognition - G06F 40/30: Semantic analysis - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/08: Learning methods - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06Q 50/01 - G06V 10/764: using classification, e.g. of video objects - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/063: {Training} - G10L 15/07: to the speaker - G10L 15/16: using artificial neural networks - G10L 15/183: using context dependencies, e.g. language models - G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 17/06: Decision making techniques; Pattern matching strategies - G10L 17/22: Interactive procedures; Man-machine interfaces - H04L 5/02: Channels characterised by the type of signal - H04L 12/2816: {Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)} - H04L 41/20: {Network management software packages} - H04L 41/22: comprising specially adapted graphical user interfaces [GUI] - H04L 43/0882: Utilisation of link capacity - H04L 43/0894: Packet rate - H04L 51/18: Commands or executable codes - H04L 51/216: Handling conversation history, e.g. grouping of messages in sessions or threads - H04L 51/52: for supporting social networking services - H04L 67/306: User profiles - H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)} - H04L 67/5651: Reducing the amount or size of exchanged application data - H04L 67/75: Indicating network or usage conditions on the user display - H04W 12/08: Access security - G06F 2216/13: Prefetching - G10L 13/00: Speech synthesis; Text to speech systems - G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/225: {Feedback of the input speech} - H04L 51/046: Interoperability with other network applications or services - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04L 67/53: using third party service providers - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract In one embodiment, a method includes receiving a user query inputted on a head-mounted device from the head-mounted device, wherein the user query corresponds to multiple dialog-intents, executing multiple tasks corresponding to the multiple dialog-intents, generating a multi-perspective response by a stitching model based on two or more of execution results of the multiple tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results, and sending instructions to the head-mounted device for presenting the multi-perspective response on the head-mounted device. ## Claims ### Claim 1 A method comprising, by one or more computing systems: receiving, from a head-mounted device, a user query inputted on the head-mounted device, wherein the user query corresponds to a plurality of dialog-intents; executing a plurality of tasks corresponding to the plurality of dialog-intents; generating, by a stitching model, a multi-perspective response based on two or more of execution results of the plurality of tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results; and sending, to the head-mounted device, instructions for presenting the multi-perspective response on the head-mounted device. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining, based on the user query by a natural-language understanding module, the plurality of dialog-intents. ### Claim 3 (depends on 1) The method of claim 1, wherein each of the plurality of dialog-intents is associated with a particular agent of a plurality of agents. ### Claim 4 (depends on 3) The method of claim 3, further comprising: receiving, from the plurality of agents, a plurality of execution results corresponding to the plurality of tasks, respectively; and selecting the two or more of the execution results from the plurality of execution results for combination. ### Claim 5 (depends on 4) The method of claim 4, further comprising: determining respective relevance scores of the execution results with respect to the user query; and ranking the execution results based on their respective relevance scores; wherein selecting the two or more of the execution results for combination comprises selecting execution results based on their respective rankings. ### Claim 6 (depends on 4) The method of claim 4, wherein selecting the two or more of the execution results for combination comprises: determining, based on one or more machine-learning models, whether one or more of the execution results are mutually exclusive results with respect to one or more of the other execution results; and filtering one or more of the mutually exclusive results from the execution results; wherein selecting the two or more of the execution results for combination comprises selecting two or more execution results from the post-filtered execution results. ### Claim 7 (depends on 4) The method of claim 4, wherein selecting the two or more of the execution results for combination comprises: calculating, for each execution result of the execution results, an entropy value based on an information entropy of the respective execution result; and calculating, for each execution result of the execution results, information-gain values of the execution result based on its respective entropy value with respect to the entropy values of each other execution result of the execution results; wherein selecting the two or more of the execution results for combination comprises selecting two or more execution results based on their respective information-gain values. ### Claim 8 (depends on 1) The method of claim 1, further comprising: determining, by a natural-language understanding module, one or more slots associated with each of the plurality of dialog-intents. ### Claim 9 (depends on 1) The method of claim 1, wherein generating the multi-perspective response comprises: determining an order of the two or more of the execution results; and combining, by the stitching model, the two or more of the execution results based on the determined order. ### Claim 10 (depends on 9) The method of claim 9, wherein the order of the two or more of the execution results is determined based on a sequential-language model. ### Claim 11 (depends on 10) The method of claim 10, wherein the sequential-language model is trained based on a plurality of training data of human-combined utterances. ### Claim 12 (depends on 9) The method of claim 9, wherein the order of the two or more of the execution results is determined based on one or more predefined rules. ### Claim 13 (depends on 9) The method of claim 9, wherein the user query is associated with a first user, and wherein the order of the two or more of the execution results is determined based on user profile data associated with the first user. ### Claim 14 (depends on 9) The method of claim 9, wherein the order of the two or more of the execution results is determined based on execution times associated with the respective execution results. ### Claim 15 (depends on 9) The method of claim 9, wherein the user query is associated with a first user, and wherein the order of the two or more of the execution results is determined based on history data of user interactions by the first user with agents corresponding to the two or more of the execution results. ### Claim 16 (depends on 9) The method of claim 9, wherein the order of the two or more of the execution results is determined based on linguistic grounding. ### Claim 17 (depends on 1) The method of claim 1, wherein the multi-perspective response is based on one or more modalities. ### Claim 18 (depends on 17) The method of claim 17, wherein the one or more modalities comprise one or more of text, audio, image, or video. ### Claim 19 One or more computer-readable non-transitory storage media embodying software that is operable when executed to: receive, from a head-mounted device, a user query inputted on the head-mounted device, wherein the user query corresponds to a plurality of dialog-intents; execute a plurality of tasks corresponding to the plurality of dialog-intents; generate, by a stitching model, a multi-perspective response based on two or more of execution results of the plurality of tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results; and send, to the head-mounted device, instructions for presenting the multi-perspective response on the head-mounted device. ### Claim 20 A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to: receive, from a head-mounted device, a user query inputted on the head-mounted device, wherein the user query corresponds to a plurality of dialog-intents; execute a plurality of tasks corresponding to the plurality of dialog-intents; generate, by a stitching model, a multi-perspective response based on two or more of execution results of the plurality of tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results; and send, to the head-mounted device, instructions for presenting the multi-perspective response on the head-mounted device.