# US 12,111,834 B1 · Ambient multi-device framework for agent companions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,111,834 B1 - Document ID: US12111834B1 - Status: Granted - Kind: B1 - Application number: 18/390,768 - Filed: 2023-12-20 - Granted: 2024-10-08 - Earliest claimed: 2023-12-20 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Victor Carbune; Arash Sadr; Matthew Sharifi - Web page: https://patentgrep.com/patent/US12111834B1 ## CPC - G06F 16/24553: {of query operations} - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 16/248: Presentation of query results - G06F 16/9035: Filtering based on additional data, e.g. user or group profiles - G06F 16/9038: Presentation of query results - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods for generating and providing outputs in a multi-device system can include leveraging environment-based prompt generation and generative model response generation to provide dynamic response generation and display. The systems and methods can obtain input data associated with one or more computing devices within an environment, can obtain environment data descriptive of the plurality of computing devices within the environment, and can generate a prompt based on the input data and environment data. The prompt can be processed with a generative model to generate a model-generated output. The model-generated output can then be transmitted to a particular computing device of the plurality of computing devices. ## Claims ### Claim 1 A computing system for determining an output device for providing a query response, the system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining input data, wherein the input data comprises a query associated with a particular user; obtaining environment data, wherein the environment data is descriptive of a plurality of computing devices in an environment of the user, wherein the plurality of computing devices are associated with a plurality of different output components; generating a prompt based on the input data and the environment data, wherein the prompt comprises data descriptive of the query and device information associated with at least a subset of the plurality of computing devices, wherein generating the prompt comprises: determining output capabilities for plurality of computing devices based on the environment data; generating a representation of performance capabilities of the plurality of computing devices; and generating the prompt based on the representation of performance capabilities of the plurality of computing devices and the input data; processing the prompt with a generative model to generate a model-generated output based on the data descriptive of the query and the device information, wherein processing the prompt with the generative model to generate the model-generated output comprises: generating a response to the query based on processing the data descriptive of the query with the generative model; and generating, with the generative model and based on the representation and the response, a model-generated output to be provided with a particular computing device of the plurality of computing devices based on the device information; and transmitting the model-generated output to the particular computing device. ### Claim 2 (depends on 1) The system of claim 1, wherein the model-generated output is generated to be provided with a particular output component of the plurality of different output components; wherein the generative model generates output device instructions, wherein the output device instructions are descriptive of a particular computing device of the plurality of computing devices to provide the model-generated output, wherein the particular computing device is associated with the particular output component; and wherein the model-generated output is transmitted to the particular computing device based on the output device instructions. ### Claim 3 (depends on 1) The system of claim 1, wherein processing the prompt with the generative model to generate the model-generated output comprises: generating a plurality of model outputs, wherein the plurality of model outputs comprises a plurality of candidate responses; and wherein transmitting the model-generated output to the particular computing device comprises: transmitting a first model output of the plurality of model outputs to a first computing device of the plurality of computing devices; and transmitting a second model output of the plurality of model outputs to a second computing device of the plurality of computing devices. ### Claim 4 (depends on 3) The system of claim 3, wherein the first model output comprises visual data for display via a visual display, and wherein the second model output comprises audio data for playback via a speaker component. ### Claim 5 (depends on 4) The system of claim 4, wherein the first computing device comprises a smart television, and wherein the second computing device comprises a smart speaker. ### Claim 6 (depends on 1) The system of claim 1, wherein generating the prompt based on the input data and the environment data comprises: determining an environment-specific device configuration based on the environment data; obtaining a prompt template from a prompt library based on the environment-specific device configuration; and augmenting the prompt template based on the input data to generate the prompt. ### Claim 7 (depends on 6) The system of claim 6, wherein the environment-specific device configuration is descriptive of respective output types and respective output quality for the plurality of computing devices, and wherein the prompt library comprises a plurality of different prompt templates associated with a plurality of different device configurations. ### Claim 8 (depends on 1) The system of claim 1, wherein the plurality of computing devices are connected via a cloud computing system, wherein each of the plurality of computing devices are registered with a platform of the cloud computing system, wherein the environment data is obtained with the cloud computing system, and wherein the model-generated output is transmitted via the cloud computing system. ### Claim 9 (depends on 1) The system of claim 1, wherein the plurality of computing devices are located proximate to each of the other computing devices within the plurality of computing devices, wherein the plurality of computing devices are communicatively connected over a local network, and wherein a particular computing device of the plurality of computing devices facilitates input data obtainment and model-generated output transmission. ### Claim 10 (depends on 1) The system of claim 1, wherein the generative model is communicatively connected with a search engine via an application programming interface, and wherein processing the prompt with the generative model to generate the model-generated output comprises: generating an application programming interface call based on the prompt; determining a plurality of search results with the search engine based on the application programming interface call; and processing the plurality of search results with the generative model to generate the model-generated output. ### Claim 11 A computer-implemented method, the method comprising: obtaining, by a computing system comprising one or more processors, input data, wherein the input data comprises a query associated with a particular user; obtaining, by the computing system, environment data, wherein the environment data is descriptive of a plurality of computing devices in an environment of the user, wherein the plurality of computing devices are associated with a plurality of different output components; generating, by the computing system, a prompt based on the input data and the environment data, wherein the prompt comprises data descriptive of the query and device information associated with at least a subset of the plurality of computing devices, wherein generating the prompt comprises: determining output capabilities for plurality of computing devices based on the environment data; generating a representation of performance capabilities of the plurality of computing devices; and generating the prompt based on the representation of performance capabilities of the plurality of computing devices and the input data; processing, by the computing system, the prompt with a generative model to generate a model-generated output and output device instructions based on the data descriptive of the query and the device information, wherein processing the prompt with the generative model to generate the model-generated output and output device instructions comprises: generating a response to the query based on processing the data descriptive of the query with the generative model; and generating, with the generative model and based on the representation and the response, a model-generated output to be provided with a particular computing device of the plurality of computing devices and output device instructions based on the device information; wherein the output device instructions comprise instructions to provide to provide the model-generated output with a particular computing device of the plurality of computing devices; and transmitting, by the computing system, the model-generated output to the particular computing device based on the output device instructions. ### Claim 12 (depends on 11) The method of claim 11, wherein the plurality of different output components are associated with a plurality of respective output capabilities associated with the plurality of computing devices, and wherein each of the plurality of respective output capabilities is descriptive of an output type and quality available via a respective computing device. ### Claim 13 (depends on 11) The method of claim 11, wherein the output device instructions comprise an application programming interface call to transmit the model-generated output to the particular computing device. ### Claim 14 (depends on 11) The method of claim 11, wherein the plurality of different output components comprise a speaker associated with a first device and a visual display associated with a second device. ### Claim 15 (depends on 11) The method of claim 11, further comprising: determining, by the computing system, the particular output component is associated with an intent of the query; and wherein the prompt is generated based on the particular output component being associated with the intent of the query. ### Claim 16 (depends on 11) The method of claim 11, further comprising: determining, by the computing system and based on the environment data, an output hierarchy based on specification information for the plurality of different output components; and wherein the prompt is generated based on the output hierarchy and the query. ### Claim 17 One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising: obtaining environment data, wherein the environment data is descriptive of a plurality of computing devices within an environment associated with a particular user; processing the environment data to determine a plurality of respective input capabilities and a plurality of respective output capabilities associated with the plurality of computing devices, wherein the plurality of respective input capabilities are associated with candidate input types associated with the plurality of computing devices, and wherein the plurality of respective output capabilities are associated with candidate output types associated with the plurality of computing devices; generating a plurality of respective interfaces for the plurality of computing devices based on the plurality of respective input capabilities and the plurality of respective output capabilities, wherein the plurality of respective interfaces are specialized for the plurality of computing devices based on the plurality of computing devices based on the plurality of respective input capabilities and the plurality of respective output capabilities; providing the plurality of respective interfaces to the plurality of computing devices; obtaining, with one or more of the plurality of respective interfaces, query from a first computing device of the plurality of computing devices; processing the query and the environment data to generate a prompt, wherein generating the prompt comprises: obtaining the determined plurality of respective input capabilities and the plurality of respective output capabilities; generating a representation of performance capabilities of the plurality of computing devices; and generating the prompt based on the representation of performance capabilities of the plurality of computing devices and the query; and processing the prompt with a generative model to generate a model-generated output and output device instructions, wherein processing the prompt with the generative model to generate the model-generated output and output device instructions comprises: generating a response to the query based on processing the query with the generative model; and generating, with the generative model and based on the plurality of respective input capabilities, the plurality of respective output capabilities, and the response, a model-generated output to be provided with a second computing device of the plurality of computing devices and output device instructions, wherein the output device instructions comprise instructions to provide the model-generated output with the second computing device of the plurality of computing devices; and transmitting the model-generated output to the second computing device based on the output device instructions. ### Claim 18 (depends on 17) The one or more non-transitory computer-readable media of claim 17, wherein the plurality of respective interfaces comprise a plurality of device indicators that indicate the plurality of computing devices within the environment associated with the particular user, and wherein the plurality of computing devices are configured as a user-specific device ecosystem that are communicatively connected for receiving inputs and providing outputs. ### Claim 19 (depends on 17) The one or more non-transitory computer-readable media of claim 17, wherein each of the plurality of respective interfaces is configured to receive particular input types and provide particular output types based on respective input capabilities and respective output capabilities for the particular computing device of the plurality of computing devices. ### Claim 20 (depends on 17) The one or more non-transitory computer-readable media of claim 17, wherein the operations further comprise: obtaining a user input via a first interface of the first computing device of the plurality of computing devices; processing the user input with a search engine to determine a plurality of search results; processing the plurality of search results with the generative model to generate a model output; and providing the model output for display via a second interface of the second computing device of the plurality of computing devices.