US 12,111,834 B1Granted

Ambient multi-device framework for agent companions

AssigneeGoogle Inc.

InventorsVictor Carbune, Arash Sadr, Matthew Sharifi

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods for generating and providing outputs in a multi-device system can include leveraging environment-based prompt generation and generative model response generation to provide dynamic response generation and display. The systems and methods can obtain input data associated with one or more computing devices within an environment, can obtain environment data descriptive of the plurality of computing devices within the environment, and can generate a prompt based on the input data and environment data. The prompt can be processed with a generative model to generate a model-generated output. The model-generated output can then be transmitted to a particular computing device of the plurality of computing devices.

Claims20

Dependent claims

A computing system for determining an output device for providing a query response, the system comprising:

one or more processors;
and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises a query associated with a particular user;
obtaining environment data, wherein the environment data is descriptive of a plurality of computing devices in an environment of the user, wherein the plurality of computing devices are associated with a plurality of different output components;
generating a prompt based on the input data and the environment data, wherein the prompt comprises data descriptive of the query and device information associated with at least a subset of the plurality of computing devices, wherein generating the prompt comprises:
determining output capabilities for plurality of computing devices based on the environment data;
generating a representation of performance capabilities of the plurality of computing devices;
and
generating the prompt based on the representation of performance capabilities of the plurality of computing devices and the input data;
processing the prompt with a generative model to generate a model-generated output based on the data descriptive of the query and the device information, wherein processing the prompt with the generative model to generate the model-generated output comprises:
generating a response to the query based on processing the data descriptive of the query with the generative model;
and
generating, with the generative model and based on the representation and the response, a model-generated output to be provided with a particular computing device of the plurality of computing devices based on the device information;
and
transmitting the model-generated output to the particular computing device.

13 more elements

A computer-implemented method, the method comprising:

obtaining, by a computing system comprising one or more processors, input data, wherein the input data comprises a query associated with a particular user;
obtaining, by the computing system, environment data, wherein the environment data is descriptive of a plurality of computing devices in an environment of the user, wherein the plurality of computing devices are associated with a plurality of different output components;
generating, by the computing system, a prompt based on the input data and the environment data, wherein the prompt comprises data descriptive of the query and device information associated with at least a subset of the plurality of computing devices, wherein generating the prompt comprises:
determining output capabilities for plurality of computing devices based on the environment data;
generating a representation of performance capabilities of the plurality of computing devices;
and
generating the prompt based on the representation of performance capabilities of the plurality of computing devices and the input data;
processing, by the computing system, the prompt with a generative model to generate a model-generated output and output device instructions based on the data descriptive of the query and the device information, wherein processing the prompt with the generative model to generate the model-generated output and output device instructions comprises:
generating a response to the query based on processing the data descriptive of the query with the generative model;
and
generating, with the generative model and based on the representation and the response, a model-generated output to be provided with a particular computing device of the plurality of computing devices and output device instructions based on the device information;
wherein the output device instructions comprise instructions to provide to provide the model-generated output with a particular computing device of the plurality of computing devices;
and
transmitting, by the computing system, the model-generated output to the particular computing device based on the output device instructions.

11 more elements

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

obtaining environment data, wherein the environment data is descriptive of a plurality of computing devices within an environment associated with a particular user;
processing the environment data to determine a plurality of respective input capabilities and a plurality of respective output capabilities associated with the plurality of computing devices, wherein the plurality of respective input capabilities are associated with candidate input types associated with the plurality of computing devices, and wherein the plurality of respective output capabilities are associated with candidate output types associated with the plurality of computing devices;
generating a plurality of respective interfaces for the plurality of computing devices based on the plurality of respective input capabilities and the plurality of respective output capabilities, wherein the plurality of respective interfaces are specialized for the plurality of computing devices based on the plurality of computing devices based on the plurality of respective input capabilities and the plurality of respective output capabilities;
providing the plurality of respective interfaces to the plurality of computing devices;
obtaining, with one or more of the plurality of respective interfaces, query from a first computing device of the plurality of computing devices;
processing the query and the environment data to generate a prompt, wherein generating the prompt comprises:
obtaining the determined plurality of respective input capabilities and the plurality of respective output capabilities;
generating a representation of performance capabilities of the plurality of computing devices;
and
generating the prompt based on the representation of performance capabilities of the plurality of computing devices and the query;
and
processing the prompt with a generative model to generate a model-generated output and output device instructions, wherein processing the prompt with the generative model to generate the model-generated output and output device instructions comprises:
generating a response to the query based on processing the query with the generative model;
and
generating, with the generative model and based on the plurality of respective input capabilities, the plurality of respective output capabilities, and the response, a model-generated output to be provided with a second computing device of the plurality of computing devices and output device instructions, wherein the output device instructions comprise instructions to provide the model-generated output with the second computing device of the plurality of computing devices;
and
transmitting the model-generated output to the second computing device based on the output device instructions.

14 more elements

Dependencies are inferred from the claim text.