Response generation with multimodal context
AssigneeApple Inc.
InventorsAdrian ZUMBRUNNEN, Shiraz AKMAL, St John W. COLON, Brad K. HERMAN, James N. JONES, Johnnie B. MANZARI
Life of this patent
Estimating the term…
Abstract
Systems and processes for operating an intelligent automated assistant are provided. Example methods include displaying a representation of a current field-of-view of a camera and, while displaying the representation, generating responses to user inputs based on context information determined from the current field-of-view of the camera.
Claims21
1-140. (canceled)
An electronic device, comprising:
- a display generation component;
- one or more cameras;
- one or more processors;
- a memory;
- and
- one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
- displaying, via the display generation component, a representation of a feed of camera data from the one or more cameras;
- while displaying the representation of the feed of the camera data from the one or more cameras, detecting a user input;
- determining, based on the camera data from the one or more cameras, first visual context information;
- in response to detecting the user input, providing a representation of the user input and the first visual context information to a digital assistant agent;
- and
- generating, using the digital assistant agent, a response based on the representation of the user input and the first visual context information, wherein generating the response includes:
- selecting an application intent corresponding to the user input;
- and
- providing the application intent to an application, wherein providing the application intent to the application causes the application to execute a task.
A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with a display generation component and one or more cameras, cause the electronic device to:
- display, via the display generation component, a representation of a feed of camera data from the one or more cameras;
- while displaying the representation of the feed of the camera data from the one or more cameras, detect a user input;
- determine, based on the camera data from the one or more cameras, first visual context information;
- in response to detecting the user input, provide a representation of the user input and the first visual context information to a digital assistant agent;
- and
- generate, using the digital assistant agent, a response based on the representation of the user input and the first visual context information, wherein generating the response includes:
- selecting an application intent corresponding to the user input;
- and
- providing the application intent to an application, wherein providing the application intent to the application causes the application to execute a task.
A method, comprising:
- at an electronic device with a display generation component, one or more cameras, one or more processors, and memory:
- displaying, via the display generation component, a representation of a feed of camera data from the one or more cameras;
- while displaying the representation of the feed of the camera data from the one or more cameras, detecting a user input;
- determining, based on the camera data from the one or more cameras, first visual context information;
- in response to detecting the user input, providing a representation of the user input and the first visual context information to a digital assistant agent;
- and
- generating, using the digital assistant agent, a response based on the representation of the user input and the first visual context information, wherein generating the response includes:
- selecting an application intent corresponding to the user input;
- and
- providing the application intent to an application, wherein providing the application intent to the application causes the application to execute a task.
Dependencies are inferred from the claim text.