# US 11,908,187 B2 · Systems, methods, and apparatus for providing image shortcuts for an assistant application > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,187 B2 - Document ID: US11908187B2 - Status: Granted - Kind: B2 - Application number: 18/117,798 - Filed: 2023-03-06 - Published: 2023-06-29 - Granted: 2024-02-20 - Earliest claimed: 2017-09-09 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Marcin Nowak-Przygodzki; Gökhan Bakir - Web page: https://patentgrep.com/patent/US11908187B2 ## CPC - G06V 20/20: in augmented reality scenes - G06F 3/005: {Input arrangements through a video camera} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 9/453: {Help systems} - G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information} - G06F 16/9032: Query formulation - H04N 23/63: by using electronic viewfinders - G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) - H04N 23/633: {for displaying additional information relating to control or operation of the camera} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Methods, apparatus, systems, and computer-readable media are set forth for generating and/or utilizing image shortcuts that cause one or more corresponding computer actions to be performed in response to determining that one or more features are present in image(s) from a camera of a computing device of a user (e.g., present in a real-time image feed from the camera). An image shortcut can be generated in response to user interface input, such as a spoken command. For example, the user interface input can direct the automated assistant to perform one or more actions in response to object(s) having certain feature(s) being present in a field of view of the camera. Subsequently, when the user directs their camera at object(s) having such feature(s), the assistant application can cause the action(s) to be automatically performed. For example, the assistant application can cause data to be presented and/or can control a remote device in accordance with the image shortcut. ## Claims ### Claim 1 A method implemented by one or more processors, the method comprising: receiving audio data in the form of a verbal command corresponding to a request to create an image shortcut setting; identifying, from the audio data, one or more computer actions to be performed, a condition for the image shortcut setting, and an object which is the subject of the condition; generating the image shortcut setting based on the one or more actions, the condition, and the object identified from the audio data, wherein the image shortcut setting is configured to cause the one or more computer actions to be performed in response to identifying the object from image data from a real-time image feed from a camera of a computing device when the condition is satisfied; and causing, according to the image shortcut setting, the one or more computer actions to be performed in response to identifying the object from the image data when the condition is satisfied, wherein identifying the object from the image data includes processing the image data using one or more image processing techniques. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more computer actions comprise transmitting a command to at least one peripheral device, wherein the command causes a state of the at least one peripheral device to be altered. ### Claim 3 (depends on 1) The method of claim 1, wherein identifying the object identifier corresponding to the object includes identifying multiple object identifiers corresponding to multiple different objects at which the camera of the computing device is directed, wherein the image shortcut setting is based on the multiple object identifiers. ### Claim 4 (depends on 1) The method of claim 1, further comprising: identifying, from the audio data, a context identifier for the request, wherein the image shortcut setting is generated further based on the context identifier. ### Claim 5 (depends on 4) The method of claim 4, wherein the context identifier identifies a location at which the real-time image feed is provided by the camera. ### Claim 6 (depends on 4) The method of claim 4, wherein the context identifier identifies at least one time or at least one location, and wherein causing, according to the image shortcut setting, the one or more computer actions to be performed is further in response to the image data being provided at a time that matches the at least one time, or at a location that matches the at least one location. ### Claim 7 (depends on 6) The method of claim 6, wherein the context identifier identifies the at least one location. ### Claim 8 A system, comprising: a camera; a microphone; a display device; a speaker; one or more processors in communication with the camera, the microphone, the display device, and the speaker; and memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to: receive, via the microphone, audio data in the form of a verbal command corresponding to a request to create an image shortcut setting; identify, from the audio data, one or more computer actions to be performed, a condition for the image shortcut setting, and an object which is the subject of the condition; generate the image shortcut setting based on the one or more actions, the condition, and the object identified from the audio data, wherein the image shortcut setting is configured to cause the one or more computer actions to be performed in response to identifying the object from image data from a real-time image feed from the camera when the condition is satisfied; and cause, according to the image shortcut setting, the one or more computer actions to be performed in response to identifying the object from the image data when the condition is satisfied, wherein identifying the object from the image data includes processing the image data using one or more image processing techniques. ### Claim 9 (depends on 8) The system of claim 8, wherein the one or more computer actions comprise transmitting a command to at least one peripheral device, wherein the command causes a state of the at least one peripheral device to be altered. ### Claim 10 (depends on 8) The system of claim 8, wherein in identifying the object identifier corresponding to the object one or more of the processors are to identify multiple object identifiers corresponding to multiple different objects at which the camera of the computing device is directed, wherein the image shortcut setting is based on the multiple object identifiers. ### Claim 11 (depends on 8) The system of claim 8, wherein the instructions, when executed by the one or more processors, further cause one or more of the processors to: identify, from the audio data, a context identifier for the request, wherein the image shortcut setting is generated further based on the context identifier. ### Claim 12 (depends on 11) The system of claim 11, wherein the context identifier identifies a location at which the real-time image feed is provided by the camera. ### Claim 13 (depends on 11) The system of claim 11, wherein the context identifier identifies at least one time or at least one location, and wherein causing, according to the image shortcut setting, the one or more computer actions to be performed is further in response to the image data being provided at a time that matches the at least one time, or at a location that matches the at least one location. ### Claim 14 (depends on 13) The system of claim 13, wherein the context identifier identifies the at least one location. ### Claim 15 At least one non-transitory computer readable medium configured to store instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: receiving audio data in the form of a verbal command corresponding to a request to create an image shortcut setting; identifying, from the audio data, one or more computer actions to be performed, a condition for the image shortcut setting, and an object which is the subject of the condition; generating the image shortcut setting based on the one or more actions, the condition, and the object identified from the audio data, wherein the image shortcut setting is configured to cause the one or more computer actions to be performed in response to identifying the object from image data from a real-time image feed from a camera of a computing device when the condition is satisfied; and causing, according to the image shortcut setting, the one or more computer actions to be performed in response to identifying the object from the image data when the condition is satisfied, wherein identifying the object from the image data includes processing the image data using one or more image processing techniques.