# US 2014/0195968 A1 · Inferring and acting on user intent > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0195968 A1 - Document ID: US20140195968A1 - Status: Published application - Kind: A1 - Application number: 13/737,622 - Filed: 2013-01-09 - Published: 2014-07-10 - Earliest claimed: 2013-01-09 - Assignees: Hewlett-Packard Development Company, L.P. - Applicants: Hewlett- Packard Development Company, L.P. - Inventors: Madhusudan Banavara - Web page: https://patentgrep.com/patent/US20140195968A1 ## CPC - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A method for inferring and acting on user intent includes receiving, by a computing device, a first input and a second input. The first input includes data associated with a first real world object and the second input includes selection by a user of an image representing a second real world object. A plurality of potential actions that relate to at least one of the first input and the second input are identified. The method further includes determining, from a plurality of potential actions, an action that is inferred by a relationship between the first real world object and the second real world object. The action inferred from the relationship between the first real world object and the second real world object is performed. A computing device for inferring and acting on user intent is also provided. ## Claims ### Claim 1 A method for inferring and acting on user intent comprising: receiving a first input by a computing device, the first input comprising data associated with a first real world object; receiving a second input by the computing device, the second input comprising a selection by a user of an image representing a second real world object; identifying a plurality of potential actions that relate to at least one of the first input and the second input; determining, from the plurality of potential actions, an action that is inferred by a relationship between the first real world object and the second real world object; and performing, with the computing device, the action inferred by the relationship between the first real world object and the second real world object. ### Claim 2 (depends on 1) The method of claim 1, in which the first input is at least one of data, voice, time, location, or sensor input associated with the first real world object. ### Claim 3 (depends on 1) The method of claim 1, in which data associated with the first real world object comprises an image of the first real world object. ### Claim 4 (depends on 1) The method of claim 1, in which the second input is a picture of the second real world object taken by the user with the computing device. ### Claim 5 (depends on 1) The method of claim 1, in which the selection by the user of the image representing the second real world object comprises selection of the image from a data base. ### Claim 6 (depends on 1) The method of claim 1, in which identifying the plurality of potential actions that relate to at least one of the first input and the second input comprises identifying actions that can be applied to the first real world object and actions that can be taken by the second real world object. ### Claim 7 (depends on 1) The method of claim 1, in which determining, from the plurality of potential actions, an action that is inferred by the relationship between the first real world object and the second real world object comprises determining which of the potential actions taken by the second real world object can be applied to the first real world object. ### Claim 8 (depends on 1) The method of claim 1, in which: the first real world object is a document; the second input comprises a picture of a printer taken by the user with the computing device; the action that is inferred by a relationship between the document and the printer is the printing of the document by the printer; and performing the action inferred by the relationship comprises printing the document on the printer. ### Claim 9 (depends on 8) The method of claim 8, in which taking the picture of the printer comprises taking a picture of a barcode affixed to the exterior of the printer. ### Claim 10 (depends on 1) The method of claim 1, further comprising analyzing the image to identify the second real world object in the image. ### Claim 11 (depends on 1) The method of claim 1, in which the computing device is a remote server configured to receive the first input, receive the second input from a mobile device, identify a plurality of potential actions, determine an action that is inferred and perform the action. ### Claim 12 (depends on 1) The method of claim 1, in which the computing device electronically connects to the second real world object and communicates with the second real world object to perform the action based on the relationship between the first input and the real world object. ### Claim 13 (depends on 1) The method of claim 1, in which identifying the plurality of potential actions, determining an action that is inferred by a relationship, and performing the action inferred by the relationship is executed without user involvement. ### Claim 14 (depends on 1) The method of claim 1, in which performing the action comprises the computing device sending control data to the second real world object to influence the state of the second real world object. ### Claim 15 (depends on 1) The method of claim 1, in which the first real world object is operated on by the second real world object. ### Claim 16 (depends on 1) The method of claim 1, further comprising prompting the user for confirmation of the action prior to performing the action. ### Claim 17 (depends on 1) The method of claim 1, in which an image of the first real world object and the image of the second real world object are displayed together on a screen of the computing device, the method further comprising the user gesturing from the image of the first real world object to the image of the second real world object to define a relationship between the first real world object and second real world object. ### Claim 18 A computing device for inferring and acting on user intent comprises: an input component to receive a first input and a second input, wherein the first input comprises data associated with a first real world object and the second input comprises a selection by a user of an image representing a second real world object; an input identification module to identify the first input and the second input; an inference module to identify a plurality of potential actions that relate to at least one of the first input and the second input and for determining from a plurality of potential actions, an action that is inferred by a relationship between the first real world object and the second real world object; an action module to perform the action inferred by the relationship between the first real world object and the second real world object; and a communication component to communicate the action to a second computing device. ### Claim 19 (depends on 18) The device of claim 18, in which: the first input comprises an image of a document viewed by the user; the second input comprises an image of a target printer; and the action comprises automatically and without further user action, identifying the target printer, connecting to the target printer, and printing the document on the target printer. ### Claim 20 A computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising: computer readable program code to receive a first input by a computing device, the first input comprising data associated with a first real world object; computer readable program code to receive a second input by the computing device, the second input comprising a selection by a user of an image representing a second real world object; computer readable program code to identify a plurality of potential actions that relate to at least one of the first input and the second input; computer readable program code to determine, from the plurality of potential actions, an action that is inferred by a relationship between the first real world object and the second real world object; and computer readable program code to perform, with the computing device, the action inferred by the relationship between the first real world object and the second real world object.