# US 10,372,412 B2 · Force-based interactions with digital agents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,372,412 B2
- Document ID: US10372412B2
- Status: Granted
- Kind: B2
- Application number: 15/334,153
- Filed: 2016-10-25
- Published: 2018-04-26
- Granted: 2019-08-06
- Earliest claimed: 2016-10-25
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Christian Klein; Callil R. Capuozzo; Brendan D. Elliott; Christopher M. Barth; Otso Joona Casimir Tuomi; Shuo Shen
- Web page: https://patentgrep.com/patent/US10372412B2

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 3/0414: {using force sensing means to determine a position}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/0486: Drag-and-drop
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G10L 15/18: using natural language modelling
- G10L 15/183: using context dependencies, e.g. language models
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 40/35: Discourse or dialogue representation
- G06F 2203/04108: Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
- G06F 2203/04808: Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Embodiments relate to enabling force-based interactions with an intelligent personal assistant (IPA). A computing device capable of sensing the force exerted to input touch inputs is configured with a pressure-based filter that checks pressures of touch inputs to determine which touch inputs are to be diverted to the IPA or which are to be passed on to underlying user interface that is not related to the IPA. Touch inputs designated for the IPA based on their pressure characteristics can become part of the context of the IPA. Some IPA uses of the touch inputs include selecting graphic objects on the display, resolving exophoric phrases (e.g., “that”, “those”) as referring to such selected graphic objects, displaying transient user interface to provide information about (or actions for) the selected object, incorporating a selected object into the current context of the IPA, etc.

## Claims

### Claim 1

A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components thereof, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:
executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device and invoking respective operations for the recognized commands;
executing an application comprised of a graphical user interface, the executing including displaying the graphical user interface, the graphical user interface comprising graphic objects representing objects stored on the computing device, the application configured to respond to touch inputs that are directed to the graphical user interface;
receiving the touch inputs respectively comprised of locations, the touch inputs associated with the force components, respectively, each force component corresponding to a measure of force with which a corresponding touch input was inputted via the touch input device;
evaluating the force components against a force condition, wherein
(i) each time the force condition is determined to be satisfied by an evaluated force component, based on such determination, a corresponding touch input is passed to the IPA which uses the touch input to identify a target object according to a location of a graphic representation of the target object in the graphical user interface and according to the touch input corresponding to the evaluated force component that satisfied the force condition, and
(ii) each time the force condition is determined to not be satisfied by an evaluated force component, the touch input is provided to the application based on a location of the touch input and the application responds to the provided touch input.

### Claim 2 (depends on 1)

A method according to claim 1, the method further comprising:
determining that an exophoric phrase is present in a given command being handled by the IPA; and
based on the determination of the presence of the exophoric phrase in the command, executing the command with reference to the target object.

### Claim 3 (depends on 1)

A method according to claim 1, wherein each touch input determined to satisfy the force condition is not passed to the application, and each touch input determined to not satisfy the force condition is not used to identify a target object by the IPA.

### Claim 4 (depends on 1)

A method according to claim 1, further comprising, responding, by the IPA, to identification of the target object by the IPA displaying a user interface element, wherein content of the user interface element is selected by the IPA based on one or more features of the target object.

### Claim 5

A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components thereof, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:
executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device and invoking respective operations for the recognized commands;
executing an application comprised of a graphical user interface, the executing including displaying the graphical user interface, the graphical user interface comprising graphic objects representing objects stored on the computing device, the application configured to respond to touch inputs that are directed to the graphical user interface;
receiving the touch inputs respectively comprised of locations, the touch inputs associated with the force components, respectively, each force component corresponding to a measure of force with which a corresponding touch input was inputted via the touch input device:
evaluating the force components against a force condition, wherein
(i) each time the force condition is determined to be satisfied by an evaluated force component, based on such determination, a corresponding touch input is passed to the IPA which uses the touch input to identify a target object according to a location of a graphic representation of the target object in the graphical user interface and according to the touch input corresponding to the evaluated force component that satisfied the force condition, and
(ii) each time the force condition is determined to not be satisfied by an evaluated force component, the touch input is provided to the application and the application responds to the provided touch input, and
wherein a given touch input corresponds to two candidate objects and the method further comprises the IPA and/or the application determining which of the two candidate objects will be the target object.

### Claim 6

A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components thereof, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:
executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device and invoking respective operations for the recognized commands:
executing an application comprised of a graphical user interface, the executing including displaying the graphical user interface, the graphical user interface comprising graphic objects representing objects stored on the computing device, the application configured to respond to touch inputs that are directed to the graphical user interface:
receiving the touch inputs respectively comprised of locations, the touch inputs associated with the force components, respectively, each force component corresponding to a measure of force with which a corresponding touch input was inputted via the touch input device:
evaluating the force components against a force condition, wherein
(i) each time the force condition is determined to be satisfied by an evaluated force component, based on such determination, a corresponding touch input is passed to the IPA which uses the touch input to identify a target object according to a location of a graphic representation of the target object in the graphical user interface and according to the touch input corresponding to the evaluated force component that satisfied the force condition, and
(ii) each time the force condition is determined to not be satisfied by an evaluated force component, the touch input is provided to the application and the application responds to the provided touch input, and,
wherein the force condition is used to differentiate, among the touch inputs, between first touch inputs that are to be used to identify objects by the IPA and second touch inputs that are not to be used by the IPA to identify objects.

### Claim 7 (depends on 6)

A method according to claim 6, wherein the force condition comprises at least a force threshold value and the evaluating comprises comparing the force threshold value to the force components.

### Claim 8

A computing device comprising:
processing hardware;
a microphone;
a touch and force sensing display that senses touch inputs and provides respective measures that correspond to forces of the touch inputs; and
storage hardware storing instructions configured to cause the processing hardware to perform a process comprising:
executing an application comprising an object displayed by the computing device;
receiving a voice command inputted through the microphone;
receiving a first touch input corresponding to a first force measure;
determining whether the force measure satisfies a force condition;
executing first logic when the determining determines that the force measure satisfies the force condition, the first logic, when executed, associating the voice command with the object, and executing the voice command based on the association between the voice command and the object; and
executing second logic when the determining does not determine that the force measure satisfies the force condition, the second logic, when executed, providing the touch input to the application based on a location of the touch input.

### Claim 9 (depends on 8)

A computing device according to claim 8, wherein the associating the voice command with the object comprises identifying a graphic object displayed by the display according to the touch input, and identifying the object based on the identifying of the graphic object.

### Claim 10 (depends on 8)

A computing device according to claim 8, the process further comprising determining a display region defined by the touch input and identifying a plurality of objects according to the region.

### Claim 11 (depends on 8)

A computing device according to claim 8, wherein the voice command comprises an exophoric phrase and the word “here” or “there”, the process further comprising resolving the exophoric phrase as referring to the object and resolving the word “here” or “there” according to a feature of the touch input.

### Claim 12 (depends on 11)

A computing device according to claim 11, wherein the storage stores an intelligent personal assistant (IPA), and wherein the IPA receives a reference to the object and generates context according to the object, the context used by the IPA to interpret and/or execute voice commands being handled by the IPA, the IPA configured to invoke applications on the computing device to execute the voice commands.

### Claim 13 (depends on 12)

A computing device according to claim 12, wherein the IPA is configured to display a user interface element that comprises graphic indications of a plurality of actions or information items determined by the IPA to be relevant based on one or more properties of the object.

### Claim 14 (depends on 8)

A computing device according to claim 8, the process further comprising displaying a user interface on the display, wherein the touch input is coincident with the user interface on the display, wherein the user interface is configured to respond to touch inputs, and wherein whether the touch input is received by the user interface depends on whether the touch input satisfies a threshold force condition, the threshold force condition controlling whether touch inputs coincident with the user interface are received by the user interface.

### Claim 15

A method performed by a computing device comprising storage hardware, processing hardware, a display, and a touch input device that senses touch inputs and force components, the method performed by the processing hardware executing instructions stored in the storage hardware, the method comprising:
executing an intelligent personal assistant (IPA) on the computing device, the executing comprising using speech recognition to recognize commands inputted through a microphone of the computing device;
receiving a touch input comprised of a pressure component and a location relative to the display, the pressure component corresponding to a pressure value sensed by the touch input device while sensing input points of the touch input; and
determining whether or not to activate the IPA by determining whether the pressure component of the touch input satisfies a pressure condition, wherein the instructions are configured to:
respond to a determination that the pressure component of the touch input satisfies the pressure condition by activating the IPA and providing the touch input to the IPA, and
respond to a determination that the pressure component of the touch input does not satisfy the pressure condition by, instead of activating the IPA and providing the touch input to the IPA, providing the touch input to an application based on the application being associated with the location of the touch input.

### Claim 16 (depends on 15)

A method according to claim 15, wherein the IPA interprets the command by resolving an exophoric phrase in the command.

### Claim 17 (depends on 16)

A method according to claim 16, wherein the command specifies an action or query to be invoked by the IPA, and wherein the IPA invokes the action on the object or queries for information about the object.

### Claim 18 (depends on 15)

A method according to claim 15, wherein the object comprises a file system object, an object managed by an application installed on the computing device, or a link to a network resource.

### Claim 19 (depends on 15)

A method according to claim 15, wherein the activating the IPA comprises causing the IPA to begin analyzing audio input received through a microphone of the computing device.

### Claim 20 (depends on 15)

A method according to claim 15, wherein the method is performed by an operating system of the computing device.