# US 10,769,189 B2 · Computer speech recognition and semantic understanding from activity patterns

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,189 B2
- Document ID: US10769189B2
- Status: Granted
- Kind: B2
- Application number: 15/087,224
- Filed: 2016-03-31
- Published: 2017-05-18
- Granted: 2020-09-08
- Earliest claimed: 2015-11-13
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Dikla Dotan-Cohen; Shira Weinberg
- Web page: https://patentgrep.com/patent/US10769189B2

## CPC

- G06F 16/335: Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536)
- G06F 16/957: Browsing optimisation, e.g. caching or content distillation
- G06F 40/30: Semantic analysis
- G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202)
- G06Q 10/101: Collaborative creation, e.g. joint development of products or services
- H04W 4/02: Services making use of location information
- H04W 4/021: Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
- H04W 4/029: Location-based management or tracking services
- H04W 4/18: Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
- H04W 4/023: {using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds}
- H04W 4/027: {using movement velocity, acceleration information}
- H04W 4/08: User group management
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- Y02D 30/70: in wireless communication networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

A user activity pattern may be ascertained using signal data from a set of computing devices. The activity pattern may be used to infer user intent with regards to a user interaction with a computing device or to predict a likely future action by the user. In one implementation, a set of computing devices is monitored to detect user activities using sensors associated with the computing devices. Activity features associated with the detected user activities are determined and used to identify an activity pattern based on a plurality of user activities having similar features. Examples of user activity patterns may include patterns based on time, location, content, or other context. The inferred user intent or predicted future actions may be used to facilitate understanding user speech or determining a semantic understanding of the user.

## Claims

### Claim 1

A computerized system comprising:
one or more sensors configured to provide sensor data comprising at least acoustic information;
one or more processors; and
computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, implement a method of recognizing speech, the method comprising:
a) receiving from the sensor data, information associated with an utterance by a user, the utterance corresponding to an action desired by the user;
b) determining a set of words corresponding to the utterance;
c) accessing an user activity pattern for the user;
d) determining a probable future activity event based on the activity pattern and a context determined at least in part from the set of words;
e) determining from the set of words, a subset of words based on the determined probable future activity event;
f) determining an action corresponding to the subset of words;
and
g) generating a control signal to initiate the action.

### Claim 2 (depends on 1)

The computerized system of claim 1, wherein the subset of words is determined to be consistent with the determined probable future activity event.

### Claim 3 (depends on 1)

The computerized system of claim 1, wherein the activity event comprises one or more of browsing to a website, launching an application, initiating a communication, performing a search query, setting a reminder, or scheduling an event on a calendar.

### Claim 4 (depends on 1)

The computerized system of claim 1, wherein the set of words comprises a plurality of alternative subsets of words, each subset corresponding to an alternative possible user action.

### Claim 5 (depends on 1)

The computerized system of claim 1, wherein the user is provided an indication of the determined subset of words.

### Claim 6 (depends on 1)

The computerized system of claim 1, further comprising determining the action based in part on the probable future activity event.

### Claim 7 (depends on 1)

The computerized system of claim 1, wherein the sensor data further comprises one or more of location-related data, time-related data, usage-related data, or data about the user, and wherein the context is further determined based on the location-related data, the time-related data, the usage-related data, or the data about the user.

### Claim 8 (depends on 1)

The computerized system of claim 1, wherein the utterance comprises a spoken request, command, or query to be performed via a user computing device.

### Claim 9

A method for performing disambiguation to determine a user intent for a user, comprising:
receiving an indication of a user interaction with a user device, the user interaction associated with an activity performed by the user device, the indication based at least in part on sensor data from one or more sensors associated with the user device;
determining a set of possible user intents corresponding to the user interaction;
accessing an activity pattern that is learned for the user;
determining a future activity event based on the activity pattern that is learned for the user;
determining from the set of possible user intents, a particular user intent based on the future activity event; and
performing the activity associated with the user interaction based on the determined particular user intent.

### Claim 10 (depends on 9)

The method of claim 9, wherein the activity associated with the user interaction comprises a search query, and further comprising filtering search results to be provided to the user based on the selected user intent.

### Claim 11 (depends on 9)

The method of claim 9, wherein determining a particular user intent comprises determining from the set of intents, the intent that is most consistent with the determined probable future activity event.

### Claim 12 (depends on 9)

The method of claim 9, wherein the sensor data comprises acoustic information corresponding to an utterance spoken by the user.

### Claim 13 (depends on 9)

The method of claim 9, wherein the future activity event is further determined based on a context associated with the user interaction.

### Claim 14 (depends on 13)

The method of claim 13, wherein the context is determined based on the sensor data.

### Claim 15

A computerized system comprising:
one or more sensors configured to provide sensor data, each sensor associated with a user device in a set of user devices associated with a user, at least one of the one or more sensors comprising an acoustic sensor configured to convert speech into acoustic information;
one or more processors; and
computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, implement a method of recognizing speech, the method comprising:
a) accessing an user activity pattern that is learned for a user;
b) predicting a future activity event based on the activity pattern that is learned for the user and a context determined at least in part from the sensor data;
c) receiving, from the acoustic sensor, acoustic information corresponding to a spoken interaction with a particular user device by the user;
d) based on an analysis of the acoustic information, determining a plurality of word sequences corresponding to the spoken interaction;
e) determining from the plurality of word sequences, a word sequence based on the predicted future activity event; and
f) providing the determined word sequence as recognized speech corresponding to the spoken interaction.

### Claim 16 (depends on 15)

The computerized system of claim 15, further causing to be performed an action corresponding to the spoken interaction.

### Claim 17 (depends on 15)

The computerized system of claim 15, determining the word sequence based on the predicted future activity event comprises determining the word sequence that is most consistent with the predicted future activity event.

### Claim 18 (depends on 15)

The computerized system of claim 15, wherein the predicted future activity event comprises a sequence of user actions.

### Claim 19 (depends on 15)

The computerized system of claim 15, wherein the sensor data for predicting a future activity event is provided from a sensor associated with a first user device in the set of user devices that is different from the particular user device that receives acoustic information.

### Claim 20 (depends on 15)

The computerized system of claim 15, wherein the sensor data includes one or more of location-related data, time-related data, usage-related data, or data about the user, and wherein the context is further determined based on the location-related data, the time-related data, the usage-related data, or the data about the user.