# US 2019/0317643 A1 · Intelligent productivity monitoring with a digital assistant

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0317643 A1
- Document ID: US20190317643A1
- Status: Published application
- Kind: A1
- Application number: 16/394,584
- Filed: 2019-04-25
- Published: 2019-10-17
- Earliest claimed: 2016-09-16
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Abhishek BHARDWAJ; Aditya VEGESNA; Anuja Chetan SHAH; Arnold Paul PEREIRA; Chaitanya Krishna DONTHINI; Gautam DAMBEKODI NAVINCHAND; Sachit MUCKADEN; Shefali VOHRA; Vignesh SHENOY; Yuvek Lokesh MEHTA
- Web page: https://patentgrep.com/patent/US20190317643A1

## CPC

- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A digital assistant supported across devices such as smartphones, tablets, personal computers (PCs), wearable computing devices, game consoles, and the like is configured to interact with a productivity system so that various user experiences, content, or features that enhance a user's productivity can be integrated with the digital assistant and rendered as a native digital assistant user experience. The digital assistant is configured to actively participate with the user to set and achieve productivity goals for example, by determining the user's intent, maintaining awareness of context, performing tasks and actions, providing productivity status and notifications, and interacting with the user to help the user stay productive and avoid distractions.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

One or more computer-readable memory devices storing instructions which, when executed by one or more processors disposed in a computing device of a set of computing devices, cause the computing device of the set of computing devices, cause the computing device of the set of computing devices to:
classify applications accessed by a user from the one or more computing devices of the set of computing devices wherein the classifying is by application type;
receive monitoring data from one or more computing devices of the set of computing devices associated with the user in which the monitoring data describes time-based usage of applications by the user;
aggregate the time-based usage for applications that share a common application type;
determine a level of productivity of the user using the monitoring data, in which the determination compares the time-based usage against a productivity goal associated with the user;
provide at least a portion of productivity feedback for displaying on the one or more computing devices of the set of computing devices associated with the user, wherein the productivity feedback includes the productivity goal and one or more graphs with multiple representations, and each representation in the one or more graphs is configured with the aggregated time-based usage for each respective application type.

### Claim 22 (depends on 21)

The one or more computer-readable memory devices of claim 21 in which the executed instructions further cause the computing device of the set of computing devices to correlate user behaviors with productivity level.

### Claim 23 (depends on 21)

The one or more computer-readable memory devices of claim 21 in which the executed instructions further cause the computing device of the set of computing devices to receive user behavior data describing user behaviors with the set of computing devices and contextual data describing context associated with the user, and in which the computing device of the set of computing devices employs the user behavior data or the contextual data to assess a risk that the user misses the productivity goal or determine automated actions to be performed by the digital assistant on the one or more computing devices to mitigate against the assessed risk.

### Claim 24 (depends on 23)

The one or more computer-readable memory devices of claim 23 in which the automated actions increase the level of productivity by reducing occurrences of non-automated actions performed by the user on the one or more computing devices.

### Claim 25 (depends on 21)

The one or more computer-readable memory devices of claim 21 in which the executed instructions cause the computing device of the set of computing devices to classify applications or websites accessed by the user from the one or more computing devices wherein the classification is by type or category.

### Claim 26 (depends on 21)

The one or more computer-readable memory devices of claim 21 in which the executed instructions cause the computing device of the set of computing devices to receive an interaction from the user using one of voice, physical interaction, or gesture.

### Claim 27

A method for managing use of a computing device by a user, comprising:
classifying applications accessed by the user on the computing device according to application type;
monitoring time-based use by the user of applications on the computing device, in which the monitoring is performed according to application type;
aggregating the time-based usage for applications that share a common application type;
configuring a digital assistant to interact with the user through a natural language interface on the computing device;
receiving a voice input from the user to the digital assistant using the natural language interface; and
in response to the voice input, communicating the aggregated time-based application usage to the user.

### Claim 28 (depends on 27)

The method of claim 27 in which the communicating comprises displaying, on the computing device, one or more graphs with multiple representations, and each representation in the one or more graphs is configured with the aggregated time-based usage for each respective application type.

### Claim 29 (depends on 27)

The method of claim 27 in which the communicating comprises the digital assistant speaking to the user through the natural language interface to provide information about the aggregated time-based usage for one or more of the application types.

### Claim 30 (depends on 27)

The method of claim 27 further including monitoring the user's time usage of applications or websites and wherein the monitored time usage is employed to determine whether the user is meeting one or more predetermined criteria.

### Claim 31 (depends on 30)

The method of claim 30 further including monitoring user behaviors or user interactions with the applications on the computing device in which the monitored user behaviors or user interactions are further based on data comprising one or more of time/date, location of the user or computing device, language, schedule, applications installed on the computing device, user preferences, user behaviors, user activities, stored contacts, call history, messaging history, browsing history, application usage history, computing device type, computing device capabilities, or communication network type.

### Claim 32 (depends on 27)

The method of claim 27 further including utilizing the digital assistant to provide productivity feedback to the user through a user interface (UI).

### Claim 33 (depends on 32)

The method of claim 32 in which the productivity feedback is provided to the user through the UI in one or more of graphical form or auditory form.

### Claim 34 (depends on 33)

The method of claim 33 in which the monitored user behaviors or user interactions are maintained by using sensor data collected by sensors on the computing device, the sensors including one or more of camera, accelerometer, location-awareness component, thermometer, altimeter, heart rate sensor, barometer, microphone, or proximity sensor.

### Claim 35

A computing device, comprising:
one or more processors;
a voice-based user interface (UI) configured to interact with a user of the computing device; and
a non-transitory hardware-based memory device storing computer-readable instructions which, when executed by the one or more processors, cause the computing device to
expose a digital assistant with which the user sets criteria, by voice through the voice-based UI, to trigger an alert on the computing device based on a duration of time the user interacts with applications, wherein the criteria are set according to application type,
monitor user interactions with the applications on the computing device by time,
determine whether the alert criteria are met based on the monitored user interactions, and
alert the user responsively to the determination.

### Claim 36 (depends on 35)

The computing device of claim 35 further comprising a graphical user interface (GUI), and in which the alert comprises a notification that is raised on the GUI.

### Claim 37 (depends on 35)

The computing device of claim 35 in which the alert comprises the digital assistant informing the user, by voice, about the duration of time with which the user interacts with one or more of the applications.

### Claim 38 (depends on 35)

The computing device of claim 35 in which the executed instructions further cause the computing device to operate the digital assistant to perform an action that automates tasks on behalf of the user to thereby enable the user to meet the criteria, in which the action comprises one or more of sharing contact information, reading an email to identify tasks contained therein, adding a task to a task list, scheduling a meeting or appointment, interacting with a user's calendar, making a telephone call, sending a message, operating a device, making a reservation, providing a reminder to the user, making a purchase, suggesting a workout, playing music, taking notes, providing information to the user or another party, answering a question from the user or another party, setting an alarm or reminder, checking social media for updates, visiting a website, interacting with a search service, sharing or showing files, sending a link to a website, or sending a link to a resource.

### Claim 39 (depends on 35)

The computing device of claim 35 in which the executed instructions further cause the computing device to receive a notification from a remote service describing the user's progress towards a user-defined goal.

### Claim 40 (depends on 39)

The computing device of claim 39 in which the user-defined goal comprises a productivity goal.