# US 10,121,474 B2 · Contextual note taking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,121,474 B2
- Document ID: US10121474B2
- Status: Granted
- Kind: B2
- Application number: 15/062,797
- Filed: 2016-03-07
- Published: 2017-08-17
- Granted: 2018-11-06
- Earliest claimed: 2016-02-17
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Jamie Yu; Roopesh Rajamani; Om Krishna; Priya Vaidyanathan
- Web page: https://patentgrep.com/patent/US10121474B2

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 40/106: Display of layout of documents; Previewing
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06Q 10/00: Administration; Management
- G06Q 10/10: Office automation; Time management
- G09B 5/00: Electrically-operated educational appliances (working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F)
- G09B 5/06: with both visual and audible presentation of the material to be studied
- G09B 5/08: providing for individual presentation of information to a plurality of student stations
- G10L 2015/221: {Announcement of recognition results}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Contextual note taking is described. A note taking assistant can receive an indication of a specific presentation session. This indication can be used by the note taking assistant to access information or content related to the session. The note taking assistant can receive specific presentation session content, which includes identifiable context images. Identifiable context images are meant to define an individual page, an individual slide, or other atomic unit in the presentation. The note taking assistant operates by receiving a navigation message, changing the current assistant context image to a current presenter context image based on the navigation message; receiving a speech-to-text message comprising a unit of text; displaying the current presenter context image, and displaying the unit of text associated with the current presenter context image; and storing the unit of text associated with the current presenter context image.

## Claims

### Claim 1

A method facilitating contextual note taking, the method comprising:
receiving, at an end user device, an indication of a specific presentation session;
receiving, at the end user device, specific presentation session content, the specific presentation session content comprising identifiable context images;
receiving, from a presenter device, a navigation message via a communication interface of the end user device, wherein the navigation message comprises a location identifier, the location identifier being used to identify a current presenter context image;
responsive to receiving the navigation message, changing, at the end user device, a current assistant context image to the current presenter context image, the current assistant context image corresponding to a context image to which any units of text being received are assigned and the current presenter context image corresponding to a context image a presenter is currently on;
receiving, at the end user device, a speech-to-text message comprising a unit of text associated with the current presenter context image from the presenter device;
displaying, at the end user device, the unit of text associated with the current presenter context image; and
storing the unit of text associated with the current presenter context image regardless of a context image that an end user is currently viewing at the end user device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving an indication of a target language;
as the unit of text associated with the current presenter context image is received, sending a request to a translation service, the request including the target language and the unit of text associated with the current presenter context image;
receiving a translated unit of text associated with the current presenter context image;
displaying the translated unit of text associated with the current presenter context image; and
storing the translated unit of text associated with the current presenter context image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
populating a note taking form with the specific presentation session content, including a plurality of the identifiable context images.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
identifying the current presenter context image in the note taking form, wherein the unit of text is inserted and displayed in the note taking form associated with the current presenter context image.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
receiving and displaying user input to the note taking form in a region associated with a different context image than that having the unit of text being inserted and displayed.

### Claim 6 (depends on 3)

The method of claim 3, wherein populating the note taking form with the specific presentation session content comprises:
communicating with a note taking application to input and store the specific presentation session content.

### Claim 7 (depends on 1)

The method of claim 1, wherein the receiving specific presentation session content comprises:
receiving a unified resource locator (URL) message; and
obtaining the specific presentation specific content at a URL location, identified by the URL message.

### Claim 8

A system comprising:
a processing system,
a display,
a communication interface,
one or more computer readable storage media, and
a contextual note taking program stored on the one or more computer readable storage media that, when executed by the processing system, directs the processing system to:
in response to receiving, from a presenter device, a navigation message via the communication interface, change a current assistant context image of a specific presentation session content to a current presenter context image of the specific presentation session content, the current assistant context image corresponding to a context image to which any units of text being received are assigned and the current presenter context image corresponding to a context image a presenter is currently on, and wherein the navigation message comprises a location identifier, the location identifier being used to identify the current presenter context image;
in response to receiving, from the presenter device, a speech-to-text message comprising a unit of text, at least display the unit of text associated with the current presenter context image and store the unit of text associated with the current presenter context image regardless of a context image that an end user is currently viewing in the display of the system.

### Claim 9 (depends on 8)

The system of claim 8, further comprising:
a note stored on the one or more computer readable storage media,
wherein the contextual note taking program further directs the processing system to:
populate a note taking form in the note with the specific presentation session content, including a plurality of the identifiable context images.

### Claim 10 (depends on 9)

The system of claim 9, wherein the contextual note taking program further directs the processing system to:
identify the current presenter context image in the note taking form and insert the unit of text in the note taking form associated with the current presenter context image.

### Claim 11 (depends on 10)

The system of claim 10, wherein the contextual note taking program further directs the processing system to:
in response to receiving user input to the note taking form in a region associated with a different context image than that having the unit of text being inserted, display the user input.

### Claim 12 (depends on 8)

The system of claim 8, wherein the contextual note taking program further directs the processing system to:
communicate with a note taking application to input and store the specific presentation session content.

### Claim 13 (depends on 12)

The system of claim 12, wherein the contextual note taking program further directs the processing system to:
communicate with the note taking application to input the unit of text associated with the current presenter context image.

### Claim 14 (depends on 8)

The system of claim 8, wherein the contextual note taking program further directs the processing system to:
in response to receiving a unified resource locator (URL) message for an indicated specific presentation session, obtain the specific presentation specific content at a URL location, identified by the URL message.

### Claim 15 (depends on 8)

The system of claim 8, wherein the contextual note taking program further directs the processing system to:
provide translation options for target languages;
receive an indication of a target language;
as the unit of text associated with the current presenter context image is received, send a request to a translation service, the request including the target language and the unit of text associated with the current presenter context image;
receive a translated unit of text associated with the current presenter context image;
display the translated unit of text associated with the current presenter context image; and
store the translated unit of text associated with the current presenter context image.

### Claim 16

One or more computer readable storage devices having a contextual note taking program stored thereon that when executed by a processing system, directs the processing system to:
in response to receiving, from a presenter device, a navigation message via a communication interface, change a current assistant context image of a specific presentation session content to a current presenter context image of the specific presentation session content at an end user device, the current assistant context image corresponding to a context image to which any units of text being received are assigned and the current presenter context image corresponding to a context image a presenter is currently on, and wherein the navigation message comprises a location identifier, the location identifier being used to identify the current presenter context image;
in response to receiving, from the presenter device, a speech-to-text message comprising a unit of text, at least display the unit of text associated with the current presenter context image at the end user device and store the unit of text associated with the current presenter context image regardless of a context image that an end user is currently viewing at the end user device.

### Claim 17 (depends on 16)

The media of claim 16, wherein the contextual note taking program further directs the processing system to populate a note taking form in a note with the specific presentation session content, including a plurality of the identifiable context images.

### Claim 18 (depends on 17)

The media of claim 17, wherein the contextual note taking program further directs the processing system to:
identify the current presenter context image in the note taking form and insert the unit of text in the note taking form associated with the current presenter context image.

### Claim 19 (depends on 18)

The media of claim 18, wherein the contextual note taking program further directs the processing system to:
in response to receiving user input to the note taking form in a region associated with a different context image than that having the unit of text being inserted, display the user input.

### Claim 20 (depends on 16)

The media of claim 16, wherein the contextual note taking program further directs the processing system to:
communicate with a note taking application to input and store the specific presentation session content.