# US 12,190,886 B2 · Selective inclusion of speech content in documents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,190,886 B2
- Document ID: US12190886B2
- Status: Granted
- Kind: B2
- Application number: 17/448,967
- Filed: 2021-09-27
- Published: 2023-08-24
- Granted: 2025-01-07
- Earliest claimed: 2021-09-27
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Sushain Pandit; Sarbajit K. Rakshit
- Web page: https://patentgrep.com/patent/US12190886B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- H04L 12/1831: {Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status}
- H04W 4/029: Location-based management or tracking services
- H04W 4/33: for indoor environments, e.g. buildings
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In an approach for enabling a user to visualize a transcript of a discussion via a head-mounted AR device and to selectively copy one or more parts of the transcript that is contextually relevant for inclusion in a new document file or a previously created document file, a processor captures audio of a spoken content of a first participant of a discussion via an AR device worn by a user. A processor analyzes the audio of the spoken content of the first participant. A processor converts the audio of the spoken content of the first participant to text to create a transcript. A processor creates a visualization of the transcript. A processor presents the visualization of the transcript to the user via the AR device. A processor enables the user to copy one or more parts of the transcript into a document file via a selection support.

## Claims

### Claim 1

A computer-implemented method comprising:
capturing, by one or more processors, audio of a spoken content of a first participant of a discussion via an augmented reality (AR) device worn by a user;
prior to capturing the audio of the spoken content of the first participant of the discussion via the AR device worn by the user, identifying, by the one or more processors, a plurality of participants, including the first participant, who are each physically present at a physical location to participate in the discussion, wherein each of the plurality of participants are identified based on the AR device that they are wearing;
analyzing, by the one or more processors, the audio of the spoken content of the first participant;
converting, by the one or more processors, the audio of the spoken content of the first participant to text to create a transcript;
creating, by the one or more processors, a visualization of the transcript;
presenting, by the one or more processors, the visualization of the transcript to the user via the AR device; and
enabling, by the one or more processors, the user to copy one or more parts of the transcript into a document file via a selection support,
wherein enabling the user to copy the one or more parts of the transcript into the document file via the selection support further comprises:
enabling, by the one or more processors, the user to point at a location of the first participant; and
selecting, by the one or more processors, one or more parts of the transcript associated with the first participant,
wherein the AR device provides a view of the physical location of the discussion and includes a computer-generated graphic including the visualization of the transcript overlaid on the view of the physical location.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
assigning, by the one or more processors, a first unique identification number to each participant of the plurality of participants;
authenticating, by the one or more processors, the AR device of each participant of the plurality of participants;
identifying, by the one or more processors, the physical location where the discussion is occurring;
assigning, by the one or more processors, a second unique identification number to the physical location where the discussion is occurring; and
creating, by the one or more processors, an indoor positioning system that represents the physical location where the discussion is occurring.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
prior to capturing the audio of the spoken content of the first participant of the discussion via the AR device worn by the user, enabling, by the one or more processors, the first participant to assign one or more permissions to each participant of the plurality of participants, wherein the one or more permissions determine whether each participant of the plurality of participants can copy the one or more parts of the transcript into the document file.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein analyzing the audio of the spoken content of the first participant further comprises:
identifying, by the one or more processors, the first participant by a tone of voice;
determining, by the one or more processors, a direction from which the audio of the spoken content of the first participant originated via a beam forming sensor on the AR device worn by the user and the indoor positioning system;
creating, by the one or more processors, a direction vector representing the direction from which the audio of the spoken content of the first participant originated;
determining, by the one or more processors, a time when the audio of the spoken content of the first participant originated;
creating, by the one or more processors, a time stamp representing the time when the audio of the spoken content of the first participant originated; and
preparing, by the one or more processors, a time scale to organize the audio of the spoken content of the first participant according to when the audio was captured.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the transcript includes an identifying factor of the first participant, the direction vector representing the direction from which the audio of the spoken content of the first participant originated, and the time stamp representing the time when the audio of the spoken content of the first participant originated.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the document file is a new document file or a previously created document file, and wherein the one or more parts of the transcript are placed in a user-defined position.

### Claim 7

A computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising:
program instructions to capture audio of a spoken content of a first participant of a discussion via an AR device worn by a user;
prior to capturing the audio of the spoken content of the first participant of the discussion via the AR device worn by the user, program instructions to identify, by the one or more processors, a plurality of participants, including the first participant, who are each physically present in a physical location to participate in the discussion, wherein each of the plurality of participants are identified based on the AR device that they are wearing;
program instructions to analyze the audio of the spoken content of the first participant;
program instructions to convert the audio of the spoken content of the first participant to text to create a transcript;
program instructions to create a visualization of the transcript;
program instructions to present the visualization of the transcript to the user via the AR device; and
program instructions to enable the user to copy one or more parts of the transcript into a document file via a selection support,
wherein enabling the user to copy the one or more parts of the transcript into the document file via the selection support further comprises:
program instructions to enable the user to point at a location of the first participant; and
program instructions to select one or more parts of the transcript associated with the first participant,
wherein the AR device provides a view of the physical location of the discussion and includes a computer-generated graphic including the visualization of the transcript overlaid on the view of the physical location.

### Claim 8 (depends on 7)

The computer program product of claim 7, further comprising:
program instructions to assign a first unique identification number to each participant of the plurality of participants;
program instructions to authenticate the AR device of each participant of the plurality of participants;
program instructions to identify the physical location where the discussion is occurring;
program instructions to assign a second unique identification number to the physical location where the discussion is occurring; and
program instructions to create an indoor positioning system that represents the physical location where the discussion is occurring.

### Claim 9 (depends on 7)

The computer program product of claim 7, further comprising:
prior to capturing the audio of the spoken content of the first participant of the discussion via the AR device worn by the user, program instructions to enable the first participant to assign one or more permissions to each participant of the plurality of participants, wherein the one or more permissions determine whether each participant of the plurality of participants can copy the one or more parts of the transcript into the document file.

### Claim 10 (depends on 7)

The computer program product of claim 7, wherein analyzing the audio of the spoken content of the first participant further comprises:
program instructions to identify the first participant by a tone of voice;
program instructions to determine a direction from which the audio of the spoken content of the first participant originated via a beam forming sensor on the AR device worn by the user and the indoor positioning system;
program instructions to create a direction vector representing the direction from which the audio of the spoken content of the first participant originated;
program instructions to determine a time when the audio of the spoken content of the first participant originated;
program instructions to create a time stamp representing the time when the audio of the spoken content of the first participant originated; and
program instructions to prepare a time scale to organize the audio of the spoken content of the first participant according to when the audio was captured.

### Claim 11 (depends on 10)

The computer program product of claim 10, wherein the transcript includes an identifying factor of the first participant, the direction vector representing the direction from which the audio of the spoken content of the first participant originated, and the time stamp representing the time when the audio of the spoken content of the first participant originated.

### Claim 12 (depends on 7)

The computer program product of claim 7, wherein the document file is a new document file or a previously created document file, and wherein the one or more parts of the transcript are placed in a user-defined position.

### Claim 13

A computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising:
program instructions to capture audio of a spoken content of a first participant of a discussion via an AR device worn by a user;
prior to capturing the audio of the spoken content of the first participant of the discussion via the AR device worn by the user, program instructions to identify, by the one or more processors, a plurality of participants, including the first participant, who are each physically present in a physical location to participate in the discussion, wherein each of the plurality of participants are identified based on the AR device that they are wearing;
program instructions to analyze the audio of the spoken content of the first participant;
program instructions to convert the audio of the spoken content of the first participant to text to create a transcript;
program instructions to create a visualization of the transcript;
program instructions to present the visualization of the transcript to the user via the AR device; and
program instructions to enable the user to copy one or more parts of the transcript into a document file via a selection support,
wherein enabling the user to copy the one or more parts of the transcript into the document file via the selection support further comprises:
program instructions to enable the user to point at a location of the first participant; and
program instructions to select one or more parts of the transcript associated with the first participant,
wherein the AR device provides a view of the physical location of the discussion and includes a computer-generated graphic including the visualization of the transcript overlaid on the view of the physical location.

### Claim 14 (depends on 13)

The computer system of claim 13, further comprising:
program instructions to assign a first unique identification number to each participant of the plurality of participants;
program instructions to authenticate the AR device of each participant of the plurality of participants;
program instructions to identify the physical location where the discussion is occurring;
program instructions to assign a second unique identification number to the physical location where the discussion is occurring; and
program instructions to create an indoor positioning system that represents the physical location where the discussion is occurring.

### Claim 15 (depends on 13)

The computer system of claim 13, further comprising:
prior to capturing the audio of the spoken content of the first participant of the discussion via the AR device worn by the user, program instructions to enable the first participant to assign one or more permissions to each participant of the plurality of participants, wherein the one or more permissions determine whether each participant of the plurality of participants can copy the one or more parts of the transcript into the document file.

### Claim 16 (depends on 13)

The computer system of claim 13, wherein analyzing the audio of the spoken content of the first participant further comprises:
program instructions to identify the first participant by a tone of voice;
program instructions to determine a direction from which the audio of the spoken content of the first participant originated via a beam forming sensor on the AR device worn by the user and the indoor positioning system;
program instructions to create a direction vector representing the direction from which the audio of the spoken content of the first participant originated;
program instructions to determine a time when the audio of the spoken content of the first participant originated;
program instructions to create a time stamp representing the time when the audio of the spoken content of the first participant originated; and
program instructions to prepare a time scale to organize the audio of the spoken content of the first participant according to when the audio was captured.

### Claim 17 (depends on 13)

The computer system of claim 13, wherein the document file is a new document file or a previously created document file, and wherein the one or more parts of the transcript are placed in a user-defined position.