# US 2025/0165728 A1 · Summarization with user interface (ui) stream control and actionable information extraction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165728 A1
- Document ID: US20250165728A1
- Status: Published application
- Kind: A1
- Application number: 18/862,092
- Filed: 2023-05-10
- Published: 2025-05-22
- Earliest claimed: 2022-05-10
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ruofei Du; Alex Olwal; Vikas Bahirwani; Boris Smus; Christopher Ross
- Web page: https://patentgrep.com/patent/US20250165728A1

## CPC

- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0485: Scrolling or panning
- G06F 16/345: {Summarisation for human users}
- G06F 40/166: Editing, e.g. inserting or deleting
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/47: Machine-assisted translation, e.g. using translation memory
- G06F 40/56: Natural language generation
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G02B 2027/014: {comprising information/image processing systems}
- G06F 40/216: using statistical methods
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G09B 19/06: Foreign languages (with audible presentation of material to be studied G09B5/04)
- G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Described techniques may be utilized to process transcribed text of a transcription stream using a summarization ML model to obtain a summary stream. In this way, a live conversation or other live speech may be provided to a user in real-time, using a display of a head-mounted device (HMD) or other suitable device. A user interface may be provided with a toggle or other stream selector for selecting either or both of the transcription stream or the summary stream at a given point in time. Actionable items may be identified within either or both of the transcription stream or the summary stream, and a scroll bar or other action selector may be provided in the user interface to execute corresponding actions.

## Claims

### Claim 1

A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:
receive a transcription stream including transcribed text;
input the transcription stream into a summarization machine learning (ML) model to obtain a summary stream including summarized text;
render, on a user interface (UI), a stream selector icon;
receive, via the stream selector icon, at least one selected stream of the transcription stream and the summary stream; and
cause a display, on the UI, of the at least one selected stream.

### Claim 2 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
render the stream selector icon as a toggle having a transcription position for selecting the transcription stream and a summary position for selecting the summary stream.

### Claim 3 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
render, using the UI and when the at least one selected stream includes the summary stream, a stream status indicator providing a status of the summary stream as currently processing the transcription stream using the summarization ML model.

### Claim 4 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
render, using the UI, a stream type indicator identifying the at least one selected stream as being either the transcription stream or the summary stream.

### Claim 5 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
store the transcription stream in a transcription buffer; and
store the summary stream in a summary buffer.

### Claim 6 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
render, using the UI, a scroll bar in conjunction with the at least one selected stream; and
retrieve an earlier portion of the at least one selected stream in response to movement of the scroll bar.

### Claim 7 (depends on 1)

The computer program product of claim 1, wherein the at least one selected stream includes both the transcription stream and the summary stream.

### Claim 8 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
translate the at least one selected stream into a selected language for display using the UI.

### Claim 9 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
cause the display of the at least one selected stream on a head-mounted device.

### Claim 10 (depends on 1)

The computer program product of claim 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
identify at least one entity within the at least one selected stream;
associate the at least one entity with at least one action;
render the at least one selected stream with the at least one entity being visually highlighted;
receive a selection of the at least one entity; and
invoke the at least one action.

### Claim 11

A device comprising:
at least one processor;
at least one display; and
at least one memory storing instructions, which, when executed by the at least one processor, cause the device to:
receive a transcription stream including transcribed text;
input the transcription stream into a summarization machine learning (ML) model to obtain a summary stream including summarized text;
render, on a graphical user interface (GUI), a stream selector icon;
receive, via the stream selector icon, at least one selected stream of the transcription stream and the summary stream; and
cause a display, on the GUI, of the at least one selected stream.

### Claim 12 (depends on 11)

The device of claim 11, wherein the instructions, when executed by the at least one processor, are further configured to cause the device to:
render the stream selector icon as a toggle having a transcription position for selecting the transcription stream and a summary position for selecting the summary stream.

### Claim 13 (depends on 11)

The device of claim 11, wherein the instructions, when executed by the at least one processor, are further configured to cause the device to:
render, using the GUI and when the at least one selected stream includes the summary stream, a stream status indicator providing a status of the summary stream as currently processing the transcription stream using the summarization ML model.

### Claim 14 (depends on 11)

The device of claim 11, wherein the instructions, when executed by the at least one processor, are further configured to cause the device to:
render, using the GUI, a stream type indicator identifying the at least one selected stream as being either the transcription stream or the summary stream.

### Claim 15 (depends on 11)

The device of claim 11, wherein the instructions, when executed by the at least one processor, are further configured to cause the device to:
store the transcription stream in a transcription buffer; and
store the summary stream in a summary buffer.

### Claim 16 (depends on 11)

The device of claim 11, wherein the instructions, when executed by the at least one processor, are further configured to cause the device to:
render, using the GUI, a scroll bar in conjunction with the at least one selected stream; and
retrieve an earlier portion of the at least one selected stream in response to movement of the scroll bar.

### Claim 17

A method comprising:
receiving a transcription stream including transcribed text;
inputting the transcription stream into a summarization machine learning (ML) model to obtain a summary stream including summarized text;
rendering, on a graphical user interface (GUI), a stream selector icon;
receiving, via the stream selector icon, at least one selected stream of the transcription stream and the summary stream; and
causing a display, on the GUI, of the at least one selected stream.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
rendering the stream selector icon as a toggle having a transcription position for selecting the transcription stream and a summary position for selecting the summary stream.

### Claim 19 (depends on 17)

The method of claim 17, further comprising:
storing the transcription stream in a transcription buffer; and
storing the summary stream in a summary buffer.

### Claim 20 (depends on 17)

The method of claim 17, further comprising:
rendering, using the GUI, a scroll bar in conjunction with the at least one selected stream; and
retrieving an earlier portion of the at least one selected stream in response to movement of the scroll bar.

### Claim 21

A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:
receive a transcription stream including transcribed text;
input the transcription stream into a summarization machine learning (ML) model to obtain a summary stream of summarized text;
identify, within the summarized text, summarized content that is associated with at least one action;
render, on a graphical user interface (GUI), the summarized text with the summarized content included therein with an action indicator relating the summarized content to the at least one action;
receive, via the GUI, a selection of the summarized content; and
execute the at least one action, in response to the selection.

### Claim 22 (depends on 21)

The computer program product of claim 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
identify the summarized content using an entity extraction ML model.

### Claim 23 (depends on 21)

The computer program product of claim 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
identify, within the transcribed text, transcribed content that is associated with a second action;
render, on the GUI, the transcribed text with the transcribed content included therein with a second action indicator relating the transcribed content to the second action;
receive, via the GUI, a second selection of the transcribed content; and
execute the second action, in response to the second selection.

### Claim 24 (depends on 21)

The computer program product of claim 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
store the transcription stream in a transcription buffer; and
store the summary stream in a summary buffer.

### Claim 25 (depends on 21)

The computer program product of claim 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
render, using the GUI, a scroll bar in conjunction with the summary stream; and
retrieve an earlier portion of the summary stream in response to movement of the scroll bar.

### Claim 26 (depends on 25)

The computer program product of claim 25, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
detect alignment of a scroll button of the scroll bar with the summarized content within the GUI; and
receive the selection using the scroll button, in conjunction with the alignment.

### Claim 27 (depends on 21)

The computer program product of claim 21, wherein the action indicator includes visual differentiation of the summarized content relative to remaining summarized content of the summarized text.

### Claim 28 (depends on 21)

The computer program product of claim 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
render an action selector icon using the GUI, the action selector icon identifying a type of the at least one action; and
receive the selection via the action selector icon.

### Claim 29 (depends on 21)

The computer program product of claim 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
render the GUI on a display of a head-mounted device.

### Claim 30 (depends on 21)

The computer program product of claim 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:
execute the at least one action including communicating with an application to cause the application to execute the at least one action.

### Claim at position 31

-40. (canceled)
