US 2026/0228430 A1Published application

Near real-time language model interaction

AssigneeMicrosoft Technology Licensing, LLC

InventorsMuiris WOULFE, Poornima MUTHUKUMAR, Emre ATASEVEN, Ata Taner AYDEMIR, Ievgen POLYVANYI

Life of this patent

Estimating the term…

Loading patent…

Abstract

Solutions disclosed herein provide for near real-time language model interaction. Efficient management of the historical state reduces token counts in prompts, enabling use of (near) real-time invocation of language models. Examples segment a transcript (or other streaming representation) of a conversation into an earlier portion for summarization and a later portion. The summary and later portion are included in a prompt for the language model, along with an instruction to return an update to a real-time contextual analysis product (e.g., an updated agenda or other data product) based on prompt. Using the summary, or a sliding time window, the token count in the prompt is reduced, enabling a more rapid cadence of updates. The update is provided as an output (e.g., an agenda update or an updated summary of the conversation), and updated summary may be efficiently used as the summary in the following prompt (i.e., the next update iteration).

Claims20

Dependent claims

A system comprising:

a processor;
and
a computer-readable medium storing instructions that are operative upon execution by the processor to:
on a scheduled cadence:
receive current streamed data of a conversation;
identify a current end point of the current streamed data;
determine a current segmentation point in the current streamed data, wherein the current segmentation point segments the current streamed data into a pre-segmentation portion and a post-segmentation portion, wherein the pre-segmentation portion comprises the current streamed data prior to the current segmentation point and the post-segmentation portion comprises the current streamed data following the current segmentation point up through the current end point;
generate a language model prompt for a language model, wherein the language model prompt comprises:
a current summary of the pre-segmentation portion,
a copy of the post-segmentation portion,
an instruction to return an update to a contextual analysis product, and
an instruction to return a current updated summary of the conversation using both the current summary of the pre-segmentation portion and the copy of the post-segmentation portion;
and
update the contextual analysis product using the returned update to the contextual analysis product;
wherein the current segmentation point in the current streamed data comprises a prior end point of a prior streamed data from a prior iteration of updating the contextual analysis product, and the current summary of the pre-segmentation portion comprises a prior updated summary from the prior iteration of updating the contextual analysis product;
and
wherein in a following iteration of updating the contextual analysis product, the current end point of the current streamed data becomes a following segmentation point and the current updated summary becomes a following summary of the pre-segmentation portion.

14 more elements

A method comprising:

on a scheduled cadence:
receiving current streamed data of a conversation;
identifying a current end point of the current streamed data;
determining a current segmentation point in the current streamed data, wherein the current segmentation point segments the current streamed data into a pre-segmentation portion and a post-segmentation portion, wherein the pre-segmentation portion comprises the current streamed data prior to the current segmentation point and the post-segmentation portion comprises the current streamed data following the current segmentation point up through the current end point;
generating a language model prompt for a language model, wherein the language model prompt comprises:
a current summary of the pre-segmentation portion,
a copy of the post-segmentation portion,
an instruction to return an update to a contextual analysis product, and
an instruction to return a current updated summary of the conversation using both the current summary of the pre-segmentation portion and the copy of the post-segmentation portion;
and
updating the contextual analysis product using the returned update to the contextual analysis product;
wherein the current segmentation point in the current streamed data comprises a prior end point of a prior streamed data from a prior iteration of updating the contextual analysis product, and the current summary of the pre-segmentation portion comprises a prior updated summary from the prior iteration of updating the contextual analysis product;
and
wherein in a following iteration of updating the contextual analysis product, the current end point of the current streamed data becomes a following segmentation point and the current updated summary becomes a following summary of the pre-segmentation portion.

11 more elements

A computer storage device having computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising:

on a scheduled cadence:
receiving a current streamed data of a conversation;
identifying a current end point of the current streamed data;
determining a current segmentation point in the current streamed data, wherein the current segmentation point segments the current streamed data into a pre-segmentation portion and a post-segmentation portion, wherein the pre-segmentation portion comprises the current streamed data prior to the current segmentation point and the post-segmentation portion comprises the current streamed data following the current segmentation point up through the current end point;
generating a language model prompt for a language model, wherein the language model prompt comprises:
a current summary of the pre-segmentation portion,
a copy of the post-segmentation portion,
an instruction to return an update to a contextual analysis product, and
an instruction to return a current updated summary of the conversation using both the current summary of the pre-segmentation portion and the copy of the post-segmentation portion;
and
updating the contextual analysis product using the returned update to the contextual analysis product;
wherein the current segmentation point in the current streamed data comprises a prior end point of a prior streamed data from a prior iteration of updating the contextual analysis product, and the current summary of the pre-segmentation portion comprises a prior updated summary from the prior iteration of updating the contextual analysis product;
and
wherein in a following iteration of updating the contextual analysis product, the current end point of the current streamed data becomes a following segmentation point and the current updated summary becomes a following summary of the pre-segmentation portion.

11 more elements

Dependencies are inferred from the claim text.