# US 2025/0156481 A1 · Ghosting for multimodal dialogs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156481 A1
- Document ID: US20250156481A1
- Status: Published application
- Kind: A1
- Application number: 18/509,617
- Filed: 2023-11-15
- Published: 2025-05-15
- Earliest claimed: 2023-11-15
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: . Akriti; Kumar SUBHAM; . Rishab; Srimukha PATURI; Manish GUPTA; Puneet AGRAWAL
- Web page: https://patentgrep.com/patent/US20250156481A1

## CPC

- G06F 16/3322: {using system suggestions (G06F16/3325 takes precedence)}
- G06F 16/90324: {using system suggestions}
- G06F 16/583: using metadata automatically derived from the content
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04L 51/04: Real-time or near real-time messaging, e.g. instant messaging [IM]
- H04L 51/216: Handling conversation history, e.g. grouping of messages in sessions or threads
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Systems and methods for generating autocomplete text using a language model are disclosed. An image and text-prefix may be entered at an input field of a search application. The image is processed to generate an image description. The image description and the text-prefix signals may be used as input at a language model to generate an autocomplete text by the language model. A contextual history may also be included as input to the language model. The autocomplete text is an output by the language model based on the input at the language model. The auto-complete text may be a next-word ghosting.

## Claims

### Claim 1

A computer-implemented method for generating autocomplete text, comprising:
receiving an image as input at an input field of an application;
generating an image description for the image;
receiving a text prefix as input at the input field;
generating a prompt requesting autocomplete text for the text prefix, the prompt including the image description and the text prefix;
providing the prompt as input to a language model;
receiving, from the language model in response to the prompt, the requested autocomplete text for the text prefix; and
surfacing the autocomplete text as a suffix to the text prefix, wherein the autocomplete text is displayed in a different format than the text prefix.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the image and the input text are received in an input field of a web browser.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein generating the image description for the image further comprises:
generating an image embedding for the image; and
comparing the image embedding to embeddings of prior images for which a description has been previously generated.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising, based on the comparison indicating a matching prior image, retrieving the image description of a matching prior image from a database.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, further comprising, based on the comparison indicating no matching prior image, receiving the image description from a generative artificial intelligence (AI) model.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the embeddings of prior images are stored in a database, and the method further comprises adding the embedding for the input image to the database.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the prompt further includes contextual history.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the contextual history includes an image description of a prior image in a current conversation.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein the contextual history includes a response from a prior turn in a current conversation.

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, wherein the contextual history includes browsing history.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving an additional character to the text prefix, thereby forming an updated text prefix;
determining that the additional character diverges from the autocomplete text;
based on the additional character diverging from the autocomplete text, generating a second prompt including the updated text prefix and the image description;
providing the second prompt as input to the language model; and
receiving, from the language model in response to the second prompt, updated autocomplete text for the text prefix.

### Claim 12

A system for generating autocomplete text, comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving an image as input at an input field of an application;
generating an image description for the image;
generating a prompt requesting at least one autocomplete text or potential input text, the prompt including the image description;
providing the prompt as input to a language model;
receiving, from the language model in response to the prompt, the requested at least one of autocomplete text or potential input text; and
surfacing the at least one of the autocomplete text or the potential input text.

### Claim 13 (depends on 12)

The system of claim 12, wherein a text prefix is not received, and generating the prompt includes:
accessing a zero-input prompt template, wherein the zero-input prompt template includes instructions for generating the potential text input without use of a text prefix; and
populating the zero-input prompt template with the image description to form the prompt.

### Claim 14 (depends on 13)

The system of claim 13, wherein the potential text input are generated suggested questions, surfaced adjacent the input field, that may be provided as input.

### Claim 15 (depends on 12)

The system of claim 12, wherein the operations further comprise receiving a text prefix in the input field, and generating the prompt includes:
accessing a non-zero-input prompt template, wherein the non-zero-input prompt template includes instructions for generating the autocomplete text; and
populating the non-zero-input prompt template with the image description and the text prefix to form the prompt.

### Claim 16 (depends on 15)

The system of claim 15, wherein the autocomplete text is surfaced as a suffix to the text prefix.

### Claim 17 (depends on 12)

The system of claim 12, wherein the image description is generated by providing the image to a generative AI model.

### Claim 18

A system for generating autocomplete text, comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving an image as input at an input field of an application;
generating an image description for the image;
generating a zero-input prompt requesting potential input text, the zero-input prompt including the image description;
providing the zero-input prompt as input to a language model;
receiving, from the language model in response to the zero-input prompt, the requested potential input text;
surfacing the potential input text;
receiving a text prefix in the input field;
generating a non-zero-input prompt requesting autocomplete text for the text prefix, the non-zero-input prompt including the image description and the text prefix;
providing the non-zero-input prompt as input to the language model;
receiving, from the language model in response to the non-zero-input prompt, the requested autocomplete text; and
surfacing the autocomplete text as a suffix to the text prefix.

### Claim 19 (depends on 17)

The system of claim 17, wherein the operations further comprise, in response to surfacing the autocomplete text, removing the potential input text from display.

### Claim 20 (depends on 17)

The system of claim 17, wherein the potential input text includes a plurality of questions, and surfacing the questions includes displaying a selectable user interface element for each of the questions.
