# US 2021/0065677 A1 · System and method for interactive virtual assistant generation for assemblages

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0065677 A1
- Document ID: US20210065677A1
- Status: Published application
- Kind: A1
- Application number: 16/586,423
- Filed: 2019-09-27
- Published: 2021-03-04
- Earliest claimed: 2019-08-29
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Rajat Talwar; Otar Bichiashvili; Lucas Pritz; Sagar Narla
- Web page: https://patentgrep.com/patent/US20210065677A1

## CPC

- G09B 5/06: with both visual and audible presentation of the material to be studied
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G09B 19/003: {Repetitive work cycles; Sequence of movements}
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 15/063: {Training}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/183: using context dependencies, e.g. language models
- G10L 2015/225: {Feedback of the input speech}
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods for interactive virtual assistant generation for assemblages are performed by systems and devices. Representations of existing assembly manuals of assemblages are received by a host service. The representations are received from users in the form of captured images and retrieved from websites of assemblage suppliers. The host service extracts information from representations including text and illustrations, and generates instructions including a first set of assembly instructions from the information and an instruction model, and a second set of prompt-response instructions from the information and a language model. The host service generates interactive smart manuals of assemblages using the first and second sets. Interactive smart manuals may be instances of virtual digital assistants specific to the assemblage. A user executes an interactive smart manual via a user interface (UI) and provides natural language prompts via the UI causing navigation through the manual.

## Claims

### Claim 1

A system comprising:
a processing system that includes one or more processors; and
a memory that stores computer program instructions that are executable by the processing system, the computer program instructions including:
a manual retriever that:
receives a representation of an existing assembly manual of an assemblage;
an extractor that:
extracts information from the representation;
an instruction generator that:
generates a first set of assembly instructions based at least on the information from the representation; and
generates a second set of prompt-response instructions based on a language model and the information; and
a manual generator that:
generates an interactive smart manual of the assemblage based at least on the first set and the second set.

### Claim 2 (depends on 1)

The system of claim 1, wherein the interactive smart manual is generated to be executed in association with a virtual digital assistant at a user device.

### Claim 3 (depends on 1)

The system of claim 1, wherein the extractor, to extract the information from the representation:
extracts one or more illustrations from the representation; and
extracts one or more portions of text from the representation; and
wherein the instruction converter, to generate the first set of assembly instructions:
generates the first set of assembly instructions from at least one of the one or more images or the one or more portions of text.

### Claim 4 (depends on 3)

The system of claim 3, wherein the instruction converter, to generate the first set of assembly instructions, generates, from at least one of the one or more images or the one or more portions of text, one or more of:
a number of assembly steps to complete the assemblage;
a parts list of the assemblage; and
a hardware list for assembly the assemblage.

### Claim 5 (depends on 1)

The system of claim 1, wherein the representation of the existing assembly manual comprises at least one of:
an electronic file stored with a website; and
one or more captured images stored at a user device.

### Claim 6 (depends on 5)

The system of claim 5, wherein the manual retriever:
receives an identifier of the assemblage; and
automatically locates the representation over a network.

### Claim 7 (depends on 1)

The system of claim 1, wherein the prompt-response instructions, based at least on natural language user input received via a user device:
navigate between ones of the assembly instructions;
cause a display of a portion of the assemblage by the user device; and
cause an audio description of a part of the assemblage to be played back by the user device.

### Claim 8 (depends on 1)

The system of claim 1, wherein the manual retriever:
receives a note generated by a user, the note being associated with the existing assembly manual of the assemblage; and
wherein the instruction converter:
generates the first set of assembly instructions also based on the note.

### Claim 9

A computer-implemented method, comprising:
extracting information from a representation of an existing assembly manual of an assemblage;
generating a first set of assembly instructions based on the information from the representation and an instruction model;
generating a second set of prompt-response instructions based on a language model and the information; and
generating an interactive smart manual of the assemblage based at least on the first set and the second set, the interactive smart manual being an instance of a virtual digital assistant that is specific to the assemblage.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein extracting the information from the representation comprises at least one of:
extracting one or more images from the representation, and
extracting one or more portions of text from the representation; and
wherein generating the first set of assembly instructions comprises:
generating the first set of assembly instructions from at least one of the one or more images or the one or more portions of text.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein generating the first set of assembly instructions comprises:
generating, from at least one of the one or more images or the one or more portions of text, one or more of:
a number of assembly steps to complete the assemblage;
a parts list of the assemblage; and
a hardware list for assembly the assemblage.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9, wherein the representation of the existing assembly manual comprises at least one of:
an electronic file stored with a website; and
one or more captured images stored at a user device.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, further comprising:
receiving an identifier of the assemblage; and
automatically locating the representation over a network.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9, wherein said generating the second set of prompt-response instructions comprises:
generating the second set of prompt-response instructions as including responses to vocalized natural language user inputs received via a user interface to perform at least one of:
navigating between ones of the assembly instructions;
causing a display of a portion of the assemblage by a user device; and
causing an audio description of a part of the assemblage to be played back by the user device.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, further comprising:
training at least one of the instruction model or the language model based on information received from the user device subsequent to providing the interactive smart manual to the user device.

### Claim 16 (depends on 9)

The computer-implemented method of claim 9, further comprising:
receiving a note generated by a user, the note being associated with the existing assembly manual of the assemblage; and
generating the first set of assembly instructions also based on the note.

### Claim 17

A computer-readable storage medium having program instructions recorded thereon that, when executed by a processing device, perform a method comprising:
providing a request to a service host, the request specifying generation of an interactive smart manual from an existing assembly manual of an assemblage;
receiving the interactive smart manual responsive to the request;
initiating execution of the interactive smart manual in association with a virtual digital assistant of a user interface (UI);
receiving a natural language prompt from a user during the execution; and
navigating the interactive smart manual via the UI based on the natural language prompt.

### Claim 18 (depends on 17)

The computer-readable storage medium of claim 17, wherein said navigating comprises at least one of:
providing a prior step of an assembly of the assemblage via the UI; and
providing a part from a parts list of the assemblage via the UI during presentation of a step of the assembly of the assemblage.

### Claim 19 (depends on 17)

The computer-readable storage medium of claim 17, wherein said navigating comprises:
providing a step of an assembly of the assemblage, to the user via the UI, that includes content of a note created by the user that was included in the request.

### Claim 20 (depends on 17)

The computer-readable storage medium of claim 17, wherein said providing the request comprises providing at least one of:
an image of the existing assembly manual;
an identifier of the existing assembly manual and an indication by the user to retrieve the existing assembly manual over a network; and
a note created by the user that includes information associated with an assembly step of the assemblage.
