# US 2022/0093000 A1 · Systems and methods for multimodal book reading

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0093000 A1
- Document ID: US20220093000A1
- Status: Published application
- Kind: A1
- Application number: 17/606,728
- Filed: 2021-02-27
- Published: 2022-03-24
- Earliest claimed: 2020-02-29
- Assignees: Embodied, Inc.
- Applicants: Embodied, Inc.
- Inventors: Mario E Munich; Stefan Scherer; Paolo Pirjanian; Craig Allen
- Web page: https://patentgrep.com/patent/US20220093000A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G09B 17/003: {electrically operated apparatus or devices}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 11/001: {with emotions simulating means}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 40/30: Semantic analysis
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G09B 5/062: {Combinations of audio and printed presentations, e.g. magnetically striped cards, talking books, magnetic tapes with printed texts thereon}
- G09B 5/067: {Combinations of audio and projected visual presentation, e.g. film, slides}
- G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods to process reading articles for a multimodal book application are disclosed. Exemplary implementations may: identify a title of a reading article; store the title of the reading article in a database; scan two or more pages of the reading and generating text representing content of the reading article; analyze the generated text to identify characteristics of the reading article; store the identified characteristics in the database; associate the identified characteristics with the reading article title; generate augmented content files for one or more portions of the reading article based at least in part on the identified characteristics, and; and store the augmented content files in the database and associating the augmented content files with different portions of the reading article.

## Claims

### Claim 1

A method to process reading articles utilizing a robot computing device, comprising:
accessing computer-readable instructions from one or more memory devices for execution by one or more processors of the robot computing device;
executing the computer-readable instructions accessed from the one or more memory devices by the one or more processors of the robot computing device; and
wherein executing the computer-readable instructions further comprising:
identifying a title of one or more reading articles;
storing the one or more reading article titles in a book database;
scanning two or more pages of the one or more reading articles and generating text files representing content of the one or more reading article;
analyzing the generated text files for the one or more reading articles to identify characteristics of the one or more reading articles, the identified characteristics of the one or more reading articles including genre, plot, mood, or one or more characters of the one or more reading articles;
storing the identified characteristics of the one or more reading articles in the database and associating the identified characteristics with the one or more reading article titles;
generating augmented content files for one or more portions of the one or more reading articles based at least in part on the identified characteristics, and
storing the augmented content files in the database and associating the augmented content files with the one or more portions of the one or more reading articles.

### Claim 2 (depends on 1)

The method of claim 1, wherein the augmented content files include questions relating to the one or more reading articles.

### Claim 3 (depends on 1)

The method of claim 1, the characteristics of the one or more reading articles further including pacing of the one or more reading articles.

### Claim 4 (depends on 1)

The method of claim 1, the characteristics of the one or more reading articles further including scenes, keywords, and concepts present in the one or more reading articles.

### Claim 5 (depends on 1)

The method claim 1, wherein executing the computer-readable instructions further comprising:
selecting a reading article for reading by a user;
capturing measurements and/or parameters of the user's reading of the selected reading article by following the user's reading from one or more input devices;
determining a user's progression of the user's reading of the selected reading article by reviewing the captured measurements and/or parameters from the one or more input devices;
retrieving augmented content files from an asset database, the retrieval of the augmented content files based at least in part on the captured measurements and/or parameters and the user's progression in reading the selected reading article; and
transmitting the one or more augmented content files to at least one of the one or more multimodal output devices for presentation by the robot computing device to the user.

### Claim 6 (depends on 5)

The method of claim 5, wherein selecting the reading article for reading by the user further comprises: a) requesting the selected reading article by capturing verbal commands via a microphone of the robot computing device; b) touching the selected reading article the user would like to read and having the input device capture the touching of the selected reading article; c) gesturing or pointing to the selected reading article and having the imaging device capturing the gesturing or pointing; or d) utilizing an input from mouse or keyboard to determine selection of the selected reading article.

### Claim 7 (depends on 5)

The method of claim 5, wherein selecting the reading article for reading by the user further comprises: a) analyzing a user's description to determine an appropriate reading article; b) analyzing past reading articles that the use enjoyed to determine an appropriate reading article; c) analyzing a user's reading ability to determine an appropriate reading article; or d) determining via an algorithm whether a book is being advertised as an appropriate reading article

### Claim 8 (depends on 5)

The method of claim 5, wherein the one or more input devices include one or more RFID readers to obtain information from the one or more RFID chips in the selected reading article, one or more imaging devices to capture text and/or images from the selected reading article and/or one or more microphones to capture a user's voice in reading the selected reading article and generating audio files based on the user's reading.

### Claim 9 (depends on 5)

The method of claim 5, wherein the augmented content files comprise audio files for speech to provide audible feedback to the user, mobility commands for the robot computing device to cause the robot computing device components to move, commands or instructions to be sent to the display for visual feedback to the user; or music files to be reproduced by one or more speakers, wherein the audio files may be music files or audible sounds such as laughs or sighs.

### Claim 10

(canceled)

### Claim 11 (depends on 9)

The method of claim 9, wherein the visual feedback may be instructions to cause a monitor to display different facial expressions or visual reactions and/or one or more light emitting diodes (LEDs) or light bars to display light patterns and wherein the mobility instructions may be instructions or commands to cause a head of the robot computing device to move, the appendages of the robot computing device to gesture or move, and/or the robot computing device to move.

### Claim 12

(canceled)

### Claim 13 (depends on 5)

The method of claim 5, wherein executing the computer-readable instructions further comprising generating an audio file and transmitting the audio file to a speaker of the robot computing device, the audio file representing a question or comments to be audibly played to the user via the one or more speakers of the robot computing device to request a response from the user.

### Claim 14 (depends on 13)

The method of claim 13, wherein executing the computer-readable instructions further comprising receiving a verbal response from the user as a response audio file at one or more microphones to create an interactive experience with the robot computing device; analyzing the response audio file, by the robot computing device, to determine a level of reading comprehension level of the reader; and calculating a reading level of the user based on the determined reading comprehensive level of the reader; based on characteristics of the selected reading article being read, and the characteristics including vocabulary of the selected reading article.

### Claim 15

(canceled)

### Claim 16

(cancelled)

### Claim 17

(canceled)

### Claim 18 (depends on 5)

The method of claim 5, wherein executing the computer-readable instructions further comprising calculating the reading level of a user based on the parameters and measurements captured by the one or more input devices as well as characteristics of the selected article being read and the parameters and measurements captured by the one or more input devices are a) clarity of reading of the selected reading article determined by capturing, via one or more microphones audio files of the user reading the selected reading article and analyzing the audio files; b) speed of reading of the selected reading article determined by capturing, via one or more microphones audio files of the user reading the selected reading article and analyzing the audio files; or c) fluency of reading of the selected reading article determined by capturing, via one or more microphones audio files of the user reading the selected reading article and analyzing the audio files to determine if the user is correctly pronouncing the words of the selected reading article.

### Claim 19 (depends on 18)

The method of claim 18, wherein executing the computer-readable instructions further comprising generating a report based at least in part on the captured parameters and measurements captured by the input devices, the characteristics of the reading article and/or the determined reading comprehension.

### Claim 20

A method of processing a reading article for interactive multimodal reading as a child is reading a reading article, comprising:
accessing computer-readable instructions from one or more memory devices for execution by one or more processors of the robot computing device;
executing the computer-readable instructions accessed from the one or more memory devices by the one or more processors of the robot computing device; and
wherein executing the computer-readable instructions further comprising:
capturing image files and text files of two or more pages of the reading article by one or more input devices while a user is reading the reading article;
capturing voice files, utilizing the one or more microphones of the robot computing device, of the user reading the reading article;
converting the captured voice files to voice text files;
generating measurements and/or parameters of the user's reading of the reading article by analyzing the voice text files;
analyzing the captured images files, text files and/or voice text files to identify characteristics of the reading article, the identified characteristics of the reading article including genre, plot, mood, or one or more characters of the reading article;
storing the identified characteristics of the reading article in an asset database and associating the identified characteristics with a reading article title;
generating augmented content files for one or more portions of the reading article based at least in part on the identified characteristics of the reading article, and
storing the augmented content files in the asset database and associating the augmented content files with different portions of the reading article.

### Claim 21 (depends on 20)

The method of claim 20, wherein executing the computer-readable instructions further comprising: determining a user's progression of the user's reading of the reading article by reviewing the captured measurements and/or parameters from the one or more microphones.

### Claim 22 (depends on 20)

The method of claim 20, wherein executing the computer-readable instructions further comprising:
retrieving augmented content files from an asset database, the retrieval of the augmented content files based at least in part on the captured measurements and/or parameters and the user's progression in reading the reading article; and
transmitting the one or more augmented content files to at least one of the one or more multimodal output devices for presentation by the robot computing device to the user.
