# US 2020/0401794 A1 · Nonverbal information generation apparatus, nonverbal information generation model learning apparatus, methods, and programs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401794 A1 - Document ID: US20200401794A1 - Status: Published application - Kind: A1 - Application number: 16/969,765 - Filed: 2019-02-15 - Published: 2020-12-24 - Earliest claimed: 2018-02-16 - Assignees: Nippon Telegraph and Telephone Corporation - Applicants: Nippon Telegraph and Telephone Corporation - Inventors: Ryo ISHII; Ryuichiro HIGASHINAKA; Taichi KATAYAMA; Junji TOMITA; Nozomi KOBAYASHI; Kyosuke NISHIDA - Web page: https://patentgrep.com/patent/US20200401794A1 ## CPC - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - A63H 11/00: Self-movable toy figures - G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06N 20/00: Machine learning - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/107: {Static hand or arm} - G06V 40/174: {Facial expression recognition} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/225: {Feedback of the input speech} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - A63H: TOYS, e.g. TOPS, DOLLS, HOOPS OR BUILDING BLOCKS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A nonverbal information generation apparatus includes a nonverbal information generation unit that generates time-information-stamped nonverbal information that corresponds to time-information-stamped text feature quantities on the basis of the time-information-stamped text feature quantities and a learned nonverbal information generation model. The time-information-stamped text feature quantities are configured to include feature quantities that have been extracted from text and time information representing times assigned to predetermined units of the text. The nonverbal information is information for controlling an expression unit that expresses behavior that corresponds to the text. ## Claims ### Claim 1 A nonverbal information generation apparatus comprising: a hardware processor that generates time-information-stamped nonverbal information that corresponds to time-information-stamped text feature quantities on the basis of the time-information-stamped text feature quantities and a learned nonverbal information generation model, wherein the time-information-stamped text feature quantities are configured to comprise feature quantities that have been extracted from text and time information representing times assigned to predetermined units of the text, and the nonverbal information is information for controlling an expression device that expresses behavior that corresponds to the text. ### Claim 2 A nonverbal information generation apparatus comprising: a hardware processor that generates time-information-stamped nonverbal information that corresponds to time-information-stamped voice feature quantities on the basis of the time-information-stamped voice feature quantities and a learned nonverbal information generation model, wherein the time-information-stamped voice feature quantities are configured to comprise feature quantities that have been extracted from voice information and time information representing times of predetermined units when the voice information is emitted, and the nonverbal information is information for controlling an expression device that expresses behavior that corresponds to the voice information. ### Claim 3 (depends on 1) The nonverbal information generation apparatus according to claim 1, wherein the hardware processor performs a text analysis on text that has been input, assigns a playback time when it is assumed that the text will be output externally as voice to the text for each predetermined unit as the time information, and extracts the time-information-stamped text feature quantities. ### Claim 4 (depends on 1) The nonverbal information generation apparatus according to claim 1, wherein the text feature quantities comprise at least one of a dialogue act extracted from the text and thesaurus information. ### Claim 5 (depends on 1) The nonverbal information generation apparatus according to claim 1, further comprising the expression device that expresses the nonverbal information, wherein the hardware processor controls the expression device so that the time-information-stamped nonverbal information is expressed from the expression device on the basis of time information assigned to the nonverbal information. ### Claim 6 A nonverbal information generation model learning apparatus comprising: a hardware processor that: acquires text information representing text corresponding to voice of a speaker and time information representing times assigned to predetermined units of the text; acquires nonverbal information representing information relating to behavior of the speaker when the speaker performed speaking corresponding to the text, and time information representing times at which the behavior was performed and corresponding to the nonverbal information, and creates time-information-stamped nonverbal information; extracts time-information-stamped text feature quantities representing feature quantities of the text information from the acquired text information and the time information corresponding to the text information; and learns a nonverbal information generation model for generating the acquired time-information-stamped nonverbal information on the basis of the extracted time-information-stamped text feature quantities. ### Claim 7 A nonverbal information generation model learning apparatus comprising: a hardware processor that: acquires voice information corresponding to voice of a speaker and time information representing times of predetermined units when the voice information is emitted; acquires nonverbal information representing information relating to behavior of the speaker when the speaker performed speaking corresponding to the voice and time information representing times at which the behavior was performed and corresponding to the nonverbal information, and creates time-information-stamped nonverbal information; extracts time-information-stamped voice feature quantities representing feature quantities of the voice information from the acquired voice information and the time information corresponding to the voice information; and learns a nonverbal information generation model for generating the acquired time-information-stamped nonverbal information on the basis of the extracted time-information-stamped voice feature quantities. ### Claim 8 A nonverbal information generation method comprising: generating time-information-stamped nonverbal information that corresponds to time-information-stamped text feature quantities on the basis of the time-information-stamped text feature quantities and a learned nonverbal information generation model, wherein the time-information-stamped text feature quantities are configured to comprise feature quantities that have been extracted from text and time information representing times assigned to predetermined units of the text, and the nonverbal information is information for controlling an expression device that expresses behavior that corresponds to the text. ### Claim 9 A nonverbal information generation method comprising: generating time-information-stamped nonverbal information that corresponds to time-information-stamped voice feature quantities on the basis of the time-information-stamped voice feature quantities and a learned nonverbal information generation model, wherein the time-information-stamped voice feature quantities are configured to comprise feature quantities that have been extracted from voice information, and time information representing times of predetermined units when the voice information is emitted, and the nonverbal information is information for controlling an expression device that expresses behavior that corresponds to the voice information. ### Claim 10 (depends on 1) A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to function as the nonverbal information generation apparatus according to claim 1. ### Claim 11 A nonverbal information generation model learning apparatus comprising: a hardware processor that: acquires text information representing text corresponding to voice of a speaker and time information representing times assigned to predetermined units of the text; acquires nonverbal information representing information relating to behavior of a listener of speaking of the speaker corresponding to the text when the speaker performed the speaking, and time information representing times at which the behavior was performed and corresponding to the nonverbal information, and creates time-information-stamped nonverbal information; extracts time-information-stamped text feature quantities representing feature quantities of the text information from the acquired text information and the time information corresponding to the text information; and learns a nonverbal information generation model for generating the acquired time-information-stamped nonverbal information on the basis of the extracted time-information-stamped text feature quantities. ### Claim 12 (depends on 6) The nonverbal information generation model learning apparatus according to claim 6, wherein the text feature quantities comprise at least one of a dialogue act extracted from the text and thesaurus information. ### Claim 13 A nonverbal information generation model learning apparatus comprising: a hardware processor that: acquires voice information corresponding to voice of a speaker and time information representing times of predetermined units when the voice information is emitted; acquires nonverbal information representing information relating to behavior of a listener of speaking of the speaker corresponding to the voice when the speaker performed the speaking, and time information representing times at which the behavior was performed and corresponding to the nonverbal information, and creates time-information-stamped nonverbal information; extracts time-information-stamped voice feature quantities representing feature quantities of the voice information from the acquired voice information and the time information corresponding to the voice information; and learns a nonverbal information generation model for generating the acquired time-information-stamped nonverbal information on the basis of the extracted time-information-stamped voice feature quantities. ### Claim 14 A nonverbal information generation model learning method comprising: acquiring text information representing text corresponding to voice of a speaker and time information representing times assigned to predetermined units of the text; acquiring nonverbal information representing information relating to behavior of a listener of speaking of the speaker corresponding to the text when the speaker performed the speaking, and time information representing times at which the behavior was performed and corresponding to the nonverbal information, and creating time-information-stamped nonverbal information; extracting time-information-stamped text feature quantities representing feature quantities of the text information from the acquired text information and the time information corresponding to the text information; and learns a nonverbal information generation model for generating the acquired time-information-stamped nonverbal information on the basis of the extracted time-information-stamped text feature quantities. ### Claim 15 A nonverbal information generation model learning method comprising: acquiring voice information corresponding to voice of a speaker and time information representing times of predetermined units when the voice information is emitted; acquiring nonverbal information representing information relating to behavior of a listener of speaking of the speaker corresponding to the voice when the speaker performed the speaking, and time information representing times at which the behavior was performed and corresponding to the nonverbal information, and creating time-information-stamped nonverbal information; extracting time-information-stamped voice feature quantities representing feature quantities of the voice information from the acquired voice information and the time information corresponding to the voice information; and learning a nonverbal information generation model for generating the acquired time-information-stamped nonverbal information on the basis of the extracted time-information-stamped voice feature quantities. ### Claim 16 (depends on 6) A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor or a computer, cause the computer to function as the nonverbal information generation model learning apparatus according to claim 6. ### Claim 17 (depends on 2) The nonverbal information generation apparatus according to claim 2, further comprising the expression device that expresses the nonverbal information, wherein the hardware processor controls the expression device so that the time-information-stamped nonverbal information is expressed from the expression device on the basis of time information assigned to the nonverbal information. ### Claim 18 (depends on 2) A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to function as the nonverbal information generation apparatus according to claim 2. ### Claim 19 (depends on 11) The nonverbal information generation model learning apparatus according to claim 11, wherein the text feature quantities comprise at least one of a dialogue act extracted from the text and thesaurus information. ### Claim 20 (depends on 7) A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to function as the nonverbal information generation model learning apparatus according to claim 7. ### Claim 21 (depends on 11) A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to function as the nonverbal information generation model learning apparatus according to claim 11. ### Claim 22 (depends on 13) A non-transitory computer-readable medium having computer-executable instructions that, upon execution of the instructions by a processor of a computer, cause the computer to function as the nonverbal information generation model learning apparatus according to claim 13.