# US 10,482,352 B2 · Information processing device and non-transitory computer readable medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,352 B2 - Document ID: US10482352B2 - Status: Granted - Kind: B2 - Application number: 15/693,093 - Filed: 2017-08-31 - Published: 2018-09-20 - Granted: 2019-11-19 - Earliest claimed: 2017-03-15 - Assignees: Fuji Xerox Co., Ltd. - Applicants: Fuji Xerox Co., Ltd. - Inventors: Genki Osada; Roshan Thapliya - Web page: https://patentgrep.com/patent/US10482352B2 ## CPC - G06V 10/82: using neural networks - G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling} - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06F 18/24133: {Distances to prototypes} - G06V 40/174: {Facial expression recognition} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G10L 25/63: for estimating an emotional state - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Information processing including, a first data acquirer that acquires first observational data; a second data acquirer that acquires second observational data; and a processor that performs machine learning by semi-supervised learning to output an emotion of a person as a latent factor using training data of the first observational data and the second observational data, and that infers and outputs the emotion from at least one of the first observational data and the second observational data using a trained model. ## Claims ### Claim 1 An information processing device comprising: a first data acquirer that acquires first observational data; a second data acquirer that acquires second observational data; and a processor that performs machine learning by semi-supervised learning to output an emotion of a person as a latent factor using training data of the first observational data and the second observational data, and that infers and outputs the emotion from at least one of the first observational data and the second observational data using a trained model, wherein the processor further generates the first observational data and the second observational data from the latent factor obtained through the inference, and interpolates a missing portion of at least one of the first observational data and the second observational data respectively acquired by the first data acquirer and the second data acquirer, by using at least one of the generated first observational data and second observational data, and outputs the interpolated portion. ### Claim 2 (depends on 1) The information processing device according to claim 1, wherein the processor performs the machine learning by a combination of a recurrent neural network (RNN) and a variational autoencoder (VAE). ### Claim 3 (depends on 2) The information processing apparatus according to claim 2, wherein the first observational data is motion data on a person, and the second observational data is voice data on the person. ### Claim 4 (depends on 1) The information processing device according to claim 1, wherein the processor infers the latent factor using the training data of the first observational data and the second observational data, and performs the machine learning by comparing a label obtained by the inference with a label given as correct data. ### Claim 5 (depends on 4) The information processing apparatus according to claim 4, wherein the first observational data is motion data on a person, and the second observational data is voice data on the person. ### Claim 6 (depends on 1) The information processing device according to claim 1, wherein the processor calculates a first characteristic quantity of the first observational data acquired by the first data acquirer and a second characteristic quantity of the second observational data acquired by the second data acquirer, infers the latent factor based on the first characteristic quantity and the second characteristic quantity, and calculates the first characteristic quantity and the second characteristic quantity so that the first characteristic quantity and the second characteristic quantity are correlated to each other. ### Claim 7 (depends on 6) The information processing apparatus according to claim 6, wherein the first observational data is motion data on a person, and the second observational data is voice data on the person. ### Claim 8 (depends on 1) The information processing apparatus according to claim 1, wherein the first observational data is motion data on a person, and the second observational data is voice data on the person. ### Claim 9 (depends on 1) The information processing device according to claim 1, wherein the first observational data is motion data on a person, the second observational data is voice data on the person, and the processor interpolates a missing portion of the second observational data and outputs the interpolated portion. ### Claim 10 An information processing device comprising: a first data acquirer that acquires first observational data; a second data acquirer that acquires second observational data; and a processor that performs machine learning by semi-supervised learning to output an emotion of a person as a latent factor using training data of the first observational data and the second observational data, and that infers and outputs the emotion from at least one of the first observational data and the second observational data using a trained model, wherein the processor infers the latent factor using the training data of the first observational data and the second observational data, generates the first observational data and the second observational data from the latent factor obtained through the inference, and performs the machine learning by comparing the generated first observational data and second observational data with the first observational data and the second observational data acquired by the first data acquirer and the second data acquirer, respectively. ### Claim 11 (depends on 10) The information processing apparatus according to claim 10, wherein the first observational data is motion data on a person, and the second observational data is voice data on the person. ### Claim 12 An information processing device comprising: a first data acquirer that acquires first observational data; a second data acquirer that acquires second observational data; and a processor that performs machine learning by semi-supervised learning to output an emotion of a person as a latent factor using training data of the first observational data and the second observational data, and that infers and outputs the emotion from at least one of the first observational data and the second observational data using a trained model, wherein the processor performs the machine learning by a combination of a recurrent neural network (RNN) and a variational autoencoder (VAE), and wherein the processor uses the RNN to calculate a first characteristic quantity of the first observational data acquired by the first data acquirer and a second characteristic quantity of the second observational data acquired by the second data acquirer, and uses the VAE to infer the latent factor using the first characteristic quantity and the second characteristic quantity. ### Claim 13 (depends on 12) The information processing apparatus according to claim 12, wherein the first observational data is motion data on a person, and the second observational data is voice data on the person. ### Claim 14 An information processing device comprising: a first data acquirer that acquires first observational data; a second data acquirer that acquires second observational data; and a processor that performs machine learning by semi-supervised learning to output an emotion of a person as a latent factor using training data of the first observational data and the second observational data, and that infers and outputs the emotion from at least one of the first observational data and the second observational data using a trained model, wherein the processor performs the machine learning by a combination of a recurrent neural network (RNN) and a variational autoencoder (VAE), wherein the first observational data is motion data on a person, and the second observational data is voice data on the person, and wherein the processor converts the interpolated second observational data to text data, and outputs the text data. ### Claim 15 A non-transitory computer readable medium storing a program causing a computer to execute a process, the process comprising: inputting first observational data and second observational data to the computer; performing machine learning by semi-supervised learning to output an emotion of a person as a latent factor using training data of the first observational data and the second observational data; inferring and outputting the emotion from at least one of the first observational data and the second observational data using a trained model; generating the first observational data and the second observational data from the latent factor obtained through the inference; interpolating a missing portion of at least one of the first observational data and the second observational data respectively input to the computer, by using at least one of the generated first observational data and second observational data; and outputting the interpolated portion.