# US 2021/0241009 A1 · Emotion prediction device, method, program, and vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241009 A1
- Document ID: US20210241009A1
- Status: Published application
- Kind: A1
- Application number: 17/105,869
- Filed: 2020-11-27
- Published: 2021-08-05
- Earliest claimed: 2020-02-05
- Assignees: Toyota Jidosha Kabushiki Kaisha
- Applicants: Toyota Jidosha Kabushiki Kaisha
- Inventors: Midori ICHIKAWA; Toshiya SATAKE; Yasunao YOSHIZAKI
- Web page: https://patentgrep.com/patent/US20210241009A1

## CPC

- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 40/174: {Facial expression recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An emotion prediction device mounted on a vehicle includes a control unit. The control unit is capable of executing functions of a first prediction unit configured to predict an emotion of a user based on a facial expression of the user, a second prediction unit configured to predict an emotion of the user based on the motion of the user, and a third prediction unit configured to predict the emotion of the user based on the emotion predicted by the first prediction unit and the emotion predicted by the second prediction unit. The control unit is capable of restricting execution of at least a part of the functions of the first, second, and third prediction units, based on at least a processing load of the control unit.

## Claims

### Claim 1

An emotion prediction device mounted on a vehicle, the emotion prediction device comprising:
a control unit configured to execute functions of:
a first prediction unit configured to predict an emotion of a user based on a facial expression of the user;
a second prediction unit configured to predict an emotion of the user based on a motion of the user; and
a third prediction unit configured to predict an emotion of the user based on the emotion predicted by the first prediction unit and the emotion predicted by the second prediction unit,
wherein the control unit is configured to restrict execution of at least a part of the functions of the first prediction unit, the second prediction unit, and the third prediction unit, based on at least a processing load of the control unit.

### Claim 2 (depends on 1)

The emotion prediction device according to claim 1, wherein the restriction of the execution includes stopping or extending a cycle of executing any one of the functions of the first prediction unit, the second prediction unit, and the third prediction unit.

### Claim 3 (depends on 1)

The emotion prediction device according to claim 1, wherein the control unit is configured to determine a state of the user or the vehicle, and restrict the execution based on the determination result.

### Claim 4 (depends on 3)

The emotion prediction device according to claim 3, wherein the control unit is configured to, upon determining that the user is not seated, stop all the functions of the first prediction unit, the second prediction unit, and the third prediction unit, as the restriction of the execution.

### Claim 5 (depends on 3)

The emotion prediction device according to claim 3, wherein, the control unit is configured to, upon determining that the facial expression of the user is not acquirable, stop the functions of the first prediction unit and the third prediction unit, as the restriction of the execution.

### Claim 6 (depends on 3)

The emotion prediction device according to claim 3, wherein the control unit is configured to, upon determining that the user is operating a predetermined device, stop the functions of the second prediction unit and the third prediction unit, as the restriction of the execution.

### Claim 7 (depends on 3)

The emotion prediction device according to claim 3, wherein the control unit is configured not to use any information acquired from one or more image sensors provided in the vehicle to determine whether the restriction of the execution is executed.

### Claim 8

An emotion prediction method executed by a computer of an emotion prediction device mounted on a vehicle, the emotion prediction method comprising:
a first prediction step of predicting an emotion of a user based on a facial expression of the user;
a second prediction step of predicting an emotion of the user based on a motion of the user;
a third prediction step of predicting an emotion of the user based on the emotion predicted in the first prediction step and the emotion predicted in the second prediction step; and
an execution restriction step of restricting execution of at least a part of the first prediction step, the second prediction step, and the third prediction step, based on at least a processing load of the computer.

### Claim 9

An emotion prediction program causing a computer of an emotion prediction device mounted on a vehicle to execute:
a first prediction step of predicting an emotion of a user based on a facial expression of the user;
a second prediction step of predicting an emotion of the user based on a motion of the user;
a third prediction step of predicting an emotion of the user based on the emotion predicted in the first prediction step and the emotion predicted in the second prediction step; and
an execution restriction step of restricting execution of at least a part of the first prediction step, the second prediction step, and the third prediction step, based on at least a processing load of the computer.

### Claim 10 (depends on 1)

A vehicle comprising the emotion prediction device according to claim 1.