# US 2021/0280188 A1 · Information processing method, information processing system, and non-transitory computer-readable recording medium recording information processing program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280188 A1
- Document ID: US20210280188A1
- Status: Published application
- Kind: A1
- Application number: 17/322,180
- Filed: 2021-05-17
- Published: 2021-09-09
- Earliest claimed: 2019-05-17
- Assignees: Panasonic Intellectual Property Management Co., Ltd.
- Applicants: Panasonic Intellectual Property Management Co., Ltd.
- Inventors: Masaaki HARADA
- Web page: https://patentgrep.com/patent/US20210280188A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- A61B 5/4809: {Sleep detection, i.e. determining whether a subject is asleep or not}
- A61B 5/4812: {Detecting sleep stages or cycles}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- A61B 5/0205: Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

A server acquires biological data from a biosensor that senses a biological state of a user; acquires sound data from a microphone that senses sound in space in which the user is present; determines a sleep state of the user based on the biological data; converts the sound data into text data; and controls output of a device control command based on the text data by controlling necessity of operation determination to determine whether a predetermined operation has been performed by the user based on the sleep state.

## Claims

### Claim 1

An information processing method comprising, by a computer:
acquiring biological data from a biosensor that senses a biological state of a user;
acquiring sound data from a microphone that senses sound in space in which the user is present;
determining a sleep state of the user based on the biological data;
converting the sound data into text data; and
controlling output of a device control command based on the text data by controlling necessity of operation determination to determine whether a predetermined operation has been performed by the user based on the sleep state.

### Claim 2 (depends on 1)

The information processing method according to claim 1, wherein the control of the output of the device control command includes:
determining the necessity of the operation determination based on the sleep state;
when it is determined that the operation determination is necessary,
(i) determining whether the predetermined operation has been performed by the user, and
(ii) outputting the device control command when it is determined that the predetermined operation has been performed; and
when it is determined that the operation determination is unnecessary, outputting the device control command.

### Claim 3 (depends on 2)

The information processing method according to claim 2, further comprising acquiring environmental information indicating an environmental state in the space,
wherein the control of the output of the device control command includes determining the necessity of the operation determination based on the sleep state and the environmental information.

### Claim 4 (depends on 2)

The information processing method according to claim 2, wherein the control of the output of the device control command includes transmitting the device control command to a device associated with the device control command.

### Claim 5 (depends on 2)

The information processing method according to claim 2, wherein
the determination of the necessity of the operation determination includes: determining whether the sleep state is a lying in bed state in which the user lies in bed; and determining that the operation determination is unnecessary when it is determined that the sleep state is the lying in bed state.

### Claim 6 (depends on 2)

The information processing method according to claim 2, wherein
the determination of the necessity of the operation determination includes: determining whether the sleep state is a nocturnal awakening state in which the user awakes during a sleeping period from lying in bed to rising; and determining that the operation determination is unnecessary when it is determined that the sleep state is the nocturnal awakening state.

### Claim 7 (depends on 2)

The information processing method according to claim 2, wherein
the determination of the necessity of the operation determination includes: determining whether the sleep state is a nocturnal awakening sign state indicating a sign of shifting to the nocturnal awakening state in which the user awakes during the sleeping period from lying in bed to rising; and determining that the operation determination is unnecessary when it is determined that the sleep state is the nocturnal awakening sign state.

### Claim 8 (depends on 2)

The information processing method according to claim 2, wherein
the determination of the necessity of the operation determination includes: determining whether the sleep state is a first awake state from when the user lies in bed to falling asleep; and determining that the operation determination is unnecessary when it is determined that the sleep state is the first awake state.

### Claim 9 (depends on 2)

The information processing method according to claim 2, wherein
the determination of the necessity of the operation determination includes: determining whether the sleep state is a second awake state during a period from a scheduled rising time of the user to a predetermined time; and determining that the operation determination is unnecessary when it is determined that the sleep state is the second awake state.

### Claim 10 (depends on 2)

The information processing method according to claim 2, wherein
the determination of the necessity of the operation determination includes: determining whether the sleep state is a state from when the user lies in bed to rising; and determining that the operation determination is unnecessary when it is determined that the sleep state is the state from when the user lies in bed to rising.

### Claim 11 (depends on 5)

The information processing method according to claim 5, wherein the control of the output of the device control command includes generating the device control command based on the predetermined sleep state when it is determined that the sleep state is the predetermined sleep state.

### Claim 12 (depends on 11)

The information processing method according to claim 11, wherein the generation of the device control command includes generating the device control command according to a time length from the scheduled rising time of the user to current time when it is determined that the sleep state is the predetermined sleep state.

### Claim 13 (depends on 5)

The information processing method according to claim 5, wherein the control of the output of the device control command includes generating only the predetermined device control command related to going to sleep of the user when it is determined that the sleep state is the predetermined sleep state.

### Claim 14 (depends on 1)

The information processing method according to claim 1, wherein the predetermined operation includes any of speaking a predetermined wake word by the user, a predetermined gesture by the user, and predetermined line-of-sight movement by the user.

### Claim 15

An information processing system comprising:
a biosensor configured to sense a biological state of a user;
a microphone configured to sense a sound in space in which the user is present; and
an information processing device,
wherein the information processing device includes:
a biological data acquisition unit configured to acquire biological data from the biosensor;
a sound data acquisition unit configured to acquire sound data from the microphone;
a sleep state determination unit configured to determine a sleep state of the user based on the biological data;
a conversion unit configured to convert the sound data into text data; and
an output control unit configured to control output of a device control command based on the text data by controlling necessity of operation determination to determine whether a predetermined operation has been performed by the user based on the sleep state.

### Claim 16

A non-transitory computer-readable recording medium recording an information processing program for causing a computer to perform functions of:
acquiring biological data from a biosensor that senses a biological state of a user;
acquiring sound data from a microphone that senses sound in space in which the user is present;
determining a sleep state of the user based on the biological data;
converting the sound data into text data; and
controlling output of a device control command based on the text data by controlling necessity of operation determination to determine whether a predetermined operation has been performed by the user based on the sleep state.
