# US 2024/0004606 A1 · Audio playback method and apparatus, computer readable storage medium, and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004606 A1
- Document ID: US20240004606A1
- Status: Published application
- Kind: A1
- Application number: 18/247,754
- Filed: 2022-02-14
- Published: 2024-01-04
- Earliest claimed: 2021-04-16
- Assignees: Shenzhen Horizon Robotics Technology Co., Ltd.
- Applicants: Shenzhen Horizon Robotics Technology Co., Ltd.
- Inventors: Changbao ZHU; Jianwei NIU; Kai YU
- Web page: https://patentgrep.com/patent/US20240004606A1

## CPC

- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G10L 25/54: for retrieval
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06V 20/50: Context or environment of the image
- G06V 40/166: {using acquisition arrangements}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/176: {Dynamic expression}
- G10H 1/0008: {Associated control or indicating means}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 21/028: using properties of sound source
- G10L 25/63: for estimating an emotional state
- G10H 2210/056: for extraction or identification of individual instrumental parts, e.g. melody, chords, bass; Identification or separation of instrumental parts by their characteristic voices or timbres
- G10H 2210/066: for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
- G10H 2240/085: Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
- G10H 2240/131: Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
- G10L 21/0272: Voice signal separating
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE

## Abstract

Disclosed in the embodiments of the present disclosure are an audio playback method and apparatus, a computer readable storage medium, and an electronic device. The method includes: acquiring intention determination data collected for at least one user within a target space; determining that at least one user has a target vocal intention based on the intention determination data, and then determining feature information representing a current feature of the at least one user; and extracting and playing an audio corresponding to the feature information from a preset audio library. According to the embodiments of the present disclosure, it is achieved to automatically perform a determination on the target vocal intention of the user by the electronic device, without triggering an audio playing operation by the user, the steps of performing the audio playing operation by the user being omitted, and the convenience of the audio playing operation being improved. In addition, by determining the current feature of the user, the played audio is adapted to the feature of the user so as to achieve an effect of more accurately playing the audio which the user wants to listen to and improve pertinence of automatic playback of the audio.

## Claims

### Claim 1

An audio playback method, including:
acquiring intention determination data collected for at least one user within a target space;
determining a target vocal intention which the at least one user has based on the intention determination data;
determining feature information representing a current feature of the at least one user based on the target vocal intention; and
extracting and playing an audio corresponding to the feature information from a preset audio library.

### Claim 2 (depends on 1)

The method according to claim 1, after extracting and playing the audio corresponding to the feature information, the method further includes:
extracting user audio information from a current mixed sound signal; and
playing the user audio information in a case that the user audio information meets a preset condition.

### Claim 3 (depends on 2)

The method according to claim 2, after playing the user audio information, the method further includes:
determining a target user corresponding to the user audio information from the at least one user and acquiring a face image of the target user;
inputting the face image of the target user corresponding to the user audio information into a pre-trained first emotion recognition model to obtain emotion category information corresponding to the target user;
determining a first score representing a matching degree of an emotion of the target user corresponding to the user audio information and a type of the currently played audio based on the emotion category information, and/or
determining a second score representing a matching degree of the user audio information and the currently played audio based on the user audio information; and
determining and outputting a score of the user audio information based on the first score and/or the second score.

### Claim 4 (depends on 2)

The method according to claim 2, after playing the user audio information, the method further including:
determining a target user corresponding to the user audio information from the at least one user and acquiring a face image of the target user;
inputting the face image of the target user corresponding to the user audio information and the user audio information into a pre-trained second emotion recognition model to obtain emotion category information; and
determining and outputting a score representing a matching degree of an emotion of the target user corresponding to the user audio information and a type of the currently played audio based on the emotion category information.

### Claim 5 (depends on 4)

The method according to claim 4, wherein inputting the face image of the target user corresponding to the user audio information and the user audio information into the pre-trained second emotion recognition model to obtain the emotion category information includes:
inputting the face image of the target user corresponding to the user audio information and the user audio information into the second emotion recognition model to obtain a third emotion category information sequence, wherein the emotion category information in the third emotion category information sequence respectively corresponds to one face image sub-sequence; and
determining the score representing the matching degree of the emotion of the target user corresponding to the user audio information and the type of the currently played audio based on the emotion category information includes:
acquiring a video corresponding to the currently played audio, and extracting a face image sequence of a target person from the video;
inputting the face image sequence and the currently played audio into the second emotion recognition model to obtain a fourth emotion category information sequence;
determining a similarity between the third emotion category information sequence and the fourth emotion category information sequence; and
determining the score representing the matching degree of the emotion of the user corresponding to the user audio information and the type of the currently played audio based on the similarity.

### Claim 6 (depends on 2)

The method according to claim 2, wherein extracting the user audio information from the current mixed sound signal includes:
acquiring initial audio information collected by an audio collection device arranged in the target space, the initial audio information including the mixed sound signal; and
performing human voice separation on the initial audio information to obtain at least one path of user audio information, wherein the at least one path of user audio information corresponds to one user, respectively.

### Claim 7 (depends on 2)

The method according to claim 2, wherein playing the user audio information based on the user audio information includes:
performing melody recognition on the user audio information to obtain user melody information; and performing matching on the user melody information and melody information of the currently played audio and playing the user audio information based on an obtained first matching result, and/or
performing speech recognition on the user audio information to obtain a speech recognition result; and performing matching on the speech recognition result and corresponding text information of the currently played audio, and playing the user audio information based on an obtained second matching result.

### Claim 8 (depends on 1)

The method according to claim 1, wherein determining the target vocal intention which the at least one user has based on the intention determination data includes:
in response to a determination that the intention determination data includes a face image of the at least one user, inputting the face image into a pre-trained third emotion recognition model to obtain emotion category information, wherein, if the emotion category information is preset emotion type information, it is determined that the at least one user has the target vocal intention; or
in response to a determination that the intention determination data includes sound information of the at least one user, performing speech recognition on the sound information to obtain a speech recognition result, wherein, if the speech recognition result represents that the at least one user commands to play the audio, it is determined that the at least one user has the target vocal intention; or
in response to a determination that the intention determination data includes sound information of the at least one user, performing melody recognition on the sound information to obtain a melody recognition result, wherein, if the melody recognition result represents that the at least one user currently is producing voice in a target form, it is determined that the at least one user has the target vocal intention.

### Claim 9 (depends on 1)

The method according to claim 1, wherein determining the feature information representing the current feature of the at least one user includes:
acquiring a historical audio playback record for the at least one user; determining listening habit information of the at least one user based on the historical audio playback record; and determining the feature information based on the listening habit information; and/or
acquiring a face image of the at least one user, and inputting the face image into a pre-trained fourth emotion recognition model to obtain emotion category information representing a current emotion of the at least one user; and determining the feature information based on the emotion category information; and/or
acquiring an environment image of an environment where the at least one user is located, and inputting the environment image into a pre-trained environment recognition model to obtain environment type information; and determining the feature information based on the environment type information; and/or
acquiring an intra-space image obtained by shooting the target space; determining a number of people in the target space based on the intra-space image; and determining the feature information based on the number of people.

### Claim 10

(canceled)

### Claim 11 (depends on 1)

A non-transitory computer readable storage medium, in which a computer program is stored, the computer program being for executing the method according to claim 1.

### Claim 12 (depends on 1)

An electronic device, including:
a processor; and
a memory for storing a processor executable instruction,
the processor being configured for reading and executing the executable instruction from the memory to implement the method according to claim 1.

### Claim 13 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, after extracting and playing the audio corresponding to the feature information, the method further includes:
extracting user audio information from a current mixed sound signal; and
playing the user audio information in a case that the user audio information meets a preset condition.

### Claim 14 (depends on 13)

The non-transitory computer readable storage medium according to claim 13, after playing the user audio information, the method further includes:
determining a target user corresponding to the user audio information from the at least one user and acquiring a face image of the target user;
inputting the face image of the target user corresponding to the user audio information into a pre-trained first emotion recognition model to obtain emotion category information corresponding to the target user;
determining a first score representing a matching degree of an emotion of the target user corresponding to the user audio information and a type of the currently played audio based on the emotion category information, and/or
determining a second score representing a matching degree of the user audio information and the currently played audio based on the user audio information; and
determining and outputting a score of the user audio information based on the first score and/or the second score.

### Claim 15 (depends on 13)

The non-transitory computer readable storage medium according to claim 13, after playing the user audio information, the method further including:
determining a target user corresponding to the user audio information from the at least one user and acquiring a face image of the target user;
inputting the face image of the target user corresponding to the user audio information and the user audio information into a pre-trained second emotion recognition model to obtain emotion category information; and
determining and outputting a score representing a matching degree of an emotion of the target user corresponding to the user audio information and a type of the currently played audio based on the emotion category information.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein inputting the face image of the target user corresponding to the user audio information and the user audio information into the pre-trained second emotion recognition model to obtain the emotion category information includes:
inputting the face image of the target user corresponding to the user audio information and the user audio information into the second emotion recognition model to obtain a third emotion category information sequence, wherein the emotion category information in the third emotion category information sequence respectively corresponds to one face image sub-sequence; and
determining the score representing the matching degree of the emotion of the target user corresponding to the user audio information and the type of the currently played audio based on the emotion category information includes:
acquiring a video corresponding to the currently played audio, and extracting a face image sequence of a target person from the video;
inputting the face image sequence and the currently played audio into the second emotion recognition model to obtain a fourth emotion category information sequence;
determining a similarity between the third emotion category information sequence and the fourth emotion category information sequence; and
determining the score representing the matching degree of the emotion of the user corresponding to the user audio information and the type of the currently played audio based on the similarity.

### Claim 17 (depends on 13)

The non-transitory computer readable storage medium according to claim 13, wherein extracting the user audio information from the current mixed sound signal includes:
acquiring initial audio information collected by an audio collection device arranged in the target space, the initial audio information including the mixed sound signal; and
performing human voice separation on the initial audio information to obtain at least one path of user audio information, wherein the at least one path of user audio information corresponds to one user, respectively.

### Claim 18 (depends on 13)

The non-transitory computer readable storage medium according to claim 13, wherein playing the user audio information based on the user audio information includes:
performing melody recognition on the user audio information to obtain user melody information; and performing matching on the user melody information and melody information of the currently played audio and playing the user audio information based on an obtained first matching result, and/or
performing speech recognition on the user audio information to obtain a speech recognition result; and performing matching on the speech recognition result and corresponding text information of the currently played audio, and playing the user audio information based on an obtained second matching result.

### Claim 19 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, wherein determining the target vocal intention which the at least one user has based on the intention determination data includes:
in response to a determination that the intention determination data includes a face image of the at least one user, inputting the face image into a pre-trained third emotion recognition model to obtain emotion category information, wherein, if the emotion category information is preset emotion type information, it is determined that the at least one user has the target vocal intention; or
in response to a determination that the intention determination data includes sound information of the at least one user, performing speech recognition on the sound information to obtain a speech recognition result, wherein, if the speech recognition result represents that the at least one user commands to play the audio, it is determined that the at least one user has the target vocal intention; or
in response to a determination that the intention determination data includes sound information of the at least one user, performing melody recognition on the sound information to obtain a melody recognition result, wherein, if the melody recognition result represents that the at least one user currently is producing voice in a target form, it is determined that the at least one user has the target vocal intention.

### Claim 20 (depends on 11)

The non-transitory computer readable storage medium according to claim 11, wherein determining the feature information representing the current feature of the at least one user includes:
acquiring a historical audio playback record for the at least one user; determining listening habit information of the at least one user based on the historical audio playback record; and determining the feature information based on the listening habit information; and/or
acquiring a face image of the at least one user, and inputting the face image into a pre-trained fourth emotion recognition model to obtain emotion category information representing a current emotion of the at least one user; and determining the feature information based on the emotion category information; and/or
acquiring an environment image of an environment where the at least one user is located, and inputting the environment image into a pre-trained environment recognition model to obtain environment type information; and determining the feature information based on the environment type information; and/or
acquiring an intra-space image obtained by shooting the target space; determining a number of people in the target space based on the intra-space image; and determining the feature information based on the number of people.

### Claim 21 (depends on 12)

The electronic device according to claim 12, after extracting and playing the audio corresponding to the feature information, the method further includes:
extracting user audio information from a current mixed sound signal; and
playing the user audio information in a case that the user audio information meets a preset condition.
