# US 2019/0378530 A1 · Voice enhancement method and apparatus of smart device and smart device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378530 A1
- Document ID: US20190378530A1
- Status: Published application
- Kind: A1
- Application number: 16/475,013
- Filed: 2018-07-05
- Published: 2019-12-12
- Earliest claimed: 2017-10-13
- Assignees: Goertek Inc.
- Applicants: Goertek Inc.
- Inventors: Jian ZHU; Xiangdong ZHANG; Zhenyu YU; Zhiping LUO; Dong YAN
- Web page: https://patentgrep.com/patent/US20190378530A1

## CPC

- G10L 21/0208: Noise filtering
- G10L 21/028: using properties of sound source
- G01S 5/20: Position of source determined by a plurality of spaced direction-finders
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 21/0216: characterised by the method used for estimating noise
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G10L 2021/02166: {Microphone arrays; Beamforming}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A voice enhancement method and apparatus of a smart device and a smart device are disclosed. The method comprises: monitoring and collecting a voice signal sent by a user in real time; determining a direction of the user according to the voice signal; collecting a depth image in the direction of the user; determining a sound source direction of the user according to the depth image; and adjusting a beamforming direction of a microphone array on the smart device according to the sound source direction of the user, and performing enhancement processing on the voice signal.

## Claims

### Claim 1

A voice enhancement method of a smart device, comprising:
monitoring and collecting a voice signal sent by a user in real time;
determining a direction of the user according to the voice signal;
collecting a depth image in the direction of the user;
determining a sound source direction of the user according to the depth image; and
adjusting a beamforming direction of a microphone array on the smart device according to the sound source direction of the user, and performing enhancement processing on the voice signal.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
monitoring the user's movement in real time;
collecting a moving direction of the user when it is monitored that the user is moving; and
controlling the smart device to move toward the moving direction of the user, and collecting a depth image after the user moves.

### Claim 3 (depends on 1)

The method according to claim 1, wherein
using the microphone array to monitor and collect the voice signal sent by a user in real time;
using a depth camera to collect the depth image in the direction of the user; and
the step of determining a sound source direction of the user according to the depth image comprising: determining a spatial position coordinate of a preset part of the user according to the depth image; and determining the sound source direction of the user according to a spatial position coordinate of the microphone array and the spatial position coordinate of the preset part of the user.

### Claim 4 (depends on 3)

The method according to claim 3, further comprising:
determining a distance between the smart device and the user according to the depth image in the direction of the user;
determining whether to control the smart device to move according to the distance between the smart device and the user; and
if yes, controlling the smart device to move toward the direction of the user to shorten the distance between the smart device and the user.

### Claim 5 (depends on 4)

The method according to claim 4, wherein
the step of determining a distance between the smart device and the user according to the depth image in the direction of the user comprises: determining a spatial position coordinate of a preset part of the user according to the depth image in the direction of the user; and determining a distance between the smart device and the preset part of the user according to the spatial position coordinate of the microphone array and the spatial position coordinate of the preset part of the user; and
the step of determining whether to control the smart device to move according to the distance between the smart device and the user comprising: if a distance between the smart device and a preset part of the user is greater than a preset distance threshold, controlling the smart device to move.

### Claim 6 (depends on 3)

The method according to claim 3, wherein the preset part of the user is a head or neck of the user.

### Claim 7

A voice enhancement apparatus of a smart device, comprising:
a voice signal collecting unit for monitoring and collecting a voice signal sent by a user in real time;
a user direction determining unit for determining a direction of the user according to the voice signal;
a depth image collecting unit for collecting a depth image in the direction of the user;
a sound source direction determining unit for determining a sound source direction of the user according to the depth image; and
an enhancement processing unit for adjusting a beamforming direction of a microphone array on the smart device according to the sound source direction of the user, and performing enhancement processing on the voice signal.

### Claim 8 (depends on 7)

The apparatus according to claim 7, further comprising:
a movement monitoring unit for monitoring the user's movement in real time;
a moving direction collecting unit for collecting a moving direction of the user when it is monitored that the user is moving; a movement controlling unit for controlling the smart device to move toward the moving direction of the user; and
the depth image collecting unit is further for collecting a depth image after the user moves.

### Claim 9 (depends on 7)

The apparatus according to claim 7, wherein the sound source direction determining unit is particularly for:
determining a spatial position coordinate of a preset part of the user according to the depth image; and
determining the sound source direction of the user according to a spatial position coordinate of the microphone array and the spatial position coordinate of the preset part of the user.

### Claim 10 (depends on 9)

The apparatus according to claim 9, further comprising:
a distance determining unit for determining a distance between the smart device and the user according to the depth image in the direction of the user;
a judging unit for determining to control the smart device to move when a distance between the smart device and a preset part of the user is greater than a preset distance threshold; and
a movement controlling unit for controlling the smart device to move toward the direction of the user to shorten the distance between the smart device and the user when it is determined to control the smart device to move.

### Claim 11

A smart device, comprising a memory and a processor which are communicatively connected by an internal bus, wherein the smart device further comprises a voice collector and a depth camera that are connected to the processor respectively;
the voice collector monitors and collects in real time a voice signal sent by a user, and sends the voice signal to the processor;
the depth camera collects a depth image in the direction of the user, and sends the depth image to the processor; the memory stores program instructions executable by the processor, and when executed by the processor, the program instructions is able to implement the steps of:
determining a direction of the user according to the voice signal received;
receiving a depth image in the direction of the user, and determining a sound source direction of the user according to the depth image; and
adjusting a beamforming direction of a microphone array on the smart device according to the sound source direction of the user, and performing enhancement processing on the voice signal.

### Claim 12 (depends on 11)

The smart device according to claim 11, wherein the depth camera is further for monitoring the user's movement in real time, and sending the user's movement to the processor;
when executed by the processor, the program instructions further implement the steps of:
according to the received movement state of the user, collecting a moving direction of the user when it is monitored that the user is moving; and
controlling the smart device to move toward the moving direction of the user, and collecting a depth image after the user moves.

### Claim 13 (depends on 11)

The smart device according to claim 11, wherein the step of determining a sound source direction of the user according to the depth image by the processor comprises:
determining a spatial position coordinate of a preset part of the user according to the depth image; and
determining the sound source direction of the user according to a spatial position coordinate of the microphone array and the spatial position coordinate of the preset part of the user.

### Claim 14 (depends on 13)

The smart device according to claim 13, wherein when executed by the processor, the program instructions further implement the steps of:
determining a distance between the smart device and the user according to the depth image in the direction of the user;
determining whether to control the smart device to move according to the distance between the smart device and the user; and
if yes, controlling the smart device to move toward the direction of the user to shorten the distance between the smart device and the user.

### Claim 15 (depends on 14)

The smart device according to claim 14, wherein
the step of determining a distance between the smart device and the user according to the depth image in the direction of the user by the processor comprises: determining a spatial position coordinate of a preset part of the user according to the depth image in the direction of the user; and determining a distance between the smart device and the preset part of the user according to the spatial position coordinate of the microphone array and the spatial position coordinate of the preset part of the user; and
the step of determining whether to control the smart device to move according to the distance between the smart device and the user by the processor comprises: if a distance between the smart device and a preset part of the user is greater than a preset distance threshold, controlling the smart device to move.

### Claim 16 (depends on 11)

The smart device according to claim 11, wherein the voice collector is a microphone array.