# US 2023/0066616 A1 · User interaction method and apparatus, device and medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0066616 A1
- Document ID: US20230066616A1
- Status: Published application
- Kind: A1
- Application number: 17/420,733
- Filed: 2020-12-01
- Published: 2023-03-02
- Earliest claimed: 2020-05-21
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science and Technology Co., Ltd.
- Inventors: Xiaokun ZHAO; Zhiwei QIN; Kun SONG
- Web page: https://patentgrep.com/patent/US20230066616A1

## CPC

- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06N 20/00: Machine learning
- G06V 10/82: using neural networks
- G06V 40/174: {Facial expression recognition}
- G06V 40/176: {Dynamic expression}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Provided are a user interaction method and apparatus, a device and a medium, which relates to the field of artificial intelligence. An implementation solution is as follows: acquiring current facial expression information of a user; determining a current smiling level of the user according to the current facial expression information; and performing a corresponding interaction operation according to the current smiling level.

## Claims

### Claim 1

A user interaction method, comprising:
acquiring current facial expression information of a user;
determining a current smiling level of the user according to the current facial expression information; and
performing a corresponding interaction operation according to the current smiling level.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the current smiling level of the user according to the current facial expression information comprises:
determining a current expression coefficient according to the current facial expression information; and
determining the current smiling level of the user according to the current expression coefficient.

### Claim 3 (depends on 2)

The method of claim 2, wherein determining the current smiling level of the user according to the current expression coefficient comprises:
acquiring a starting smiling threshold corresponding to the user; and
determining the current smiling level of the user according to the current expression coefficient and the starting smiling threshold.

### Claim 4 (depends on 3)

The method of claim 3, wherein acquiring the starting smiling threshold corresponding to the user comprises:
acquiring historical expression coefficients of the user;
training a smiling threshold recognition model corresponding to the user by using the historical expression coefficients; and
acquiring the starting smiling threshold corresponding to the user according to a training result of the smiling threshold recognition model, wherein the training result comprises expression coefficient ranges of at least two smiling classifications.

### Claim 5 (depends on 4)

The method of claim 4, wherein acquiring the starting smiling threshold corresponding to the user according to the training result of the smiling threshold recognition model comprises:
determining a smiling classification having a smallest expression coefficient value in the training result as a normal expression classification; and
determining a maximum value in an expression coefficient range corresponding to the normal expression classification as the starting smiling threshold corresponding to the user.

### Claim 6 (depends on 3)

The method of claim 3, wherein determining the current smiling level of the user according to the current expression coefficient and the starting smiling threshold comprises:
determining level value intervals corresponding to at least two preset smiling levels according to the starting smiling threshold; and
determining the current smiling level of the user according to the current expression coefficient and the level value intervals corresponding to the at least two preset smiling levels.

### Claim 7 (depends on 2)

The method of claim 2, wherein determining the current expression coefficient according to the current facial expression information comprises:
inputting the current facial expression information into an expression recognition model to obtain the current expression coefficient outputted.

### Claim 8 (depends on 1)

The method of claim 1, wherein acquiring the current facial expression information of the user comprises:
capturing, by an augmented reality (AR) engine, facial expression data of the user in real time at a set frequency as the current facial expression information of the user.

### Claim 9 (depends on 1)

The method of claim 1, wherein performing the corresponding interaction operation comprises at least one of:
displaying a corresponding like effect picture; or
performing corresponding user feedback information statistics.

### Claim 10

A user interaction apparatus, comprising:
an expression acquisition module, which is configured to acquire current facial expression information of a user;
a level determination module, which is configured to determine a current smiling level of the user according to the current facial expression information; and
an operation performing module, which is configured to perform a corresponding interaction operation according to the current smiling level.

### Claim 11 (depends on 1)

An electronic device, comprising:
at least one processor; and
a memory in communication connection with the at least one processor,
wherein the memory is configured to store an instruction executable by the at least one processor, and the instruction is executed by the at least one processor to cause the at least one processor to perform the method of claim 1.

### Claim 12 (depends on 1)

A non-transient computer-readable storage medium storing a computer instruction for causing a computer to perform the method of claim 1.

### Claim 13

The electronic device of 11, wherein determining the current smiling level of the user according to the current facial expression information comprises:
determining a current expression coefficient according to the current facial expression information; and
determining the current smiling level of the user according to the current expression coefficient.

### Claim 14

The electronic device of 13, wherein determining the current smiling level of the user according to the current expression coefficient comprises:
acquiring a starting smiling threshold corresponding to the user; and
determining the current smiling level of the user according to the current expression coefficient and the starting smiling threshold.

### Claim 15

The electronic device of 14, wherein acquiring the starting smiling threshold corresponding to the user comprises:
acquiring historical expression coefficients of the user;
training a smiling threshold recognition model corresponding to the user by using the historical expression coefficients; and
acquiring the starting smiling threshold corresponding to the user according to a training result of the smiling threshold recognition model, wherein the training result comprises expression coefficient ranges of at least two smiling classifications.

### Claim 16

The electronic device of 15, wherein acquiring the starting smiling threshold corresponding to the user according to the training result of the smiling threshold recognition model comprises:
determining a smiling classification having a smallest expression coefficient value in the training result as a normal expression classification; and
determining a maximum value in an expression coefficient range corresponding to the normal expression classification as the starting smiling threshold corresponding to the user.

### Claim 17

The electronic device of 14, wherein determining the current smiling level of the user according to the current expression coefficient and the starting smiling threshold comprises:
determining level value intervals corresponding to at least two preset smiling levels according to the starting smiling threshold; and
determining the current smiling level of the user according to the current expression coefficient and the level value intervals corresponding to the at least two preset smiling levels.

### Claim 18

The electronic device of 13, wherein determining the current expression coefficient according to the current facial expression information comprises:
inputting the current facial expression information into an expression recognition model to obtain the current expression coefficient outputted.

### Claim 19

The electronic device of 11, wherein acquiring the current facial expression information of the user comprises:
capturing, by an augmented reality (AR) engine, facial expression data of the user in real time at a set frequency as the current facial expression information of the user.

### Claim 20

The electronic device of 11, wherein performing the corresponding interaction operation comprises at least one of:
displaying a corresponding like effect picture; or
performing corresponding user feedback information statistics.
