# US 2021/0174592 A1 · Augmented reality method and device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174592 A1
- Document ID: US20210174592A1
- Status: Published application
- Kind: A1
- Application number: 16/771,120
- Filed: 2018-11-06
- Published: 2021-06-10
- Earliest claimed: 2017-12-11
- Assignees: Beijing Jingdong Shangke Information Technology Co., Ltd.; Beijing Jingdong Century Trading Co., Ltd.
- Applicants: Beijing Jingdong Shangke Information Technology Co., Ltd.; Beijing Jingdong Century Trading Co., Ltd.
- Inventors: Weihua ZHANG; Jiangxu WU; Fan LI; Ganglin PENG; Hongguang ZHANG; Leifeng KONG
- Web page: https://patentgrep.com/patent/US20210174592A1

## CPC

- A63F 13/655: by importing photos, e.g. of the player
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
- G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence)
- G06F 18/23213: with fixed number of clusters, e.g. K-means clustering
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 40/166: {using acquisition arrangements}
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The invention discloses an augmented reality method and device, and relates to the field of computer technology. A specific implementation of the method includes: acquiring video information of a target, and acquiring real image information and real sound information of the target from the same; using the real image information to determine at least one image-based target state data, and using the real sound information to determine at least one sound-based target state data; fusing the image-based and sound-based target state data of the same type to obtain a target portrait data; and acquiring virtual information corresponding to the target portrait data, and superimposing the virtual information on the video information. This implementation can identify the current state of the target based on the image information and sound information of the target, and fuse the two identification results to obtain an accurate target portrait. Based on the target portrait, virtual information display matching the user status can be displayed, thereby improving augmented reality and user experience.

## Claims

### Claim 1

An augmented reality method comprising:
acquiring video information of a target;
acquiring real image information of the target and real sound information of the target from the video information;
using the real image information to determine at least one image-based target state data;
using the real sound information to determine at least one sound-based target state data;
fusing the image-based target state data and the sound-based target state data of a same type to obtain target portrait data;
acquiring virtual information corresponding to the target portrait data; and
superimposing the virtual information on the video information.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the image-based target state data includes at least one of emotion data, age data, and gender data, wherein the sound-based target state data includes at least one of emotion data, age data, and gender data, and wherein at least one of the image-based target state data and the sound-based target state data includes a judgment result and a confidence degree corresponding to the judgment result.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the real image information includes facial image information of the target; and wherein using the real image information to determine the at least one image-based target state data includes:
determining position information of a plurality of critical points from the facial image information;
performing tilt correction on the facial image information using the position information;
extracting a plurality of facial feature values in the corrected facial image information;
inputting the plurality of facial feature values into a pre-established image classification model to obtain the at least one image-based target state data.

### Claim 4 (depends on 1)

The method according to claim 1, wherein using the real sound information to determine the at least one sound-based target state data includes:
extracting a plurality of audio feature parameters in the real sound information;
performing clustering of the audio feature parameters; and
inputting the clustered audio feature parameters into a pre-established sound classification model to obtain the at least one sound-based target state data.

### Claim 5 (depends on 2)

The method according to claim 2, wherein the image-based target state data includes first state data including a first judgment result and a first confidence degree and the sound-based target state data includes second state data including a second judgment results and a second confidence degree and wherein fusing the image-based target state data and the sound-based target state data of the same type to obtain the target portrait data includes:
comparing whether the first judgment result is identical with the second judgment result;
when the comparison result indicates the first judgment result is identical to the second judgment result are identical:
detecting whether the sum of the first confidence degree and the second confidence degree is greater than a first confidence threshold; and
when the sum of the first confidence degree and the second confidence degree is greater than the first confidence threshold, determining the first judgment result or the second judgment result as the target portrait data; and
when the comparison result indicates the first judgment result is different from the second judgment result:
detecting whether a greater one of the first confidence degree and the second confidence degree is greater than a second confidence threshold; and
when the greater one of the first confidence degree and the second confidence degree is greater than the second confidence threshold, determining the judgment result corresponding to the greater one of the first confidence degree and the second confidence degree as the target portrait data.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the second confidence threshold is greater than the first confidence threshold.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the virtual information includes at least one of visual information, acoustic information, and effect information.

### Claim at position 8

-16. (canceled)

### Claim 17

An electronic apparatus, comprising:
one or more processors; and
a storage device for storing one or more programs;
wherein the one or more processors are configured, via execution of the one or more programs, to:
acquire video information of a target;
acquire real image information of the target from the video information;
acquire real sound information of the target from the video information;
use the real image information to determine at least one image-based target state data;
use the real sound information to determine at least one sound-based target state data;
fuse the image-based and sound-based target state data of a same type to obtain target portrait data;
acquire virtual information corresponding to the target portrait data; and
superimpose the virtual information on the video information.

### Claim 18 (depends on 17)

The electronic apparatus according to claim 17, wherein the image-based target state data includes at least one of emotion data, age data, and gender data, wherein the sound-based target state data includes at least one of emotion data, age data, and gender data, and wherein at least one of the image-based target state data and the sound-based target state data includes a judgment result and a confidence degree corresponding to the judgment result.

### Claim 19 (depends on 17)

The electronic apparatus according to claim 17, wherein the real image information includes facial image information of the target and wherein the one or more processors are configured to use the real image information to determine the at least one image-based target state data by:
determining position information of a plurality of critical points from the facial image information;
performing tilt correction on the facial image information using the position information;
extracting a plurality of facial feature values in the corrected facial image information; and
inputting the plurality of facial feature values into a pre-established image classification model to obtain the at least one image-based target state data.

### Claim 20 (depends on 17)

The electronic apparatus according to claim 17, wherein the one or more processors are configured to use the real sound information to determine the at least one sound-based target state data by:
extracting a plurality of audio feature parameters in the real sound information;
performing clustering of the audio feature parameters; and
inputting the clustered audio feature parameters into a pre-established sound classification model to obtain the at least one sound-based target state data.

### Claim 21 (depends on 18)

The electronic apparatus according to claim 18, wherein the image-based target state data includes first state data including a first judgment result and a first confidence degree and the sound-based target state data includes second state data including a second judgment results and a second confidence degree and wherein the one or more processors are configured to fuse the image-based target state data and the sound-based target state data of the same type to obtain the target portrait data by:
comparing whether the first judgment result is identical with the second judgment result:
when the comparison result indicates the first judgment result is identical to the second judgment result are identical,
detecting whether the sum of the first confidence degree and the second confidence degree is greater than a first confidence threshold; and
when the sum of the first confidence degree and the second confidence degree is greater than the first confidence threshold, determining the first judgment result or the second judgment result as the target portrait data; and
when the comparison result indicates the first judgment result is different from the second judgment result,
detecting whether a greater one of the first confidence degree and the second confidence degree is greater than a second confidence threshold; and
when the greater one of the first confidence degree and the second confidence degree is greater than the second confidence threshold, determining the judgment result corresponding to the greater one of the first confidence degree and the second confidence degree as the target portrait data.

### Claim 22 (depends on 21)

The electronic apparatus according to claim 21, wherein the second confidence threshold is greater than the first confidence threshold.

### Claim 23 (depends on 17)

The electronic apparatus according to claim 17, wherein the virtual information includes at least one of visual information, acoustic information, and effect information.

### Claim 24

A computer readable storage medium having a computer program stored thereon executable by a processor to perform a set of functions, the set of functions comprising:
acquiring video information of a target;
acquiring real image information of the target from the video information;
acquiring real sound information of the target from the video information;
using the real image information to determine at least one image-based target state data;
using the real sound information to determine at least one sound-based target state data;
fusing the image-based and sound-based target state data of a same type to obtain target portrait data;
acquiring virtual information corresponding to the target portrait data; and
superimposing the virtual information on the video information.

### Claim 25 (depends on 24)

The computer readable storage medium according to claim 24, wherein the image-based target state data includes at least one of emotion data, age data, and gender data, wherein the sound-based target state data includes at least one of emotion data, age data, and gender data, and wherein at least one of the image-based target state data and the sound-based target state data includes a judgment result and a confidence degree corresponding to the judgment result.

### Claim 26 (depends on 24)

The computer readable storage medium according to claim 24, wherein the real image information includes facial image information of the target and wherein using the real image information to determine the at least one image-based target state data includes:
determining position information of a plurality of critical points from the facial image information;
performing tilt correction on the facial image information using the position information;
extracting a plurality of facial feature values in the corrected facial image information;
inputting the plurality of facial feature values into a pre-established image classification model to obtain the at least one image-based target state data.

### Claim 27 (depends on 24)

The computer readable storage medium according to claim 24, wherein using the real sound information to determine at least one sound-based target state data includes:
extracting a plurality of audio feature parameters in the real sound information;
performing clustering of the audio feature parameters; and
inputting the clustered audio feature parameters into a pre-established sound classification model to obtain the at least one sound-based target state data.

### Claim 28 (depends on 25)

The computer readable storage medium according to claim 25, wherein the image-based target state data includes first state data including a first judgment result and a first confidence degree and the sound-based target state data includes second state data including a second judgment results and a second confidence degree and wherein fusing the image-based target state data and the sound-based target state data of the same type to obtain the target portrait data includes:
comparing whether the first judgment result is identical with the second judgment result:
when the comparison result indicates the first judgment result is identical to the second judgment result are identical,
detecting whether the sum of the first confidence degree and the second confidence degree is greater than a first confidence threshold; and
when the sum of the first confidence degree and the second confidence degree is greater than the first confidence threshold, determining the first judgment result or the second judgment result as the target portrait data; and
when the comparison result indicates the first judgment result is different from the second judgment result,
detecting whether a greater one of the first confidence degree and the second confidence degree is greater than a second confidence threshold; and
when the greater one of the first confidence degree and the second confidence degree is greater than the second confidence threshold, determining the judgment result corresponding to the greater one of the first confidence degree and the second confidence degree as the target portrait data.

### Claim 29 (depends on 28)

The computer readable storage medium according to claim 28, wherein the second confidence threshold is greater than the first confidence threshold.
