# US 2022/0277481 A1 · Panoramic video processing method and apparatus, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277481 A1
- Document ID: US20220277481A1
- Status: Published application
- Kind: A1
- Application number: 17/618,398
- Filed: 2020-06-05
- Published: 2022-09-01
- Earliest claimed: 2019-06-10
- Assignees: Beijing Bytedance Network Technology Co., Ltd.
- Applicants: Beijing Bytedance Network Technology Co., Ltd.
- Inventors: Hequn BAI
- Web page: https://patentgrep.com/patent/US20220277481A1

## CPC

- G06T 7/75: {involving models}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/248: {involving reference images or patches}
- G06T 7/66: of image moments or centre of gravity
- G06T 7/74: {involving reference images or patches}
- H04N 5/265: Mixing
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/234: Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N 21/81: Monomedia components thereof
- H04S 7/30: {Control circuits for electronic adaptation of the sound field}
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04S: STEREOPHONIC SYSTEMS

## Abstract

A method and device for processing a panoramic video, and a non-transitory storage medium are provided. The method includes: determining a reference frame in video frames of the panoramic video, and determining a target object in the reference frame; acquiring position information about a position of the target object in the reference frame; acquiring a motion track of the target object in the panoramic video based on the position information; and processing a to-be-processed audio frame based on the motion track to obtain a target audio frame capable of characterizing a position of the target object in the panoramic video, where the target audio frame is used to be synthesized with the video frames of the panoramic video to obtain a panoramic video file.

## Claims

### Claim 1

A method for processing a panoramic video, comprising:
determining a reference frame in video frames of the panoramic video, and determining a target object in the reference frame;
acquiring position information about a position of the target object in the reference frame;
acquiring a motion track of the target object in the panoramic video based on the position information; and
processing a to-be-processed audio frame based on the motion track to obtain a target audio frame capable of characterizing a position of the target object in the panoramic video, wherein the target audio frame is used to be synthesized with the video frames of the panoramic video to obtain a panoramic video file.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the acquiring position information about a position of the target object in the reference frame comprises:
identifying a plurality of feature points of the target object in the reference frame to obtain a feature point set corresponding to the target object; and
acquiring the position information about the position of the target object in the reference frame based on the feature points in the feature point set, wherein
the position information comprises: angle information about the position of the target object in the panoramic video; and a center distance about the position of the target object in the panoramic video, wherein the center distance is a distance between the position of the target object in the panoramic video and a center of a three-dimensional model corresponding to the reference frame.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the acquiring the position information about the position of the target object in the reference frame based on the feature points in the feature point set comprises:
determining a centre-of-gravity position of the plurality of feature points in the feature point set;
determining the angle information about the position of the target object in the panoramic video based on the centre-of-gravity position;
acquiring a pixel area of a region occupied by the plurality of feature points in the reference frame; and
determining the center distance about the position of the target object in the panoramic video based on the pixel area.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the determining the angle information about the position of the target object in the panoramic video based on the centre-of-gravity position comprises:
determining a pixel point coordinate of a pixel point corresponding to the centre-of-gravity position; and
converting the pixel point coordinate into a spherical coordinate in the three-dimensional model, wherein the spherical coordinate indicates an angle corresponding to the position of the target object in the panoramic video.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the determining the center distance about the position of the target object in the panoramic video based on the pixel area comprises:
determining the center distance about the position of the target object in the panoramic video based on an inverse relationship between the pixel area and a square of the center distance.

### Claim 6 (depends on 2)

The method according to claim 2, further comprising:
performing feature point matching on the plurality of feature points in the feature point set and feature points in a non-reference frame of the panoramic video to obtain a feature point matching result; and
determining a position of the target object in the non-reference frame based on the feature point matching result.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the determining a position of the target object in the non-reference frame based on the feature point matching result comprises:
selecting, based on the feature point matching result, a feature point in the non-reference frame a similarity between which and one of the feature points in the feature point set is greater than a first similarity threshold; and
determining the position of the target object in the non-reference frame based on the selected feature point in the non-reference frame the similarity between which and one of the feature points in the feature point set is greater than the first similarity threshold.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the acquiring a motion track of the target object in the panoramic video based on the position information comprises:
acquiring position information about a position of the target object in each non-reference frame of the panoramic video; and
determining the motion track of the target object in the panoramic video based on the position information about the position of the target object in the reference frame and the position information about a position of the target object in each non-reference frame.

### Claim 9 (depends on 6)

The method according to claim 6, further comprising:
selecting, based on the feature point matching result, a feature point in the non-reference frame a similarity between which and each of the feature points in the feature point set is less than a second similarity threshold; and
updating the feature points in the feature point set based on the selected feature point in the non-reference frame the similarity between which and each of the feature points in the feature point set is less than the second similarity threshold.

### Claim 10 (depends on 6)

The method according to claim 6, further comprising:
acquiring the feature point matching results between each of a preset number of non-reference frames and the feature points in the feature point set, wherein the preset number of non-reference frames are continuous video frames of the panoramic video;
determining a to-be-deleted feature point in the feature point set based on the feature point matching results, wherein a similarity between the to-be-deleted feature point and each of the feature points in the preset number of non-reference frames is less than a third similarity threshold; and
deleting the to-be-deleted feature point from the feature point set.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the determining a target object in the reference frame comprises:
receiving an object selection operation for the reference frame; and
determining the target object in the reference frame based on a region in the reference frame selected by the object selection operation.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the processing a to-be-processed audio frame based on the motion track to obtain a target audio frame capable of characterizing a position of the target object in the panoramic video comprises:
invoking a three-dimensional audio engine to perform mixing processing on the to-be-processed audio frame based on the motion track, to obtain the target audio frame capable of characterizing the position of the target object in the panoramic video.

### Claim 13

A device for processing a panoramic video, comprising:
a memory storing executable instructions;
a processor configured to execute the executable instructions to implement operations comprising:
determining a reference frame in video frames of the panoramic video, and determine a target object in the reference frame;
acquiring position information about a position of the target object in the reference frame, and acquire a motion track of the target object in the panoramic video based on the position information; and
processing a to-be-processed audio frame based on the motion track to obtain a target audio frame capable of characterizing a position of the target object in the panoramic video, wherein the target audio frame is used to be synthesized with the video frames of the panoramic video to obtain a panoramic video file.

### Claim 14 (depends on 13)

The device according to claim 13, wherein the processor configured to execute the executable instructions to implement operations comprising:
identifying a plurality of feature points of the target object in the reference frame to obtain a feature point set corresponding to the target object; and
acquiring the position information about the position of the target object in the reference frame based on the feature points in the feature point set, wherein
the position information comprises: angle information about the position of the target object in the panoramic video; and a center distance about the position of the target object in the panoramic video, wherein the center distance is a distance between the position of the target object in the panoramic video and a center of a three-dimensional model corresponding to the reference frame.

### Claim 15 (depends on 14)

The device according to claim 14, wherein the processor configured to execute the executable instructions to implement operations comprising:
determining a centre-of-gravity position of the plurality of feature points in the feature point set;
determining the angle information about the position of the target object in the panoramic video based on the centre-of-gravity position;
acquiring a pixel area of a region occupied by the plurality of feature points in the reference frame; and
determining the center distance about the position of the target object in the panoramic video based on the pixel area.

### Claim 16 (depends on 15)

The device according to claim 15, wherein the processor configured to execute the executable instructions to implement operations comprising:
determining a pixel point coordinate of a pixel point corresponding to the centre-of-gravity position; and
converting the pixel point coordinate into a spherical coordinate in the three-dimensional model, wherein the spherical coordinate indicates an angle corresponding to the position of the target object in the panoramic video.

### Claim 17 (depends on 15)

The device according to claim 15, wherein the processor configured to execute the executable instructions to implement an operation comprising:
determining the center distance about the position of the target object in the panoramic video based on an inverse relationship between the pixel area and a square of the center distance.

### Claim 18 (depends on 14)

The device according to claim 14, further the processor configured to execute the executable instructions to implement operations comprising:
performing feature point matching on the plurality of feature points in the feature point set and feature points in a non-reference frame of the panoramic video to obtain a feature point matching result;
determining a position of the target object in the non-reference frame based on the feature point matching result.

### Claim 19 (depends on 18)

The device according to claim 18, wherein the processor configured to execute the executable instructions to implement operations comprising:
selecting, based on the feature point matching result, a feature point in the non-reference frame a similarity between which and one of the feature points in the feature point set is greater than a first similarity threshold; and
determining the position of the target object in the non-reference frame based on the selected feature point in the non-reference frame the similarity between which and one of the feature points in the feature point set is greater than the first similarity threshold.

### Claim at position 20

20-25. (canceled)

### Claim 26

A non-transitory storage medium storing executable instructions, wherein the executable instructions are used to implement operations comprising:
determining a reference frame in video frames of the panoramic video, and determining a target object in the reference frame;
acquiring position information about a position of the target object in the reference frame;
acquiring a motion track of the target object in the panoramic video based on the position information; and
processing a to-be-processed audio frame based on the motion track to obtain a target audio frame capable of characterizing a position of the target object in the panoramic video, wherein the target audio frame is used to be synthesized with the video frames of the panoramic video to obtain a panoramic video file.
