# US 2024/0196152 A1 · Spatial audio processing method and apparatus therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0196152 A1
- Document ID: US20240196152A1
- Status: Published application
- Kind: A1
- Application number: 18/136,788
- Filed: 2023-04-19
- Published: 2024-06-13
- Earliest claimed: 2022-12-13
- Assignees: Gaudio Lab, Inc.
- Applicants: Gaudio Lab, Inc.
- Inventors: Hyunjoo CHUNG; Seungbum KIM; Jeonghun SEO; Jaegyu NOH; Changsub KIM; Hwanhee KIM; Hyunoh OH
- Web page: https://patentgrep.com/patent/US20240196152A1

## CPC

- H04S 7/303: {Tracking of listener position or orientation}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 40/172: {Classification, e.g. identification}
- H04S 7/304: {For headphones}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30201: Face
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S: STEREOPHONIC SYSTEMS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A spatial audio processing method may include an operation of obtaining first movement information of a video reproduction device, an operation of obtaining second movement information of an audio reproduction device, an operation of obtaining an audio signal, and an operation of performing, based on whether the first movement information satisfies a predetermined condition, spatial audio processing on the audio signal, wherein, in case that the first movement information satisfies the predetermined condition, the spatial audio processing may be performed based on the second movement information, and in case that the first movement information does not satisfy the predetermined condition, the spatial audio processing may be performed based on the first movement information and the second movement information.

## Claims

### Claim 1

A spatial audio processing method, the method comprising:
obtaining first movement information of a video reproduction device;
obtaining second movement information of an audio reproduction device;
obtaining an audio signal; and
performing spatial audio processing on the audio signal based on whether the first movement information satisfies a predetermined condition,
wherein, when the first movement information satisfies the predetermined condition, the spatial audio processing is performed based on the second movement information, and
wherein, when the first movement information does not satisfy the predetermined condition, the spatial audio processing is performed based on the first movement information and the second movement information.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the first movement information comprises at least one of spatial information and direction information of the video reproduction device, and
wherein the second movement information comprises at least one of spatial information and direction information of the audio reproduction device.

### Claim 3 (depends on 2)

The method of claim 2,
wherein the first movement information is obtained by an inertial measurement unit (IMU) of the video reproduction device,
wherein the second movement information is obtained by an IMU of the audio reproduction device, and
wherein each of the IMU of the video reproduction device and the IMU of the audio reproduction device comprises at least one of an acceleration sensor, an angular velocity sensor (gyroscope), and a geomagnetic sensor (magnetometer).

### Claim 4 (depends on 3)

The method of claim 3,
wherein the predetermined condition is a case in which a quaternion value of the video reproduction device is greater than a predetermined value, and
wherein the quaternion value is obtained based on a value obtained from at least one of the acceleration sensor, the angular velocity sensor, and the geomagnetic sensor of the video reproduction device.

### Claim 5 (depends on 3)

The method of claim 3,
wherein the predetermined condition satisfies at least one of a case in which an acceleration obtained via the acceleration sensor of the video reproduction device is greater than a first value, a case in which a variation in an angular velocity obtained via the angular velocity sensor of the video reproduction device is greater than a second value, and a case in which a variation in a magnetic field direction obtained via the geomagnetic sensor is greater than a third value.

### Claim 6 (depends on 3)

The method of claim 3,
wherein the predetermined condition is a case in which the video reproduction device is determined as moving in a predetermined pattern repeatedly based on the first movement information.

### Claim 7 (depends on 5)

The method of claim 5,
wherein the first value, the second value, and the third value are values configured by learning information associated with a movement of a user of the video reproduction device and the audio reproduction device, and
wherein the learning is performed via machine learning.

### Claim 8 (depends on 3)

The method of claim 3,
wherein the predetermined condition is a case in which the video reproduction device is located beyond a range of an angle of field of the user of the video reproduction device and the audio reproduction device, and
wherein the range of the angle of field of the user is determined based on the audio reproduction device.

### Claim 9 (depends on 6)

The method of claim 6,
wherein the predetermined pattern corresponds to the first movement information repeated during a predetermined period of time.

### Claim 10 (depends on 3)

The method of claim 3,
wherein the video reproduction device comprises at least one of a sensor that recognizes a face of a user of the video reproduction device and the audio reproduction device, a sensor that recognizes a direction of a line of sight of the user, and a sensor that recognizes a direction of the face of the user, and
wherein the predetermined condition satisfies at least one of a case in which the face of the user is not recognized, a case in which the direction of the face of the user is not toward a display of the video reproduction device that displays a video, and a case in which the direction of the line of sight of the user is not toward the display.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
obtaining information associated with whether the display of the video reproduction device that displays a video is activated,
in wherein the predetermined condition is a case in which the display is deactivated.

### Claim 12

A spatial audio processing apparatus for performing spatial audio processing, the apparatus comprising:
a processor,
wherein the processor is configured to:
obtain first movement information of a video reproduction device;
obtain second movement information of an audio reproduction device;
obtain an audio signal; and
perform spatial audio processing on the audio signal based on whether the first movement information satisfies a predetermined condition,
wherein the spatial audio processing is performed based on the second information in case that the first movement information satisfies the predetermined condition,
wherein the spatial audio processing is performed based on the first movement information and the second movement information in case that the first movement information does not satisfy the predetermined condition, and
wherein the spatial audio processing apparatus is the video reproduction device or the audio reproduction device.

### Claim 13 (depends on 11)

The apparatus of claim 11,
wherein the first movement information comprises at least one of spatial information and direction information of the video reproduction device, and
wherein the second movement information comprises at least one of spatial information and direction information of the audio reproduction device.

### Claim 14 (depends on 13)

The apparatus of claim 13,
wherein the first movement information is obtained via an inertial measurement unit (IMU) of the video reproduction device,
wherein the second movement information is obtained via an IMU of the audio reproduction device, and
wherein each of the IMU of the video reproduction device and the IMU of the audio reproduction device comprises at least one of an acceleration sensor, an angular velocity sensor (gyroscope), and a geomagnetic sensor (magnetometer).

### Claim 15 (depends on 14)

The apparatus of claim 14,
wherein the predetermined condition is a case in which a quaternion value associated with the video reproduction device is greater than a predetermined value, and
wherein the quaternion value is obtained based on a value obtained from at least one of the acceleration sensor, the angular velocity sensor, and the geomagnetic sensor of the video reproduction device.

### Claim 16 (depends on 14)

The apparatus of claim 14,
wherein the predetermined condition satisfies at least one of a case in which an acceleration obtained via the acceleration sensor of the video reproduction device is greater than a first value, a case in which a variation in an angular velocity obtained via the angular velocity sensor of the video reproduction device is greater than a second value, and a case in which a variation in a magnetic field direction obtained via the geomagnetic sensor is greater than a third value.

### Claim 17 (depends on 14)

The apparatus of claim 14,
wherein the predetermined condition is a case in which the video reproduction device is determined as moving in a predetermined pattern repeatedly based on the first movement information.

### Claim 18 (depends on 14)

The apparatus of claim 14,
wherein the predetermined condition is a case in which the video reproduction device is located beyond a range of an angle of field of the user of the video reproduction device and the audio reproduction device, and
wherein the range of the angle of field of the user is determined based on the audio reproduction device.

### Claim 19 (depends on 14)

The apparatus of claim 14,
wherein the video reproduction device comprises at least one of a sensor that recognizes a face of a user of the video reproduction device and the audio reproduction device, a sensor that recognizes a direction of a line of sight of the user, and a sensor that recognizes a direction of the face of the user, and
wherein the predetermined condition satisfies at least one of a case in which the face of the user is not recognized, a case in which the direction of the face of the user is not toward a display of the video reproduction device that displays a video, and a case in which the direction of the line of sight of the user is not toward the display.

### Claim 20 (depends on 11)

The apparatus of claim 11,
wherein the processor is configured to obtain information associated with whether the display of the video reproduction device that displays a video is activated,
wherein the predetermined condition is a case in which the display is deactivated.
