# US 2021/0174535 A1 · Configuration of audio reproduction system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174535 A1
- Document ID: US20210174535A1
- Status: Published application
- Kind: A1
- Application number: 16/707,148
- Filed: 2019-12-09
- Published: 2021-06-10
- Earliest claimed: 2019-12-09
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: SHARANAPPAGOUDA PATIL; KANEAKI FUJISHITA
- Web page: https://patentgrep.com/patent/US20210174535A1

## CPC

- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G02B 27/0172: {characterised by optical features}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 20/00: Machine learning
- G06T 15/005: {General purpose rendering architectures}
- H04R 29/001: {for loudspeakers (H04R29/007 takes precedence)}
- H04S 1/002: {Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution (control circuits for electronic adaptation of the sound field H04S7/30)}
- H04S 7/303: {Tracking of listener position or orientation}
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06T 2207/10012: Stereo images
- G06T 2207/20081: Training; Learning
- G06T 2210/04: Architectural design, interior design
- H04R 2420/05: Detection of connection of loudspeakers or headphones to amplifiers
- H04S 7/307: {Frequency adjustment, e.g. tone control (H04S7/301 takes precedence)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

An electronic apparatus and method for configuration of an audio reproduction system is provided. The electronic apparatus captures a set of stereo images of the listening environment and identifies a plurality of objects, including a display device and a plurality of audio devices, in the set of stereo images. The electronic apparatus estimates first location information of the plurality of audio devices and second location information of the display device. Based on first location information and the second location information, the electronic apparatus identifies a layout of the plurality of audio devices. The electronic apparatus receives an audio signal from each audio device and determines a distance between each audio device of the plurality of audio devices and a user location based on the received audio signal. The electronic apparatus determines an anomaly in connection of at least one audio device and generates connection information based on the determined anomaly.

## Claims

### Claim 1

An electronic apparatus, comprising:
control circuitry configured to:
identify a plurality of objects in a set of stereo images of a listening environment,
wherein the identified plurality of objects comprises a display device and a plurality of audio devices of an audio reproduction system;
estimate first location information of the plurality of audio devices in the listening environment based on the set of stereo images;
estimate second location information of the display device in the listening environment;
identify a layout of the plurality of audio devices in the listening environment based on estimated first location information and the estimated second location information;
receive an audio signal from each audio device of the plurality of audio devices;
determine a distance between each audio device of the plurality of audio devices and a user location in the listening environment based on the received audio signal;
determine an anomaly in connection of at least one audio device of the plurality of audio devices based on the estimated first location information, the identified layout, and the determined distance; and
generate connection information associated with the plurality of audio devices based on the determined anomaly.

### Claim 2 (depends on 1)

The electronic apparatus according to claim 1, wherein the audio reproduction system is an “M:N” surround sound system, and
wherein the plurality of audio devices comprises “M” number of speakers and “N” number of sub-woofers.

### Claim 3 (depends on 1)

The electronic apparatus according to claim 1, wherein the control circuitry is further configured to identify the plurality of objects by application of a Machine Learning (ML) model on the set of stereo images.

### Claim 4 (depends on 1)

The electronic apparatus according to claim 1, wherein the control circuitry is further configured to determine a type of listening environment based on the identified plurality of objects.

### Claim 5 (depends on 1)

The electronic apparatus according to claim 1, wherein the identified plurality of objects further comprises a seating structure in the listening environment.

### Claim 6 (depends on 5)

The electronic apparatus according to claim 5, wherein the control circuitry is further configured to:
extract a plurality of contours corresponding to the display device, the seating structure, and the plurality of audio devices,
wherein the plurality of contours is extracted based on the set of stereo images; and
output a layout map for the listening environment based on the extracted plurality of contours,
wherein the layout map is indicative of relative placement of the display device, the seating structure, and the plurality of audio devices in the listening environment.

### Claim 7 (depends on 6)

The electronic apparatus according to claim 6, wherein the control circuitry is further configured to:
receive a second user input on the output layout map, wherein the second user input is indicative of the user location in the listening environment; and
determine the user location in the listening environment based on the received second user input.

### Claim 8 (depends on 6)

The electronic apparatus according to claim 6, wherein the control circuitry is further configured to:
transmit three-dimensional (3D) information comprising the output layout map for the listening environment to a head-mounted device, wherein the head-mounted device is configured to:
receive the transmitted 3D information;
render a 3D model representing the output layout map on a display of the head-mounted device based on the received 3D information.

### Claim 9 (depends on 1)

The electronic apparatus according to claim 1, wherein the control circuitry is further configured to:
receive, via a camera device in the listening environment, an image of the listening environment;
determine a change in the user location relative to the layout of the plurality of audio devices based on user detection in the received image.

### Claim 10 (depends on 1)

The electronic apparatus according to claim 1, further comprising an image-capture device configured to capture the set of stereo images of the listening environment based on a first user input,
wherein the set of stereo images comprises a first stereo image captured from a first viewpoint of the listening environment and a second stereo image captured from a second viewpoint of the listening environment.

### Claim 11 (depends on 10)

The electronic apparatus according to claim 10, the control circuitry is further configured to:
determine a first reference location at which the image-capture device captures the first stereo image; and
determine a second reference location at which the image-capture device captures the second stereo image.

### Claim 12 (depends on 11)

The electronic apparatus according to claim 11, wherein the control circuitry is further configured to:
compute a disparity-measure of pixel information from the set of stereo images for each audio device of the plurality of audio devices; and
compute an in-image location of each audio device of the plurality of audio devices based on computed disparity-measure for the corresponding audio device of the plurality of audio devices.

### Claim 13 (depends on 12)

The electronic apparatus according to claim 12, wherein the control circuitry is further configured to:
calculate a first distance value between the computed in-image location of each audio device of the plurality of audio devices and the determined first reference location or the determined second reference location;
calculate a second distance value between a pair of audio devices of the plurality of audio devices based the calculated first distance value; and
calculate a third distance value between the determined first reference location and the determined second reference location based on the calculated first distance value and the calculated second distance value.

### Claim 14 (depends on 13)

The electronic apparatus according to claim 13, wherein the control circuitry is further configured to estimate angle information for the plurality of audio devices based on the calculated first distance value and one of the calculated second distance value or the calculated third distance value.

### Claim 15 (depends on 14)

The electronic apparatus according to claim 14, wherein the angle information is estimated with respect to a reference axis, and
wherein the estimated angle information is indicative of a specific direction in which each corresponding audio device of the plurality of audio devices is located in the listening environment with respect to the reference axis.

### Claim 16 (depends on 14)

The electronic apparatus according to claim 14, wherein the control circuitry is configured to estimate the first location information of the plurality of audio devices based on the estimated angle information and the calculated first distance value.

### Claim 17 (depends on 1)

The electronic apparatus according to claim 1, wherein the control circuitry is further configured to estimate third location information of a seating structure in the listening environment.

### Claim 18 (depends on 17)

The electronic apparatus according to claim 17, wherein the control circuitry is configured to identify the layout of the plurality of audio devices further based on the estimate third location information.

### Claim 19 (depends on 1)

The electronic apparatus according to claim 1, wherein the audio signal is received from a mono-microphone of a user device.

### Claim 20 (depends on 1)

The electronic apparatus according to claim 1, wherein the circuitry is further configured to:
estimate a Euclidean distance between the user location and a location value for each audio device of the plurality of audio devices in the estimated first location information;
compare the estimated Euclidean distance with the distance determined based on the received audio signal; and
determine the anomaly in connection of at least one audio device of the plurality of audio devices based on the comparison,
wherein the determined anomaly is one of an incorrect connection or a missing connection of the at least one audio device with an audio-video receiver (AVR) of the audio reproduction system.

### Claim 21 (depends on 1)

The electronic apparatus according to claim 1, wherein the control circuitry is further configured to:
output configuration information for calibration of the plurality of audio devices based on one or more of: the identified layout, the user location, the estimated first location information, and the generated connection information; and
communicate the output configuration information to an audio-video receiver (AVR) of the audio reproduction system.

### Claim 22 (depends on 21)

The electronic apparatus according to claim 21, wherein the configuration information comprises a plurality of fine-tuning parameters, and
wherein the plurality of fine-tuning parameters comprises a delay parameter, a level parameter, an equalization (EQ) parameter, left/right audio device layout, room environment information, or the anomaly in the connection of the at least one audio device.

### Claim 23 (depends on 21)

The electronic apparatus according to claim 21, wherein the control circuitry is further configured to:
detect a change in the user location from the listening environment to a different listening environment comprising a different audio reproduction system;
share the configuration information generated for the audio reproduction system, to the different audio reproduction system; and
configure the different audio reproduction system in the different listening environment based on the shared configuration information.

### Claim 24

A method comprising:
in an electronic apparatus:
identifying a plurality of objects in a set of stereo images of a listening environment,
wherein the identified plurality of objects comprises a display device and a plurality of audio devices of an audio reproduction system;
estimating first location information of the plurality of audio devices in the listening environment based on the set of stereo images;
estimating second location information of the display device in the listening environment;
identifying a layout of the plurality of audio devices in the listening environment based on estimated first location information and the estimated second location information;
receiving an audio signal from each audio device of the plurality of audio devices;
determining a distance between each audio device of the plurality of audio devices and a user location in the listening environment based on the received audio signal;
determining an anomaly in connection of at least one audio device of the plurality of audio devices based on the estimated first location information, the identified layout, and the determined distance; and
generating connection information associated with the plurality of audio devices based on the determined anomaly.
