Manifold learning for sound field estimation
AssigneeAmazon Technologies, Inc.
InventorsKarim Helwani, Michael Mark Goodwin, Paris Smaragdis
Life of this patent
Estimating the term…
Abstract
System and methods are provided for estimating the sound field from partial observations. Estimating an acoustic environment for virtual reality and augmented reality applications is a step in the creation of simulated acoustic sound scenes. In particular, the impulse responses of room can be estimated with a generative model. In a teleconferencing scenario with remote participants and a group of participants in a common physical space, giving the remote participants the impression that all other participants are sitting is in the same room acoustically requires filtering the speech of the remote participants with impulse responses estimated at the desired rendering position in the conference room.
Claims20
A computer-implemented method for estimating a sound field for virtual reality or augmented reality, comprising:
- receiving, for a first position associated with a near end room, room data comprising (i) input audio data and (ii) target audio data;
- generating measurement vector data from the input audio data;
- generating initial input vector data for a second position associated with the near end room;
- generating input data from (i) the measurement vector data, (ii) the first position, (iii) the initial input vector data, and (iv) the second position;
- applying initial filter parameters to the input data that results in filtered data;
- generating target data from the target audio data;
- determining an estimated loss from the filtered data and the target data;
- determining a matrix from a decoder of a trained variational autoencoder;
- combining the matrix, the estimated loss, and a step value that results in a tangent vector;
- applying the decoder to a point in a tangent space indicated by the tangent vector, wherein the decoder outputs updated filter parameters;
- receiving far end audio data;
- in response to receiving the far end audio data, substantially in real-time:
- generating near end audio data from (i) the far end audio data, (ii) the updated filter parameters, and (iii) the second position;
- and
- outputting the near end audio data.
One or more non-transitory computer-readable storage media storing computer executable instructions that when executed by a computing system perform operations comprising:
- receiving, for a first position associated with a near end room, room data comprising (i) input audio data and (ii) target audio data;
- generating measurement vector data from the input audio data;
- generating initial input vector data for a second position associated with the near end room;
- generating input data from (i) the measurement vector data, (ii) the first position, (iii) the initial input vector data, and (iv) the second position;
- applying initial filter parameters to the input data that results in filtered data;
- generating target data from the target audio data;
- determining an estimated loss from the filtered data and the target data;
- determining a matrix from a decoder of a trained generative model;
- combining the matrix, the estimated loss, and a step value that results in a tangent vector;
- applying the decoder to a point in a tangent space indicated by the tangent vector, wherein the decoder outputs updated filter parameters;
- receiving far end audio data;
- in response to receiving the far end audio data, substantially in real-time:
- generating near end audio data from (i) the far end audio data, (ii) the updated filter parameters, and (iii) the second position;
- and
- transmitting the near end audio data.
A system comprising:
- a non-transitory data storage medium;
- and
- a computer hardware processor in communication with the non-transitory data storage medium, wherein the computer hardware processor is configured to execute computer-executable instructions to at least:
- receive, for a first position associated with a near end room, room data comprising (i) input audio data and (ii) target audio data;
- generate measurement vector data from the input audio data;
- generate initial input vector data for a second position associated with the near end room;
- generate input data from (i) the measurement vector data, (ii) the first position, (iii) the initial input vector data, and (iv) the second position;
- apply initial filter parameters to the input data that results in filtered data;
- generate target data from the target audio data;
- determine an estimated loss from the filtered data and the target data;
- determine a matrix from a decoder of a trained generative model;
- combine the matrix, the estimated loss, and a step value that results in a tangent vector;
- apply the decoder to a point in a tangent space indicated by the tangent vector, wherein the decoder outputs updated filter parameters;
- receive far end audio data;
- generate near end audio data from (i) the far end audio data, (ii) the updated filter parameters, and (iii) the second position;
- and
- transmit the near end audio data.
Dependencies are inferred from the claim text.