# US 12,008,700 B1 · Spatial audio and avatar control at headset using audio signals > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,700 B1 - Document ID: US12008700B1 - Status: Granted - Kind: B1 - Application number: 18/120,808 - Filed: 2023-03-13 - Granted: 2024-06-11 - Earliest claimed: 2019-08-28 - Assignees: Meta Platforms Technologies, LLC - Applicants: Meta Platforms Technologies, LLC - Inventors: Nadav Grossinger; Robert Hasbun - Web page: https://patentgrep.com/patent/US12008700B1 ## CPC - G06T 13/205: {driven by audio data} - H04S 7/304: {For headphones} - G02B 27/0172: {characterised by optical features} - G06N 20/00: Machine learning - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G10L 21/10: Transforming into visible information - H04R 1/028: {associated with devices performing functions other than acoustics, e.g. electric candles} - H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)} - G02B 27/017: {Head mounted} - G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - H04R 27/00: Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F) - H04R 2430/20: Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic (H04R2203/12 takes precedence) - H04R 2499/15: Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops - H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04S: STEREOPHONIC SYSTEMS - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract An audio system in a local area providing an audio signal to a headset of a remote user is presented herein. The audio system identifies sounds from a human sound source in the local area, based in part on sounds detected within the local area. The audio system generates an audio signal for presentation to a remote user within a virtual representation of the local area based in part on a location of the remote user within the virtual representation of the local area relative to a virtual representation of the human sound source within the virtual representation of the local area. The audio system provides the audio signal to a headset of the remote user, wherein the headset presents the audio signal as part of the virtual representation of the local area to the remote user. ## Claims ### Claim 1 An audio system comprising: an acoustic sensor of an array of acoustic sensors, the acoustic sensor configured to: capture audio data from a human sound source, wherein at least a portion of a face of the human sound source is not within a field of view of an imaging device collocated with the human sound source; and a controller configured to: generate audio information for presentation to a remote user, wherein the audio information is presented to the remote user based at least in part on a location of the remote user, within a virtual representation of a local area, relative to a virtual representation of the human sound source, provide the audio information to a headset of the remote user, wherein the headset presents the audio information as part of the virtual representation of the local area to the remote user, predict a facial expression for the portion of the face based at least in part on the captured audio data from the human sound source, and provide, to the headset of the remote user, visual information associated with a virtual representation of the human sound source in the virtual representation of the local area, wherein the visual information is presented, to the headset of the remote user, such that the facial expression of the human sound source is in sync with the presented audio information. ### Claim 2 (depends on 1) The audio system of claim 1, wherein the acoustic sensor is part of a second headset or an earbud collocated with the human sound source. ### Claim 3 (depends on 1) The audio system of claim 1, wherein the controller is further configured to: provide, to the headset of the remote user via a network, the visual information comprising the predicted facial expression being in sync with the audio information. ### Claim 4 (depends on 1) The audio system of claim 1, wherein the facial expression comprises a lips movement of the virtual representation of the human sound source. ### Claim 5 (depends on 1) The audio system of claim 1, wherein the controller is further configured to: predict the facial expression for the portion of the face by applying a machine learning algorithm to the captured audio data from the human sound source. ### Claim 6 (depends on 1) The audio system of claim 1, wherein the controller is further configured to: selectively adjust the audio information in response to one or more inputs provided by the remote user; and provide the adjusted audio information to the headset of the remote user. ### Claim 7 (depends on 1) The audio system of claim 1, wherein the controller is further configured to: generate second audio information for presentation to the remote user within the virtual representation of the local area based at least in part on the location of the remote user, within the virtual representation of the local area, relative to a virtual representation of a second human sound source; provide the second audio information to the headset of the remote user; and responsive to one or more inputs provided by the remote user, selectively adjust the second audio information in accordance with the one or more inputs, and provide the adjusted second audio information to the headset of the remote user. ### Claim 8 (depends on 1) The audio system of claim 1, wherein the controller is further configured to: generate a plurality of versions of the captured audio data, each of the plurality of versions simulating sounds from a respective human sound source of a plurality of human sound sources, including and collocated with the human sound source, and sounds from a respective location of the respective human sound source; and identify the respective human sound source, based on matching each of the plurality of versions with the captured audio data. ### Claim 9 (depends on 1) The audio system of claim 1, wherein the controller is further configured to: determine a location of the human sound source by performing beam-steering type processing of the captured audio data; and determine the location of the remote user within the virtual representation of the local area, based at least in part on the determined location of the human sound source. ### Claim 10 (depends on 1) The audio system of claim 1, further comprising: a plurality of base stations in the local area including the acoustic sensor configured to detect sounds within the local area and the imaging device configured to image at least a portion of the local area. ### Claim 11 A method comprising: capturing, using an acoustic sensor of an array of acoustic sensors, audio data from a human sound source, wherein at least a portion of a face of the human sound source is not within a field of view of an imaging device collocated with the human sound source; generating audio information for presentation to a remote user, wherein the audio information is presented to the remote user based at least in part on a location of the remote user, within a virtual representation of a local area, relative to a virtual representation of the human sound source; providing the audio information to a headset of the remote user, wherein the headset presents the audio information as part of the virtual representation of the local area to the remote user; predicting a facial expression for the portion of the face based at least in part on the captured audio data from the human sound source; and providing, to the headset of the remote user, visual information associated with a virtual representation of the human sound source in the virtual representation of the local area, wherein the visual information is presented, to the headset of the remote user, such that the facial expression of the human sound source is in sync with the presented audio information. ### Claim 12 (depends on 11) The method of claim 11, wherein the acoustic sensor is part of a second headset collocated with the human sound source. ### Claim 13 (depends on 11) The method of claim 11, wherein predicting the facial expression for the portion of the face comprises predicting a lips movement of the virtual representation of the human sound source. ### Claim 14 (depends on 11) The method of claim 11, wherein predicting the facial expression for the portion of the face comprises: predicting the facial expression by applying a machine learning algorithm to the captured audio data from the human sound source. ### Claim 15 (depends on 11) The method of claim 11, wherein further comprising: selectively adjusting the audio information in response to one or more inputs provided by the remote user; and providing the adjusted audio information to the headset of the remote user. ### Claim 16 (depends on 11) The method of claim 11, further comprising: generating second audio information for presentation to the remote user within the virtual representation of the local area based at least in part on the location of the remote user, within the virtual representation of the local area, relative to a virtual representation of a second human sound source; providing the second audio information to the headset of the remote user; and responsive to one or more inputs provided by the remote user, selectively adjusting the second audio information in accordance with the one or more inputs, and providing the adjusted second audio information to the headset of the remote user. ### Claim 17 (depends on 11) The method of claim 11, further comprising: generating a plurality of versions of the captured audio data, each of the plurality of versions simulating sounds from a respective human sound source of a plurality of human sound sources, including and collocated with the human sound source, and sounds from a respective location of the respective human sound source; and identifying the respective human sound source, based on matching each of the plurality of versions with the captured audio data. ### Claim 18 (depends on 11) The method of claim 11, further comprising: determining a location of the human sound source by performing beam-steering type processing of the captured audio data; and determining the location of the remote user within the virtual representation of the local area, based at least in part on the determined location of the human sound source. ### Claim 19 A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to: instruct an acoustic sensor of an array of acoustic sensors to capture audio data from a human sound source, wherein at least a portion of a face of the human sound source is not within a field of view of an imaging device collocated with the human sound source; generate audio information for presentation to a remote user, wherein the audio information is presented to the remote user based at least in part on a location of the remote user, within a virtual representation of a local area, relative to a virtual representation of the human sound source; provide the audio information to a headset of the remote user, wherein the headset presents the audio information as part of the virtual representation of the local area to the remote user; predict a facial expression for the portion of the face based at least in part on the captured audio data from the human sound source; and provide, to the headset of the remote user, visual information associated with a virtual representation of the human sound source in the virtual representation of the local area, wherein the visual information is presented, to the headset of the remote user, such that the facial expression of the human sound source is in sync with the presented audio information. ### Claim 20 (depends on 19) The computer program product of claim 19, wherein the instructions further cause the one or more processors to: predict the facial expression for the portion of the face by predicting a lips movement of the virtual representation of the human sound source.