# US 9,883,140 B2 · Using the location of a near-end user in a video stream to adjust audio settings of a far-end system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,883,140 B2 - Document ID: US9883140B2 - Status: Granted - Kind: B2 - Application number: 15/311,824 - Filed: 2014-09-26 - Published: 2017-04-06 - Granted: 2018-01-30 - Earliest claimed: 2014-05-19 - Applicants: Apple Inc. - Inventors: Martin E. Johnson; Tom-Davy William Jendrik Saux; Ronald N. Isaac; Sylvain Choisel; Afrooz Family - Web page: https://patentgrep.com/patent/US9883140B2 ## CPC - H04N 7/15: Conference systems - H04N 7/141: {between two video terminals, e.g. videophone (telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085)} - H04R 3/12: for distributing signals to two or more loudspeakers {(specially adapted for hearing aids H04R25/407)} - H04S 7/30: {Control circuits for electronic adaptation of the sound field} - H04S 7/305: {Electronic adaptation of stereophonic audio signals to reverberation of the listening space (H04S7/301 takes precedence)} - H04R 27/00: Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F) - H04R 2430/01: Aspects of volume control, not necessarily automatic, in sound systems - H04S 2400/13: Aspects of volume control, not necessarily automatic, in stereophonic sound systems - H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - H04S: STEREOPHONIC SYSTEMS ## Abstract A video conferencing system is described that includes a near-end and a far-end system. The near-end system records both audio and video of one or more users proximate to the near-end system. This recorded audio and video is transmitted to the far-end system through the data connection. The video stream and/or one or more settings of the recording camera are analyzed to determine the amount of a video frame occupied by the recorded user(s). The video conferencing system may directly analyze the video frames themselves and/or a zoom setting of the recording camera to determine a ratio or percentage of the video frame occupied by the recorded user(s). By analyzing video frames associated with an audio stream, the video conferencing system may drive a speaker array of the far-end system to more accurately reproduce sound content based on the position of the recorded user in a video frame. ## Claims ### Claim 1 A method for adjusting sound emitted by a far-end system, comprising: receiving, by the far-end system, a video stream and an audio stream captured by a near-end system; determining an amount of area in a first frame of the video stream occupied by a near-end audio source; determining a first set of audio parameters to apply to the audio stream based on the determined amount of area in the first frame of the video stream occupied by the near-end audio source; and playing the audio stream using the first set of audio parameters to emit sound through one or more transducers in a speaker system of the far-end system. ### Claim 2 (depends on 1) The method of claim 1, wherein the near-end audio source is a user of the near-end system captured by a near-end video camera. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the amount of area in the first frame occupied by the near-end audio source is determined based on one of (1) a zoom setting of a near-end video camera that captured the first frame and (2) an analysis of the first frame using facial recognition. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the first set of audio parameters comprises: setting a loudness parameter to a first loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the loudness parameter to a second loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first loudness level is larger than the second loudness level. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the first set of audio parameters comprises: setting a directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first directivity level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second directivity level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first directivity level is greater than the second directivity level. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the first set of audio parameters comprises: setting an equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first equalization setting upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second equalization setting upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first equalization setting filters more high frequency elements of the audio stream than the second equalization setting. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the first set of audio parameters comprises: setting a reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first reverberation level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second reverberation level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first reverberation level is less than the second reverberation level. ### Claim 8 (depends on 7) The method of claim 7, wherein playing the audio stream using the first set of audio parameters, including the reverberation parameter, comprises: detecting separately by one or more microphones in the near-end system (1) reverberant sounds and (2) direct sounds produced by the near-end audio source, wherein the audio stream includes both the reverberant sounds and direct sounds from the near-end audio source; and driving the speaker system in the far-end system to emit the reverberant sounds off axis from the direct sounds, wherein the reverberation parameter represents a loudness of reverberant sounds. ### Claim 9 (depends on 8) The method of claim 8, wherein the one or more microphones vary widths or directions of one or more associated beam patterns over time to capture the reverberant sounds and the direct sounds. ### Claim 10 (depends on 1) The method of claim 1, further comprising: determining an amount of area in a second frame of the video stream occupied by the near-end audio source; determining a second set of audio parameters to apply to the audio stream based on the determined amount of area in the second frame of the video stream occupied by the near-end audio source; determining one or more intermediary audio parameters between the first set of audio parameters and the second set of audio parameters; and playing the audio stream to emit sound through the speaker system of the far-end system sequentially using the first set of audio parameters, the one or more intermediate audio parameters, and the second set of audio parameters. ### Claim 11 An article of manufacture, comprising: a non-transitory machine-readable storage medium that stores instructions which, when executed by a processor in a computer, determine an amount of area in a first frame of a video stream occupied by a near-end audio source, wherein the video stream is captured by a near-end system along with a corresponding audio stream; determine a first set of audio parameters to apply to the audio stream based on the determined amount of area in the first frame of the video stream occupied by the near-end audio source; and play the audio stream using the first set of audio parameters to emit sound through one or more transducers in a speaker system of a far-end system. ### Claim 12 (depends on 11) The article of manufacture of claim 11, wherein determining the amount of area in the first frame occupied by the near-end audio source is determined based on one of (1) a zoom setting of a near-end video camera that captured the first frame and (2) an analysis of the first frame using facial recognition. ### Claim 13 (depends on 11) The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor: set a loudness parameter to a first loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and set the loudness parameter to a second loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first loudness level is larger than the second loudness level. ### Claim 14 (depends on 11) The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor: set a directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first directivity level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and set the directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second directivity level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first directivity level is greater than the second directivity level. ### Claim 15 (depends on 11) The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor: set a reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first reverberation level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and set the reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second reverberation level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first reverberation level is less than the second reverberation level. ### Claim 16 (depends on 11) The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor: set an equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first equalization setting upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and set the equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second equalization setting upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first equalization setting filters more high frequency elements of the audio stream than the second equalization setting. ### Claim 17 (depends on 11) The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor: determine an amount of area in a second frame of the video stream occupied by the near-end audio source; determine a second set of audio parameters to apply to the audio stream based on the determined amount of area in the second frame of the video stream occupied by the near-end audio source; determine one or more intermediary audio parameters between the first set of audio parameters and the second set of audio parameters; and play the audio stream to emit sound through the speaker system of the far-end system sequentially using the first set of audio parameters, the one or more intermediate audio parameters, and the second set of audio parameters. ### Claim 18 A far-end system for adjusting audio, comprising: a network interface for receiving an audio stream and a video stream captured by a near-end system; and a hardware processor to: determine a first set of audio parameters to apply to the audio stream based on a determined amount of area in a first frame of the video stream occupied by a near-end audio source; and play the audio stream using the first set of audio parameters to emit sound through one or more transducers in a speaker system of the far-end system. ### Claim 19 (depends on 18) The far-end system of claim 18, wherein the hardware processor further: determines the amount of area in the first frame of the video stream occupied by the near-end audio source. ### Claim 20 (depends on 18) The far-end system of claim 18, wherein determining the amount of area in the first frame occupied by the near-end audio source is determined based on one of (1) a zoom setting of a near-end video camera that captured the first frame and (2) an analysis of the first frame using facial recognition. ### Claim 21 (depends on 18) The far-end system of claim 18, wherein determining the first set of audio parameters comprises: setting a loudness parameter to a first loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the loudness parameter to a second loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first loudness level is larger than the second loudness level. ### Claim 22 (depends on 18) The far-end system of claim 18, wherein determining the first set of audio parameters comprises: setting a directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first directivity level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second directivity level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first directivity level is greater than the second directivity level. ### Claim 23 (depends on 18) The far-end system of claim 18, wherein determining the first set of audio parameters comprises: setting a reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first reverberation level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second reverberation level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first reverberation level is less than the second reverberation level. ### Claim 24 (depends on 18) The far-end system of claim 18, wherein determining the first set of audio parameters comprises: setting an equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first equalization setting upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and setting the equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second equalization setting upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first equalization setting filters more high frequency elements of the audio stream than the second equalization setting.