# US 10,122,956 B2 · Beam forming for microphones on separate faces of a camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,122,956 B2
- Document ID: US10122956B2
- Status: Granted
- Kind: B2
- Application number: 15/701,426
- Filed: 2017-09-11
- Published: 2018-03-22
- Granted: 2018-11-06
- Earliest claimed: 2016-09-16
- Assignees: GoPro, Inc.
- Applicants: GoPro, Inc.
- Inventors: Zhinian Jing; Joyce Rosenbaum
- Web page: https://patentgrep.com/patent/US10122956B2

## CPC

- H04N 5/607: {for more than one sound signal, e.g. stereo, multilanguages (H04N5/602 takes precedence)}
- H04N 23/00: Cameras or camera modules comprising electronic image sensors; Control thereof
- H04R 5/027: Spatial or constructional arrangements of microphones, e.g. in dummy heads
- H04S 7/305: {Electronic adaptation of stereophonic audio signals to reverberation of the listening space (H04S7/301 takes precedence)}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/682: {Vibration or motion blur correction}
- H04R 2203/12: Beamforming aspects for stereophonic sound reproduction with loudspeaker arrays
- H04R 2499/11: Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

A camera system capable of capturing images of an event in a dynamic environment includes two microphones configured to capture stereo audio of the event. The microphones are on orthogonal surfaces of the camera system. Because the microphones are on orthogonal surfaces of the camera system, the camera body can impact the spatial response of the two recorded audio channels differently, leading to degraded stereo recreation if standard beam forming techniques are used. The camera system includes tuned beam forming techniques to generate multi-channel audio that more accurately recreates the stereo audio by compensating for the shape of the camera system and the orientation of microphones on the camera system. The tuned beam forming techniques include optimizing a set of beam forming parameters, as a function of frequency, based on the true spatial response of the recorded audio signals.

## Claims

### Claim 1

A camera system comprising:
a camera body;
an image sensor assembly coupled to a lens assembly to detect and capture images from sources external to the camera body;
a first microphone on a first surface of the camera body;
a second microphone on a second surface of the camera body, the second surface substantially orthogonal to the first surface;
an audio assembly coupling the first microphone and the second microphone, the audio assembly configured to capture a first audio channel via the first microphone and a second audio channel via the second microphone from an audio source during the capture of images by the image sensor, the audio assembly further configured to output the audio channels; and
a processor coupled to the audio assembly, the processor coupled with a memory storing instructions that when executed causes the processor to:
receive the captured audio channels,
apply a set of tuned beam forming parameters tuned for temporal, spectral, and spatial characteristics in the audio channels associated with the orthogonality of the first surface and the second surface, the characteristics affecting the representation of the audio source during stereo audio playback;
process the captured audio channels using the set of tuned beam forming parameters, the beam forming parameters applying a delay to the first audio channel and the second audio channel associated with the orthogonality of the first surface and the second surface to generate a first virtual microphone channel and a second virtual microphone channel which improves the representation of the audio source during stereo audio playback;
combine the first virtual audio channel and the second virtual audio channel into an audio stream that is configured for stereo audio playback; and
output the audio stream.

### Claim 2

The camera system of 1, wherein the set of tuned beam forming parameters applied to the audio channels is associated with the linear distance between the first microphone and the second microphone across the surface of the camera body.

### Claim 3

The camera system of 1, wherein the set of tuned beam forming parameters applied is associated with the linear distance between the first microphone and the second microphone through the body of the camera.

### Claim 4

The camera system of 1, wherein the set of tuned beam forming parameters applied is based on maximizing a noise ratio between opposing hemispheres of the spatial response of the first and second audio signals.

### Claim 5

The camera system of 1, wherein set of tuned beam forming parameters can be independently applied to separate frequency sub-bands of the audio channels based on the shape of the camera body and the frequency range of the sub-band.

### Claim 6

The camera system of 1, wherein set of tuned beam forming parameters access gain and delay values stored in the system memory associated with the characteristics and shape of the camera body.

### Claim 7

The camera system of 1, wherein the camera system determines the gain and delay to apply with the set of tuned beam forming parameters based on the characteristics and shape of the camera body.

### Claim 8

An electronic device with audio recording capability comprising:
a device body;
a first microphone on a first surface of the electronic device;
a second microphone on a second surface of the electronic device, the second surface substantially orthogonal to the first surface;
an audio assembly coupling the first microphone and the second microphone, the audio assembly configured to capture a first audio channel and a second audio channel from an audio source via the first microphone and the second microphone, the audio assembly further configured to output the audio channels; and
a processor coupled to the audio assembly, the processor coupled with a memory storing instructions that when executed causes the processor to:
receive the captured audio channels,
apply a set of tuned beam forming parameters tuned for temporal, spectral, and spatial characteristics in the audio channels associated with the orthogonality of the first surface and the second surface, the characteristics affecting the representation of the audio source during stereo audio playback;
process the captured audio channels using the set of tuned beam forming parameters, the beam forming parameters applying a delay to the first audio channel and the second audio channel associated with the orthogonality of the first surface and the second surface to generate a first virtual microphone channel and a second virtual microphone channel which improves the representation of the audio source during stereo audio playback;
combine the first virtual audio channel and the second virtual audio channel into an audio stream that is configured for stereo audio playback; and
output the audio stream.

### Claim 9

The electronic device of 8, wherein the set of tuned beam forming parameters applied to the audio channels is associated with the linear distance between the first microphone and the second microphone across the surface of the electronic device.

### Claim 10

The electronic device of 8, wherein the set of tuned beam forming parameters applied is associated with the linear distance between the first microphone and the second microphone through the body of the electronic device.

### Claim 11

The electronic device of 8, wherein tuning the set of beam forming parameters applied is based on maximizing a noise ratio between opposing hemispheres of the spatial response of the first and second audio signals.

### Claim 12

The electronic device of 8, wherein set of tuned beam forming parameters can be independently applied to separate frequency sub-bands of the audio channels based on the shape of the camera body and the frequency range of the sub-band.

### Claim 13

The electronic device of 8, wherein set of tuned beam forming parameters access gain and delay values stored in the system memory associated with the characteristics and shape of the camera body.

### Claim 14

The electronic device of 8, wherein the camera system determines the gain and delay to apply with the set of tuned beam forming parameters based on the characteristics and shape of the camera body.

### Claim 15

A method for generating stereo audio from at least two microphones on a camera body comprising:
capturing a first audio channel from an audio source via a first microphone on a first surface of the camera body during the capture of images by the camera system;
capturing a second audio channel from the audio source via a second microphone on a second surface of the camera body during the capture of images by the camera system, the second surface orthogonal to the first surface;
separating the first and the second audio channels into frequency sub-bands;
selecting a set of frequency sub-bands of the audio channels;
applying a set of tuned beam forming parameters to the selected frequency sub-bands, the tuned beam forming parameters tuned for temporal, spectral, and spatial characteristics in the audio channels associated with the orthogonality of the first surface and the second surface, the characteristics affecting the representation of the audio source during stereo audio playback;
processing the captured audio channels using the tuned set of beam forming parameters, the beam forming parameters applying a delay to the first audio channel and the second audio channel associated with the orthogonality of the first surface and the second surface to generate a first virtual microphone channel and a second virtual microphone channel which improves the representation of the audio source during stereo audio playback;
combining the first virtual audio channel and the second virtual audio channel into an audio stream that is configured for stereo audio playback; and
outputting the audio stream.

### Claim 16

The method of 15, wherein the set of tuned beam forming parameters applied to the audio channels is associated with the linear distance between the first microphone and the second microphone across the surface of the camera body.

### Claim 17

The method of 15, wherein the set of tuned beam forming parameters applied is associated with the linear distance between the first microphone and the second microphone through the body of the camera.

### Claim 18

The method of 15, wherein tuning the set of beam forming parameters applied is based on maximizing a noise ratio between opposing hemispheres of the spatial response of the first and second audio signals.

### Claim 19

The method of 15, wherein set of tuned beam forming parameters can be independently applied to separate frequency sub-bands of the audio channels based on the shape of the camera body and the frequency range of the sub-band.

### Claim 20

The method of 15, wherein set of tuned beam forming parameters access gain and delay values stored in the system memory associated with the characteristics and shape of the camera body.