# US 10,204,457 B2 · Digital camera system with acoustic modeling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,204,457 B2
- Document ID: US10204457B2
- Status: Granted
- Kind: B2
- Application number: 15/967,537
- Filed: 2018-04-30
- Published: 2018-08-30
- Granted: 2019-02-12
- Earliest claimed: 2016-05-28
- Assignees: Acoustic Knowledge, LLC
- Applicants: Acoustic Knowledge, LLC
- Inventors: Charles A. G. Boyle; Caleb J. Madsen
- Web page: https://patentgrep.com/patent/US10204457B2

## CPC

- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56)
- H04N 5/772: {the recording apparatus and the television camera being placed in the same enclosure}
- H04N 5/9201: {involving the multiplexing of an additional signal and the video signal}
- H04N 5/9202: {the additional signal being a sound signal (H04N5/9155 takes precedence)}
- H04N 9/802: involving processing of the sound signal (H04N9/806, H04N9/835 take precedence)
- H04N 9/8205: {involving the multiplexing of an additional signal and the colour video signal}
- H04N 23/50: Constructional details
- G06T 7/579: from motion
- G11B 31/006: {with video camera or receiver}
- G11B 2020/10546: {specifically adapted for audio data}
- G11B 2020/10611: {3D video data}
- H04N 2101/00: Still video cameras
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A digital camera with audio, visual and motion analysis includes a digital processor, an input processing system, and one or more imaging sensors, sound sensors, and motion sensors. In a non-limiting embodiment, the input processing system including non-transitory computer readable media including code segments, executable by the digital processor, for real-time audio, visual and motion analysis to develop a digital model of an ambient environment of the digital camera from data derived from the imaging sensor(s), sound sensor(s) and motion sensor(s).

## Claims

### Claim 1

A digital camera system with acoustic modeling comprising:
a camera body;
a lens supported by the camera body;
an imaging sensor supported by the camera body in alignment with the lens;
a sound sensor supported by the camera body;
a display supported by the camera body capable of displaying an image;
an input processing system supported by the camera body including
(a) a system bus;
(b) a digital processor coupled to the bus;
(c) a sound sensor subsystem coupling an output of the sound sensor to the bus; and
(d) non-transitory digital memory coupled to the bus including code segments, executable by the digital processor, for real-time audio, visual and motion analysis to develop a digital model of an ambient environment of the digital camera from data derived from the sound sensor subsystem, and code segments, executable by the digital processor, for overlaying an augmented reality (AR) image over an image of the ambient environment on the display.

### Claim 2 (depends on 1)

A digital camera system with acoustic modeling as recited in claim 1 further comprising a high precision clock, separate from a system clock, to provide synchronous clock signals to the sound sensor subsystem.

### Claim 3 (depends on 2)

A digital camera system with acoustic modeling as recited in claim 2 further comprising a motion sensor and a motion sensor subsystem coupling the motion sensor to the system bus, wherein the digital model of the ambient environment is also derived from data from the motion sensor subsystem.

### Claim 4 (depends on 3)

A digital camera system with acoustic modeling as recited in claim 3 further comprising an environmental sensor and an environmental sensor subsystem coupling the environmental sensor to the system bus, wherein the digital model of the ambient environment is also derived from data from the environmental sensor subsystem.

### Claim 5 (depends on 3)

A digital camera system with acoustic modeling as recited in claim 3 further comprising a global positioning sensor and a global positioning sensor subsystem coupling the global positioning sensor to the system bus, wherein the digital model of the ambient environment is also derived from data from the global positioning subsystem.

### Claim 6 (depends on 1)

A digital camera system with acoustic modeling as recited in claim 1 further comprising code segments implementing a Structure from Motion (SfM) process.

### Claim 7 (depends on 1)

A digital camera system with acoustic modeling as recited in claim 1 further comprising code segments implementing a Simultaneous Localization And Mapping (SLAM) process.

### Claim 8 (depends on 1)

A digital camera system with acoustic modeling as recited in claim 1 further comprising code segments deriving metadata from raw input data from a plurality of sensors used to develop the digital model of the ambient environment.

### Claim 9

A method for providing augmented reality by acoustic modeling on a digital camera comprising:
scanning an ambient environment with a digital camera having a plurality of sensors including, at least, an imaging sensor, a sound sensor and a motion sensor;
obtaining raw input data from the plurality of sensors;
deriving metadata from the raw input data;
processing the metadata and the raw input data with a digital processor of the digital camera to develop a model of the ambient environment of the digital camera that includes physical objects and their properties from data derived from the imaging sensor, the sound sensor and the motion sensor;
developing an augmented reality (AR) image related to the ambient environment image; and
displaying the ambient environment image and the AR image on a screen of the digital camera.

### Claim 10 (depends on 9)

A method for providing an augmented reality by acoustic modeling on a digital camera as recited in claim 9 wherein the AR image includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 11 (depends on 9)

A method for providing an augmented reality by acoustic modeling on a digital camera as recited in claim 9 further comprising developing AR sound related to sound of the ambient environment, wherein the AR sound includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 12 (depends on 9)

A method for modeling an ambient environment by acoustic modeling on a digital camera as recited in claim 9 further comprising implementing a Structure from Motion (SfM) process.

### Claim 13 (depends on 9)

A method for modeling an ambient environment by acoustic modeling on a digital camera as recited in claim 9 further comprising implementing a Simultaneous Localization And Mapping (SLAM) process.

### Claim 14 (depends on 9)

A method for modeling an ambient environment by acoustic modeling on a digital camera as recited in claim 9 further comprising a database including at least one of virtual objects with virtual properties and physical objects with physical properties.

### Claim 15

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera comprising:
code segments to scan an ambient environment with a digital camera having a plurality of sensors including, at least, an imaging sensor, a sound sensor and a motion sensor;
code segments to obtain raw input data from the plurality of sensors;
code segments to derive metadata from the raw input data;
code segments to process the metadata and the raw input data with a digital processor of the digital camera to develop a model of the ambient environment of the digital camera that includes physical objects and their properties from data derived from the imaging sensor, the sound sensor and the motion sensor; and
code segments to develop augmented reality (AR) to overlay the ambient environment.

### Claim 16 (depends on 15)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 15 wherein developing AR comprises overlaying an AR image over an image of the ambient environment, wherein the AR image includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 17 (depends on 15)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 15 wherein developing AR comprises overlaying AR sound over sound of the ambient environment, wherein the AR sound includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 18 (depends on 15)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 15 further comprising code segments implementing a Structure from Motion (SfM) process.

### Claim 19 (depends on 15)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 15 further comprising code segments implementing a Simultaneous Localization And Mapping (SLAM) process.

### Claim 20 (depends on 15)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 15 further comprising a database including at least one of virtual objects with virtual properties and physical objects with physical properties.

### Claim 21 (depends on 15)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 15 wherein developing an AR to overlay the ambient environment includes accelerating an AR computational process using at least one of a sensor input or a neural network.
