# US 2019/0089911 A1 · Digital camera with audio, visual and motion analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0089911 A1
- Document ID: US20190089911A1
- Status: Published application
- Kind: A1
- Application number: 16/304,299
- Filed: 2017-05-30
- Published: 2019-03-21
- Earliest claimed: 2016-05-28
- Assignees: Acoustic Knowledge, LLC
- Applicants: Acoustic Knowledge, LLC
- Inventors: Charles A. G. BOYLE; Caleb J. MADSEN
- Web page: https://patentgrep.com/patent/US20190089911A1

## CPC

- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56)
- H04N 5/772: {the recording apparatus and the television camera being placed in the same enclosure}
- H04N 5/9201: {involving the multiplexing of an additional signal and the video signal}
- H04N 5/9202: {the additional signal being a sound signal (H04N5/9155 takes precedence)}
- H04N 9/802: involving processing of the sound signal (H04N9/806, H04N9/835 take precedence)
- H04N 9/8205: {involving the multiplexing of an additional signal and the colour video signal}
- H04N 23/50: Constructional details
- G06T 7/579: from motion
- G11B 31/006: {with video camera or receiver}
- G11B 2020/10546: {specifically adapted for audio data}
- G11B 2020/10611: {3D video data}
- H04N 2101/00: Still video cameras
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A digital camera with audio, visual and motion analysis includes a digital processor, an input processing system, and one or more imaging sensors, sound sensors, and motion sensors. In a non-limiting embodiment, the input processing system including non-transitory computer readable media including code segments, executable by the digital processor, for real-time audio, visual and motion analysis to develop a digital model of an ambient environment of the digital camera from data derived from the imaging sensor(s), sound sensor(s) and motion sensor(s).

## Claims

### Claim 1

A digital camera with audio, visual and motion analysis comprising:
a camera body;
a lens supported by the camera body;
an imaging sensor supported by the camera body in alignment with the lens;
a sound sensor supported by the camera body;
a motion sensor supported by the camera body;
an input processing system supported by the camera body including
(a) a system bus;
(b) a digital processor coupled to the bus;
(c) a sound sensor subsystem coupling an output of the sound sensor to the bus;
(d) an imaging sensor subsystem coupling an output of the imaging sensor to the bus;
(e) a motion sensor subsystem coupling an output of the motion sensor to the bus;
(f) a high precision clock, separate from a system clock, to provide synchronous clock signals to the motion sensor subsystem, the imaging sensor subsystem, and the sound sensor subsystem; and
(g) non-transitory digital memory coupled to the bus including code segments, executable by the digital processor, for real-time audio, visual and motion analysis to develop a digital model of an ambient environment of the digital camera from data derived from the motion sensor subsystem, the imaging sensor subsystem and the sound sensor subsystem.

### Claim 2 (depends on 1)

A digital camera with audio, visual and motion analysis as recited in claim 1 further comprising an environmental sensor and an environmental sensor subsystem coupling the environmental sensor to the system bus, wherein the digital model of the ambient environment is also derived from data from the environmental sensor subsystem.

### Claim 3 (depends on 1)

A digital camera with audio, visual and motion analysis as recited in claim 1 further comprising a global positioning sensor and a global positioning sensor subsystem coupling the global positioning sensor to the system bus, wherein the digital model of the ambient environment is also derived from data from the global positioning subsystem.

### Claim 4 (depends on 1)

A digital camera with audio, visual and motion analysis as recited in claim 1 further comprising code segments implementing a Structure from Motion (SfM) process.

### Claim 5 (depends on 1)

A digital camera with audio, visual and motion analysis as recited in claim 1 further comprising code segments implementing a Simultaneous Localization And Mapping (SLAM) process.

### Claim 6 (depends on 1)

A digital camera with audio, visual and motion analysis as recited in claim 1 further comprising code segments deriving metadata from raw input data from a plurality of sensors used to develop the digital model of the ambient environment.

### Claim 7 (depends on 1)

A digital camera with audio, visual and motion analysis as recited in claim 1 further comprising a display supported by the camera body capable of displaying an image of the ambient environment.

### Claim 8 (depends on 7)

A digital camera with audio, visual and motion analysis as recited in claim 7 further comprising code segments overlaying an augmented reality (AR) image over the image of the ambient environment.

### Claim 9

A method for modeling an ambient environment with a digital camera comprising:
collecting raw input data from a plurality of sensors including, at least, an imaging sensor, a sound sensor and a motion sensor;
deriving metadata from the raw input data; and
processing the metadata and the raw input data to develop a model of the ambient environment of the digital camera.

### Claim 10 (depends on 9)

A method for modeling an ambient environment with a digital camera as recited in claim 9 further comprising implementing a Structure from Motion (SfM) process.

### Claim 11 (depends on 9)

A method for modeling an ambient environment with a digital camera as recited in claim 9 further comprising implementing a Simultaneous Localization And Mapping (SLAM) process.

### Claim 12 (depends on 9)

A method for modeling an ambient environment with a digital camera as recited in claim 9 wherein the raw input data includes only one of imaging sensor data and sound sensor data.

### Claim 13

A method for providing augmented reality with a digital camera comprising:
scanning an ambient environment with a digital camera including at least an imaging sensor, a sound sensor and a motion sensor;
developing a model of the ambient environment with a digital processor of the digital camera that includes physical objects and their properties from data derived from the imaging sensor, the sound sensor and the motion sensor; and
developing augmented reality (AR) to overlay the ambient environment.

### Claim 14 (depends on 13)

A method for providing an augmented reality with a digital camera as recited in claim 13 wherein developing AR comprises overlaying an AR image over an image of the ambient environment, wherein the AR image includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 15 (depends on 13)

A method for providing an augmented reality with a digital camera as recited in claim 13 wherein developing AR comprises overlaying AR sound over sound of the ambient environment, wherein the AR sound includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 16 (depends on 13)

A method for modeling an ambient environment with a digital camera as recited in claim 13 further comprising implementing a Structure from Motion (SfM) process.

### Claim 17 (depends on 13)

A method for modeling an ambient environment with a digital camera as recited in claim 13 further comprising implementing a Simultaneous Localization And Mapping (SLAM) process.

### Claim 18 (depends on 13)

A method for modeling an ambient environment with a digital camera as recited in claim 13 further comprising a database including virtual objects and virtual properties.

### Claim 19

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera for:
scanning an ambient environment with a digital camera including at least an imaging sensor, a sound sensor and a motion sensor;
developing a model of the ambient environment with a digital processor of the digital camera that includes physical objects and their properties from data derived from the imaging sensor, the sound sensor and the motion sensor; and
developing augmented reality (AR) to overlay the ambient environment.

### Claim 20 (depends on 19)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 19 wherein developing AR comprises overlaying an AR image over an image of the ambient environment, wherein the AR image includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 21 (depends on 19)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 19 wherein developing AR comprises overlaying AR sound over sound of the ambient environment, wherein the AR sound includes virtual objects with virtual properties that are related to physical properties of the ambient environment.

### Claim 22 (depends on 19)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 19 further comprising implementing a Structure from Motion (SfM) process.

### Claim 23 (depends on 19)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera recited in claim 19 further comprising implementing a Simultaneous Localization And Mapping (SLAM) process.

### Claim 24 (depends on 19)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 19 further comprising a database including virtual objects and virtual properties.

### Claim 25 (depends on 19)

A non-transitory computer readable medium comprising code segments executable on a digital processor of a digital camera as recited in claim 19 wherein developing an AR to overlay the ambient environment includes accelerating an AR computational process using at least one of a sensor input or a neural network.
