# US 10,372,991 B1 · Systems and methods that leverage deep learning to selectively store audiovisual content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,372,991 B1
- Document ID: US10372991B1
- Status: Granted
- Kind: B1
- Application number: 15/944,415
- Filed: 2018-04-03
- Granted: 2019-08-06
- Earliest claimed: 2018-04-03
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: James Niemasik; Manoj Plakal
- Web page: https://patentgrep.com/patent/US10372991B1

## CPC

- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06F 16/23: Updating
- G06F 16/583: using metadata automatically derived from the content
- G06F 16/683: using metadata automatically derived from the content
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/172: {Classification, e.g. identification}
- G10L 25/30: using neural networks
- G10L 25/57: for processing of video signals
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06V 20/44: {Event detection}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

Systems, methods, and devices for curating audiovisual content are provided. A mobile image capture device can be operable to capture one or more images; receive an audio signal; analyze at least a portion of the audio signal with a first machine-learned model to determine a first audio classifier label descriptive of an audio event; identify a first image associated with the first audio classifier label; analyze the first image with a second machine-learned model to determine a desirability of a scene depicted by the first image; and determine, based at least in part on the desirability of the scene depicted by the first image, whether to store a copy of the first image associated with the first audio classifier label in the non-volatile memory of the mobile image capture device or to discard the first image without storing a copy of the first image.

## Claims

### Claim 1

A mobile image capture device, comprising:
a network interface;
a power source;
an image sensor;
an audio sensor;
at least one processor; and
a memory that stores a plurality of machine-learned models, the plurality of machine-learned models comprising a first machine-learned model configured to determine an audio classifier label and a second machine-learned model configured to determine a desirability of a scene depicted by an image, the memory comprising at least a temporary image buffer, an audio buffer, and a non-volatile memory;
wherein the mobile image capture device is operable to:
receive an audio signal;
analyze at least a portion of the audio signal with the first machine-learned model to determine a first audio classifier label descriptive of an audio event;
responsive to determining the first audio classifier label descriptive of the audio event, identify a first image associated with the first audio classifier label descriptive of the audio event;
analyze the first image with the second machine-learned model to determine a desirability of a scene depicted by the first image; and
determine, based at least in part on the desirability of the scene depicted by the first image, whether to store a copy of the first image associated with the first audio classifier label in the non-volatile memory of the mobile image capture device or to discard the first image without storing a copy of the first image;
wherein the first image comprises an image captured by the mobile image capture device.

### Claim 2 (depends on 1)

The mobile image capture device of claim 1, wherein the first image is temporally displaced from the audio event, and wherein to identify the first image, the mobile image capture device is operable to identify the first image based on a time displacement associated with the first audio classifier label.

### Claim 3 (depends on 2)

The mobile image capture device of claim 2, wherein the time displacement comprises a time period preceding the audio event.

### Claim 4 (depends on 3)

The mobile image capture device of claim 3, wherein the first audio classifier label comprises laughter.

### Claim 5 (depends on 2)

The mobile image capture device of claim 2, wherein the time displacement comprises a time period following the audio event.

### Claim 6 (depends on 5)

The mobile image capture device of claim 5, wherein the first audio classifier label comprises an image capture cue.

### Claim 7 (depends on 1)

The mobile image capture device of claim 1, wherein the first image associated with the audio label comprises an image captured concurrently with the audio event.

### Claim 8 (depends on 1)

The mobile image capture device of claim 1, wherein the second machine-learned model comprises one or more of:
a face detection neural network that detects a presence of one or more faces in the scene of each image;
a face recognition neural network that matches one or more faces in the scene of each image to one or more other faces;
a face attributes neural network that detects various facial characteristics of one or more faces in the scene of each image;
an image content neural network that outputs one or more semantic labels that describe the scene of each image; and
a photo quality neural network that outputs a photo score that describes various photographic quality characteristics of each input image.

### Claim 9 (depends on 1)

The mobile image capture device of claim 1, wherein the mobile image capture device is further operable to:
capture a plurality of images;
maintain a copy of each captured image in the plurality of images in the temporary image buffer; and
wherein the first image associated with the audio classifier label comprises one of the captured images in the plurality of images.

### Claim 10 (depends on 1)

The mobile image capture device of claim 1, wherein to analyze the audio signal with the first machine-learned model to determine the first audio classifier label, the mobile image capture device is operable to:
maintain a copy of the audio signal in the audio buffer;
input at least a portion of the copy of the audio signal into the first machine-learned model; and
receive the first audio classifier label descriptive of an audio event as an output of the first machine-learned model.

### Claim 11

A method to generate an edited video including an ambient soundtrack, comprising:
obtaining, by one or more computing devices, data indicative of a video stream comprising a plurality of sequential image frames and an associated audio signal;
analyzing, by the one or more computing devices, the audio signal with a machine-learned audio classifier model to determine a plurality of audio classifier labels associated with a plurality of portions of the audio signal;
generating, by the one or more computing devices, an ambient soundtrack comprising at least a first portion of the audio signal based at least in part on the plurality of audio classifier labels, wherein generating the ambient soundtrack based at least in part on the plurality of audio classifier labels comprises selecting for inclusion in the ambient soundtrack portions of the audio signal that have audio classifier labels associated with ambient sound while rejecting portions of the audio signal that have audio classifier labels associated with foreground sound;
generating, by the one or more computing devices, an image stream comprising at least a subset of the plurality of sequential image frames; and
generating, by the one or more computing devices, an edited video comprising the ambient soundtrack and the image stream.

### Claim 12 (depends on 11)

The method of claim 11, wherein the data indicative of the video stream is obtained via a mobile image capture device.

### Claim 13 (depends on 11)

The method of claim 11, wherein the one or more computing devices consist of a mobile image capture device, such that the method is performed by the mobile image captured device.

### Claim 14 (depends on 11)

The method of claim 11, further comprising:
receiving, by the one or more computing devices, an audio selection input descriptive of a first audio classifier label from a user; and
wherein generating, by the one or more computing devices, the ambient soundtrack comprises selecting, by the one or more computing devices, at least one portion of the audio signal labeled with the first audio classifier label for inclusion in the ambient soundtrack.

### Claim 15 (depends on 11)

The method of claim 11, wherein the audio classifier labels associated with foreground sound comprise audio classifier labels associated with human speech.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
analyzing, by the one or more computing devices, at least a portion of the plurality of sequential image frames with one or more image analysis machine-learned models to determine one or more image labels descriptive of a scene; and
wherein the ambient soundtrack is further generated based at least in part on the one or more image labels.

### Claim 17 (depends on 11)

The method of claim 11, wherein generating, by the one or more computing devices, the image stream comprising at least a subset of the plurality of sequential images comprises:
selecting, by the one or more computing devices, a plurality of non-sequential image frames; and
wherein the image stream comprises a hyper-lapse image stream comprising the plurality of non-sequential image frames arranged consecutively.

### Claim 18 (depends on 11)

The method of claim 11, wherein generating, by the one or more computing devices, the image stream comprising at least a subset of the plurality of sequential images comprises:
selecting, by the one or more computing devices, a first subset of sequential image frames;
selecting, by the one or more computing devices, a second subset of sequential image frames; and
wherein the image stream comprises a shortened image stream comprising the first subset of sequential image frames and the second subset of sequential image frames arranged consecutively.

### Claim 19

A method for capturing an image, comprising:
capturing, by an image sensor of an image capture device, one or more images;
maintaining, by the image capture device, a copy of the one or more captured images in a temporary image buffer of the image capture device;
receiving, by the image capture device, an audio signal;
analyzing, by the image capture device, at least a portion of the audio signal with a machine-learned audio classifier model to determine a first audio classifier label descriptive of an audio event;
responsive to determining the first audio classifier label, determining, by the image capture device, a time displacement amount associated with the first audio classifier label;
using, by the image capture device, the time displacement amount to identify a first captured image associated with the first audio classifier label from the one or more captured images maintained in the temporary image buffer, wherein the first captured image is temporally displaced from the portion of the audio signal labeled with the first audio classifier label by the time displacement amount; and
storing, by the image capture device, a copy of the first captured image in a non-volatile memory of the image capture device.

### Claim 20 (depends on 19)

The method of claim 19, wherein the first audio classifier label comprises a laughter label and the time displacement amount comprises a negative amount of time, such that the image capture device identifies the first captured image that was captured by the image capture device at a time prior to receipt of the portion of the audio signal labeled with the laughter label.