# US 2020/0401367 A1 · Determining that Audio Includes Music and then Identifying the Music as a Particular Song > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401367 A1 - Document ID: US20200401367A1 - Status: Published application - Kind: A1 - Application number: 17/010,694 - Filed: 2020-09-02 - Published: 2020-12-24 - Earliest claimed: 2017-10-03 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Dominik Roblek; Blaise Hilary Aguera-Arcas; Thomas W. Hume; Marvin Karl Ritter; Brandon Charles Barbello; Kevin I. Kilgour; Mihajlo Velimirovic; Christopher Thornton; Gabriel Oak Taubman; James David Lyon; Jan Heinrich Althaus; Katsiaryna Naliuka; Julian James Odell; Matthew Sharifi; Beat Gfeller - Web page: https://patentgrep.com/patent/US20200401367A1 ## CPC - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G06F 16/635: Filtering based on additional data, e.g. user or group profiles - G06F 16/683: using metadata automatically derived from the content - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G10H 2210/031: Musical analysis, i.e. isolation, extraction or identification of musical elements or musical parameters from a raw acoustic signal or from an encoded audio signal - G10H 2210/046: for differentiation between music and non-music signals, based on the identification of musical parameters, e.g. based on tempo detection - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE ## Abstract In general, the subject matter described in this disclosure can be embodied in methods, systems, and program products. A computing device stores reference song characterization data and receives digital audio data. The computing device determines whether the digital audio data represents music and then performs a different process to recognize that the digital audio data represents a particular reference song. The computing device then outputs an indication of the particular reference song. ## Claims ### Claim 1 A computer-implemented method, comprising: storing, by a computing device, reference song characterization data that identify a plurality of audio characteristics for each reference song in a plurality of reference songs; receiving, by the computing device, digital audio data that represents audio recorded by a microphone; determining, by a first processor of the computing device and using a music determination process, whether the digital audio data represents music, wherein the determining includes converting the digital audio data from a time-domain format into a first frequency-domain format; recognizing, by a second processor of the computing device after determining that the digital audio data represents music, that the digital audio data represents a particular reference song from among the plurality of reference songs, wherein the recognizing includes converting the digital audio data from the time-domain format into a second frequency-domain format, wherein the first processor and the second processor are distinct hardware processors included in the computing device, the first processor operating at a lower voltage than the second processor; and outputting, by the computing device in response to recognizing that the digital audio data represents a particular reference song from among the plurality of reference songs, an indication of the particular reference song. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of reference songs includes at least ten thousand reference songs, such that the reference song characterization data identify audio characteristics for the at least ten thousand reference songs. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the reference song characterization data includes reference song characterization values, wherein the reference song characterization values for the reference songs in the plurality of reference songs are limited to a binary one or a binary zero, such that each reference song characterization value is limited to a binary one or a binary zero. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein determining whether the digital audio data represents music includes: using the digital audio data in the first frequency-domain format in the music determination process, and outputting an indication that the digital audio data represents music. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the music determination process includes executing a machine learning system that has been trained to determine whether audio represents music. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, wherein the frequency-domain conversion process is a first frequency-domain conversion process, and recognizing that the digital audio data represents the particular reference song includes: (i) converting the digital audio data from the time-domain format into the second frequency-domain format during a second frequency-domain conversion process, (ii) using the digital audio data in the second frequency-domain format in a music-characterization process that receives the digital audio data in the second frequency-domain format and outputs a collection of characterization values for the digital audio data, and (iii) comparing the collection of characterization values for the digital audio data to a plurality of characterization values for each of at least a subset of the plurality of reference songs, to determine that the plurality of characterization values for the particular reference song are most relevant to the collection of characterization values for the digital audio data. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the computing device determines whether the digital audio data represents music without accessing the reference song characterization data that identify the plurality of audio characteristics for each reference song in the plurality of reference songs. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein determining whether the digital audio data represents music includes the computing device determining, multiple times without receipt of user input that initiates the music determination process, that the digital audio data does not represent music, before determining that the digital audio data represents music. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the second processor operates from a clock signal that is at least an order of magnitude faster than a clock signal from which the first processor operates. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein outputting the indication that the particular reference song is playing includes presenting a name of the particular reference song on a lock screen of the computing device, in an always on screen of the computing device, or in a notification presented over an unlocked screen of the computing device, without user input having prompted the computing device to perform a song-identification process, other than enabling a continuous song-identification process. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the computing device includes the microphone. ### Claim 12 A computerized system, comprising: one or more processors; and one or more computer-readable devices including instructions that, when executed by the one or more processors, cause performance of operations comprising: storing, by a computing device, reference song characterization data that identify a plurality of audio characteristics for each reference song in a plurality of reference songs; receiving, by the computing device, digital audio data that represents audio recorded by a microphone; determining, by a first processor of the computing device and using a music determination process, whether the digital audio data represents music, wherein the determining includes converting the digital audio from a time-domain format into a first frequency-domain format; recognizing, by a second processor of the computing device after determining that the digital audio data represents music, that the digital audio data represents a particular reference song from among the plurality of reference songs, wherein the recognizing includes converting the digital audio data from the time-domain format into a second frequency-domain format, wherein the first processor and the second processor are distinct hardware processors included in the computing device, the first processor operating at a lower voltage than the second processor; and outputting, by the computing device in response to recognizing that the digital audio data represents a particular reference song from among the plurality of reference songs, an indication of the particular reference song. ### Claim 13 A computing system, comprising: a microphone configured to record an audio signal; an analog-to-digital converter configured to generate digital audio data from a recorded audio signal; one or more computer-readable memory devices that store: (i) reference song characterization data that identify a plurality of audio characteristics for each reference song in a plurality of reference songs, and (ii) instructions that are executable by processors and configure operation of the processors; a first processor configured to output an indication that the digital audio data represents music upon determining that the digital audio data represents music, wherein the determining includes converting the digital audio data from a time-domain format into a first frequency-domain format; a second processor configured to receive from the first processor the indication that the digital audio data represents music, and recognize that the digital audio data represents a particular reference song from among the plurality of reference songs, wherein the recognizing includes converting the digital audio data from the time-domain format into a second frequency-domain format, wherein the first processor and the second processor are distinct hardware processors included in the computing system, the first processor operating at a lower voltage than the second processor; and a display device configured to present an identification of the particular reference song responsive to the second processor recognizing that the digital audio data represents the particular reference song from among the plurality of reference songs. ### Claim 14 (depends on 13) The computing system of claim 13, wherein the computing system comprises: a mobile computing device that includes the microphone, the analog-to-digital converter, the one or more computer-readable memory devices, the first processor, and the second processor. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the first processor is configured to output the indication that the digital audio data represents music and the second processor is configured to recognize that the digital audio data represents the particular reference song without the computing device sending the digital audio data to a computing system other than the computing device. ### Claim 16 (depends on 14) The computing system of claim 14, wherein determining that the digital audio data represents music includes the first processor determining, multiple times without receipt of user input that initiates a music determination process, that the digital audio data does not represent music, before the first processor determines that the digital audio data represents music. ### Claim 17 (depends on 13) The computing system of claim 13, wherein the second processor operates from a clock signal that is at least an order of magnitude faster than a clock signal from which the first processor operates. ### Claim 18 (depends on 13) The computing system of claim 13, wherein the reference song characterization data includes reference song characterization values, wherein the reference song characterization values for the reference songs in the plurality of reference songs, stored by the one or more computer-readable memory devices, are limited to a binary one or a binary zero, such that each reference song characterization value is limited to a binary one or a binary zero. ### Claim 19 (depends on 13) The computing system of claim 13, wherein the plurality of reference songs includes at least ten thousand reference songs, such that the reference song characterization data that is stored by the one or more computer-readable memory devices identify audio characteristics for the at least ten thousand reference songs. ### Claim 20 (depends on 13) The computing system of claim 13, wherein the first processor is configured to determine that the digital audio data represents music by: using the digital audio data in the first frequency-domain format in a music-determination process that receives the digital audio data in the first frequency-domain format and outputs the indication that the digital audio data represents music.