Sound search using caption embeddings

A method comprising:

• generating, by one or more processors, one or more query caption embeddings representing multiple words that together form a semantic unit, based on a query, using a machine-learning model, wherein the query caption embeddings are based on sound captions extracted from the query that are natural-language descriptions of sounds and wherein outputs of the machine-learning model are one or more caption embeddings and locations of the one or more caption embeddings in a caption embedding space are indicative of semantic relationships among the sound captions, the semantic relationships corresponding to a similarity of semantic meaning between the sound captions, the similarity of semantic meaning distinct from a text comparison;
• generating one or more context terms based on a portion of the query distinct from the semantic unit;
• filtering a set of media files to generate a set of filtered media files based on the context terms and metadata associated with the set of media files;
• comparing, by the one or more processors, the one or more query caption embeddings, in the caption embedding space, to one or more second caption embeddings associated with the set of filtered media files, wherein each second caption embedding of the one or more second caption embeddings represents a corresponding sound caption and each sound caption includes a natural-language text description of a sound, to determine, a first similarity metric indicative of similarity between the one or more caption embeddings and the one or more query caption embeddings;
• and
• generating, by the one or more processors, search results identifying one or more first media files of the set of filtered media files, each of the one or more first media files associated with at least one of the one or more caption embeddings.

A non-transitory computer-readable storage device storing instructions that are executable by one or more processors to cause the one or more processors to:

• generate one or more query caption embeddings representing multiple words that together form a semantic unit, based on a query, using a machine-learning model, wherein the query caption embeddings are based on sound captions extracted from the query that are natural-language descriptions of sounds and wherein outputs of the machine-learning model are one or more caption embeddings and locations of the one or more caption embeddings in a caption embedding space are indicative of semantic relationships among the sound captions, the semantic relationships corresponding to a similarity of semantic meaning between the sound captions, the similarity of semantic meaning distinct from a text comparison;
• generate one or more context terms based on a portion of the query distinct from the semantic unit;
• filter a set of media files to generate a set of filtered media files based on the context terms and metadata associated with the set of media files;
• compare the one or more query caption embeddings, in the caption embedding space, to one or more second caption embeddings associated with the set of filtered media files, wherein each second caption embedding of the one or more second caption embeddings represents a corresponding sound caption and each sound caption includes a natural-language text description of a sound, to determine a first similarity metric, indicative of similarity between the one or more caption embeddings and the one or more query caption embeddings;
• and
• generate search results identifying one or more first media files of the set of filtered media files, each of the one or more first media files associated with at least one of the one or more caption embeddings.

An apparatus comprising:

• means for generating, by one or more processors, one or more query caption embeddings representing multiple words that together form a semantic unit, based on a query, using a machine-learning model, wherein the query caption embeddings are based on sound captions extracted from the query that are natural-language descriptions of sounds and wherein outputs of the machine-learning model are one or more caption embeddings and locations of the one or more caption embeddings in a caption embedding space are indicative of semantic relationships among the sound captions, the semantic relationships corresponding to a similarity of semantic meaning between the sound captions, the similarity of semantic meaning distinct from a text comparison;
• means for generating one or more context terms based on a portion of the query distinct from the semantic unit;
• means for filtering a set of media files to generate a set of filtered media files based on the context terms and metadata associated with the set of media files;
• means for comparing, by the one or more processors, the one or more query caption embeddings, in the caption embedding space, to one or more second caption embeddings associated with the set of filtered media files, wherein each second caption embedding of the one or more second caption embeddings represents a corresponding sound caption and each sound caption includes a natural-language text description of a sound, to determine, a first similarity metric indicative of similarity between the one or more caption embeddings and the one or more query caption embeddings;
• and
• means for generating, by the one or more processors, search results identifying one or more first media files of the set of filtered media files, each of the one or more first media files associated with at least one of the one or more caption embeddings.