# US 2026/0288865 A1 · Audio-augmented image search for sub-location identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0288865 A1
- Document ID: US20260288865A1
- Status: Published application
- Kind: A1
- Application number: 19/685,653
- Filed: 2026-05-22
- Published: 2026-09-24
- Earliest claimed: 2015-12-08
- Assignees: Snap Inc.
- Applicants: Snap inc.
- Inventors: Kirk Ouimet; Rong Yan; Yantao Zheng; Tomothy Michael Sehn; Evan Spiegel
- Web page: https://patentgrep.com/patent/US20260288865A1

## CPC

- G06F 16/78: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/434: {using image data, e.g. images, photos, pictures taken by a user}
- G06F 16/487: using geographical or spatial information, e.g. location
- G06F 16/489: {using time information}
- G06F 16/51: Indexing; Data structures therefor; Storage structures
- G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/583: using metadata automatically derived from the content
- G06F 16/587: using geographical or spatial information, e.g. location
- G11B 27/11: by using information not detectable on the record carrier
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A search query received from a mobile computing device includes image query data and ambient audio data captured at a location of the mobile computing device. The ambient audio data is filtered using a low-pass filter to isolate background audio data, which is compared to a set of stored audio fingerprints respectively associated with a plurality of sub-locations of a geographic location. Location-identifying text is generated based on a matching audio fingerprint. Image analysis of the image query data generates a set of image-derived keywords. An augmented search query incorporating the location-identifying text and the image-derived keywords is compiled and correlated with respective sub-location indexes to identify, as a search result, a specific sub-location of the plurality of sub-locations. The search result identifying the specific sub-location is returned to the mobile computing device.

## Claims

### Claim 1

A method comprising:
receiving, from a mobile computing device associated with a user, a search query comprising image query data and ambient audio data captured at a location of the mobile computing device;
in an automated operation performed using one or more computer processor devices, performing operations comprising:
filtering the ambient audio data using a low-pass filter to isolate background audio data;
comparing the filtered background audio data to a set of stored audio fingerprints respectively associated with a plurality of sub-locations of a geographic location;
generating location-identifying text based on a matching audio fingerprint of the set of stored audio fingerprints;
performing image analysis of the image query data and generating a set of image-derived keywords based on the image analysis;
compiling an augmented search query that incorporates the location-identifying text and the set of image-derived keywords; and
identifying as a search result a specific sub-location of the plurality of sub-locations based at least in part on determining a correlation between the augmented search query and a respective sub-location index of the specific sub-location; and
returning the search result to the mobile computing device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the location-identifying text comprises one or more identifiers of the specific sub-location, of one or more displays at the specific sub-location, or of one or more content items located within a predetermined proximity of the specific sub-location.

### Claim 3 (depends on 1)

The method of claim 1, further comprising, prior to receiving the search query:
for each sub-location of the plurality of sub-locations, compiling and storing on a machine-readable memory the respective sub-location index, each respective sub-location index comprising a set of content items comprising image data captured at the respective sub-location and text data associated with the respective sub-location.

### Claim 4 (depends on 3)

The method of claim 3, wherein the text data of each respective sub-location index comprises verbiage frequently spoken at the respective sub-location.

### Claim 5 (depends on 3)

The method of claim 3, wherein the text data of each respective sub-location index comprises a textual description of the respective sub-location.

### Claim 6 (depends on 1)

The method of claim 1, wherein the performing of the image analysis comprises:
performing object recognition to identify multiple distinct objects within the image query data; and
generating a respective keyword of the set of image-derived keywords for each identified object of the multiple distinct objects.

### Claim 7 (depends on 6)

The method of claim 6, wherein the performing of the object recognition comprises:
parsing the image query data to identify different components and aspects thereof; and
generating descriptive keywords for each identified component and aspect.

### Claim 8 (depends on 1)

The method of claim 1, wherein the performing of the image analysis comprises:
providing the image query data to a third-party provider;
receiving keywords from the third-party provider based on analysis of the image query data by the third-party provider; and
incorporating the received keywords into the set of image-derived keywords.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
receiving voice audio data associated with the image query data;
performing voice recognition on the voice audio data to generate textual data; and
incorporating the generated textual data into the augmented search query.

### Claim 10 (depends on 9)

The method of claim 9, wherein the performing of the voice recognition comprises:
filtering the voice audio data using at least one of a band-pass filter and a high-pass filter to isolate voice frequencies; and
generating text from the filtered voice audio data.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
analyzing received audio data to identify spoken descriptions of content captured in the image query data; and
incorporating text corresponding to the identified spoken descriptions into the augmented search query.

### Claim 12 (depends on 1)

The method of claim 1, wherein the returning of the search result comprises directing the mobile computing device to the specific sub-location indicated by the search result.

### Claim 13 (depends on 1)

The method of claim 1, wherein the geographic location is a museum, and each sub-location of the plurality of sub-locations comprises a respective display within the museum.

### Claim 14 (depends on 1)

The method of claim 1, wherein the ambient audio data and the image query data are captured contemporaneously at the location of the mobile computing device.

### Claim 15 (depends on 1)

The method of claim 1, wherein the ambient audio data is time-stamped with capture time data, and wherein the location-identifying text is time-stamped based on the capture time data.

### Claim 16

A system comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, configure the system to perform operations comprising:
receiving, from a mobile computing device associated with a user, a search query comprising image query data and ambient audio data captured at a location of the mobile computing device;
filtering the ambient audio data using a low-pass filter to isolate background audio data;
comparing the filtered background audio data to a set of stored audio fingerprints respectively associated with a plurality of sub-locations of a geographic location;
generating location-identifying text based on a matching audio fingerprint of the set of stored audio fingerprints;
performing image analysis of the image query data and generating a set of image-derived keywords based on the image analysis;
compiling an augmented search query that incorporates the location-identifying text and the set of image-derived keywords;
identifying as a search result a specific sub-location of the plurality of sub-locations based at least in part on determining a correlation between the augmented search query and a respective sub-location index of the specific sub-location; and
returning the search result to the mobile computing device.

### Claim 17 (depends on 16)

The system of claim 16, wherein the instructions configure the system to perform further operations comprising:
for each sub-location of the plurality of sub-locations, compiling and storing on a machine-readable memory the respective sub-location index, each respective sub-location index comprising a set of content items comprising image data captured at the respective sub-location and text data comprising verbiage frequently spoken at the respective sub-location.

### Claim 18 (depends on 16)

The system of claim 16, wherein the instructions configure the system to perform further operations comprising:
performing object recognition to identify multiple distinct objects within the image query data; and
generating a respective keyword of the set of image-derived keywords for each identified object of the multiple distinct objects.

### Claim 19 (depends on 16)

The system of claim 16, wherein the instructions configure the system to perform further operations comprising:
receiving voice audio data associated with the image query data;
performing voice recognition on the voice audio data to generate textual data; and
incorporating the generated textual data into the augmented search query.

### Claim 20

A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
receiving, from a mobile computing device associated with a user, a search query comprising image query data and ambient audio data captured at a location of the mobile computing device;
filtering the ambient audio data using a low-pass filter to isolate background audio data;
comparing the filtered background audio data to a set of stored audio fingerprints respectively associated with a plurality of sub-locations of a geographic location;
generating location-identifying text based on a matching audio fingerprint of the set of stored audio fingerprints;
performing image analysis of the image query data and generating a set of image-derived keywords based on the image analysis;
compiling an augmented search query that incorporates the location-identifying text and the set of image-derived keywords;
identifying as a search result a specific sub-location of the plurality of sub-locations based at least in part on determining a correlation between the augmented search query and a respective sub-location index of the specific sub-location; and
returning the search result to the mobile computing device.
