# US 2026/0187144 A1 · Video and Audio Multimodal Searching System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0187144 A1
- Document ID: US20260187144A1
- Status: Published application
- Kind: A1
- Application number: 19/543,229
- Filed: 2026-02-18
- Published: 2026-07-02
- Earliest claimed: 2023-05-31
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Harshit Kharbanda; Belinda Luna Zeng; Viviana Caso Corella; Aashi Jain; David William Hendon; Christopher James Kelley; Jessica Lee; Dounia Berrada; Kai Yu; Louis Wang; Thomas J. Duerig; Radu Soricut; Robin Dua
- Web page: https://patentgrep.com/patent/US20260187144A1

## CPC

- G06F 16/71: Indexing; Data structures therefor; Storage structures
- G06F 16/735: Filtering based on additional data, e.g. user or group profiles
- G06F 16/732: Query formulation
- G06F 16/7328: {Query by example, e.g. a complete video frame or video sequence (graphical querying G06F16/7335)}
- G06F 16/7834: {using audio features}
- G06F 16/787: using geographical or spatial information, e.g. location
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- G06V 10/82: using neural networks
- G06V 2201/07: Target detection
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A multimodal search system using a video query is described. The system can receive video data captured by a camera of a user device. The video data can have a sequence of image frames. Additionally, the system can receive audio data associated with the video data captured by the user device. Moreover, the system can process, using one or more machine-learned models, the sequence of image frames to generate video embeddings related to the sequence of the image frames. The video embeddings can have a plurality of image embeddings associated with the sequence of image frames. Furthermore, the system can determine one or more video results based on the video embeddings and the audio data. Subsequently, the system can transmit, to the user device, the one or more video results.

## Claims

### Claim 1

A computer-implemented method for multimodal searching of video results, the method comprising:
receiving, by a computing system comprising one or more processors, image data;
receiving audio data;
processing, using one or more machine-learned models, the image data to generate an image embedding;
determining one or more video results based on the image embedding and the audio data; and
transmitting, to the user device, the one or more video results.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
processing, using the one or more machine-learned models, the image data to generate temporal information; and
wherein the determination of the one or more video results is further based on the temporal information.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
processing, using an automatic speech recognition technique, the audio data to generate a text query; and
wherein the determination of the one or more video results is further based on the text query.

### Claim 4 (depends on 1)

The method of claim 1, wherein the image data is associated with a sequence of image frames, the method further comprising:
selecting, using a frame selection algorithm, a subset of image frames from the sequence of image frames;
processing each image frame in the subset of image frames to generate a plurality of image embeddings, the plurality of image embeddings having an image embedding for each image frame in the subset of image frames; and
wherein the determination of the one or more video results is further based on the plurality of image embeddings.

### Claim 5 (depends on 4)

The method of claim 4, wherein the frame selection algorithm is a uniform random sampling of frames, wherein the subset of image frames is selected at a regular interval.

### Claim 6 (depends on 4)

The method of claim 4, wherein the frame selection algorithm is based on a position of the camera and an orientation of the camera.

### Claim 7 (depends on 4)

The method of claim 4, the method further comprising:
detecting a target object in the image data; and
wherein the frame selection algorithm is based on a spatial relationship between the camera and the target object.

### Claim 8 (depends on 7)

The method of claim 7, wherein the spatial relationship includes a translation vector that represents a position of the camera in a three-dimensional space relative to the target object, the translation vector specifying a distance that the camera is displaced with respect to the target object.

### Claim 9 (depends on 7)

The method of claim 7, wherein the spatial relationship includes a rotation matrix that represents an orientation of the camera in a three-dimensional space relative to the target object, the orientation matrix specifying a degree of rotation of the camera with respect to the target object.

### Claim 10 (depends on 4)

The method of claim 4, wherein the frame selection algorithm is running on the user device.

### Claim 11 (depends on 1)

The method of claim 1, the method further comprising:
processing the audio data to generate an input audio signature; and
wherein the determination of the one or more video results is further based on the input audio signature.

### Claim 12 (depends on 11)

The method of claim 11, wherein the determination of the one or more search results comprises:
detecting a target object in the image data;
accessing, from an audio signature database a plurality of known audio signatures that are associated with the target object;
selecting a matching audio signature from the known audio signatures, wherein a comparison score of the matching audio signature exceeds a threshold value, the matching score of the matching audio signature being calculated by comparing the input audio signature with the matching audio signature; and
wherein the one or more video results are further determined based on the matching audio signature.

### Claim 13 (depends on 1)

The method of claim 1, wherein the image data includes a plurality of image frames, the method comprises:
processing, using the one or more machine-learned model, each image frame in the plurality of image frames to generate a video embedding;
mapping the generated video embedding to an index of video embeddings; and
wherein the one or more video results are determined based on the mapping of the generated video embedding to the index of video embeddings.

### Claim 14 (depends on 13)

The method of claim 13, wherein the mapping of the generated video embedding to the index of video embeddings comprises:
ranking each video embedding in the index of video embeddings based on a comparison with the generated video embedding, each video embedding being associated with a video result; and
wherein the one or more video results are determined based on the ranking of each video embedding in the index of video embeddings.

### Claim 15 (depends on 1)

The method of claim 1, where the one or more machine-learned model includes a polymath model that is trained using tutorial videos publicly available on an online video sharing platform.

### Claim 16 (depends on 1)

The method of claim 1, wherein the one or more machine-learned model includes a multimodal multitask unified model that is trained to understand information from a plurality of formats, the plurality of formats including video and text.

### Claim 17 (depends on 1)

The method of claim 1, further comprising:
determining one or more web results based on the image embedding; and
transmitting, to the user device, the one or more web results.

### Claim 18 (depends on 1)

The method of claim 1, further comprising:
processing, using the one or more machine-learned models, the audio data with the image data to generate a video embedding, and
wherein the determination of the one or more video results is further based on the video embedding.

### Claim 19

A computer-implemented method for multimodal searching of video results, the method comprising:
capturing, by a camera of a user device, image data;
capturing, by a microphone of the user device, audio data associated with the image data;
processing, using one or more machine-learned models stored on the user device, the audio data and the image data to generate an image embedding;
transmitting, to a server, the image embedding and the audio data;
in response to the transmission of the image embedding and the audio data, receiving one or more video results from the server;
presenting, on a display of the user device, the one or more video results.

### Claim 20

A computing system, the system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving image data;
receiving audio data;
processing, using one or more machine-learned models, the image data to generate an image embedding;
determining one or more video results based on the image embedding and the audio data; and
transmitting, to the user device, the one or more video results.
