# US 2025/0094774 A1 · Implementing dialog-based music recommendations for videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094774 A1
- Document ID: US20250094774A1
- Status: Published application
- Kind: A1
- Application number: 18/368,253
- Filed: 2023-09-14
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: Lemon Inc.
- Applicants: Lemon Inc.
- Inventors: Peng Zhang; Zhikang Dong; Bin Chen; Xiulong Liu
- Web page: https://patentgrep.com/patent/US20250094774A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure describes techniques for implementing dialog-based music recommendations for input videos. A video is input into a machine learning model. The machine learning model comprises a first sub-model and a second sub-model. At least one music track is identified by the first sub-model based on the input video. A first set of sentences is generated by the second sub-model. The first set of sentences depicts the at least one music track and recommendation reasons in natural languages. The first set of sentences is caused to be displayed on a computing device associated with a user. In response to receiving input indicating that the user prefers music different from the at least one music track, at least one other music track is identified. A second set of sentences depicting the at least one other music track in natural languages is generated for display on the computing device.

## Claims

### Claim 1

A method of implementing dialog-based music recommendations for input videos, comprising:
inputting a video into a machine learning model, wherein the machine learning model comprises a first sub-model and a second sub-model, and the machine learning model is configured to implement the dialog-based music recommendations for the input videos;
identifying at least one music track by the first sub-model based on the input video;
generating a first set of sentences by the second sub-model, the first set of sentences depicting the at least one music track and recommendation reasons in natural languages;
causing to display the first set of sentences on a computing device associated with a user;
in response to receiving input indicating that the user prefers music different from the at least one music track, identifying at least one other music track based on the input video, previously recommended music track, and the input indicative of the user's preference; and
generating a second set of sentences depicting the at least one other music track in natural languages for display on the computing device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine learning model is trained on conversational music recommendation data, and wherein the conversational music recommendation data is generated by:
calculating a cosine similarity between each of a plurality of videos and each of a set of music tracks;
selecting candidate music tracks corresponding to each of the plurality of videos based on cosine similarities between each of the plurality of videos and the set of music tracks;
determining music tags and music metadata of an original music track associated with each of the plurality of videos and the candidate music tracks corresponding to each of the plurality of videos; and
generating simulated music recommendation conversations based on the music tags and the music metadata using a generative pre-training transformer (GPT).

### Claim 3 (depends on 2)

The method of claim 2, wherein each of the simulated music recommendation conversation begins with recommending one of the candidate music tracks with associated reasons, and then recommends the original music track with reasonings based on recognizing musical differences between the one of the candidate music tracks and the original music track and in response to a simulated input indicating a preference for the original music track.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
generating simulated music recommendation training data in which each sample contains information indicating a video, an original music track associated with the video, at least one candidate music track corresponding to the video, and simulated conversation text.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first sub-model is trained to pair at least one music track from a music pool to fit an overall video, and wherein the first sub-model is trained to alter music recommendation from a previously recommended music track to a currently recommended music track.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first sub-model comprises a trainable linear projection layer to project video modality, music modality, and text modality into a same embedding space.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first sub-model is trained on video data, music data, and text data comprising simulated music recommendation conversation text.

### Claim 8 (depends on 1)

The method of claim 1, wherein the second sub-model is trained to express music recommendations and recommendation reasons in natural languages.

### Claim 9 (depends on 1)

The method of claim 1, wherein the second sub-model comprises trainable linear projection layers, and wherein the second sub-model is trained on data each instance of which comprises a music representation from the first sub-model and corresponding recommendation reasoning statements from simulated conversational music recommendation dataset.

### Claim 10

A system, comprising:
at least one processor; and
at least one memory comprising computer-readable instructions that upon execution by the at least one processor cause the system to perform operations comprising:
inputting a video into a machine learning model, wherein the machine learning model comprises a first sub-model and a second sub-model, and the machine learning model is configured to implement the dialog-based music recommendations for the input videos;
identifying at least one music track by the first sub-model based on the input video;
generating a first set of sentences by the second sub-model, the first set of sentences depicting the at least one music track and recommendation reasons in natural languages;
causing to display the first set of sentences on a computing device associated with a user;
in response to receiving input indicating that the user prefers music different from the at least one music track, identifying at least one other music track based on the input video, previously recommended music track, and the input indicative of the user's preference; and
generating a second set of sentences depicting the at least one other music track in natural languages for display on the computing device.

### Claim 11 (depends on 10)

The system of claim 10, wherein the machine learning model is trained on conversational music recommendation data, and wherein the conversational music recommendation data is generated by:
calculating a cosine similarity between each of a plurality of videos and each of a set of music tracks;
selecting candidate music tracks corresponding to each of the plurality of videos based on cosine similarities between each of the plurality of videos and the set of music tracks;
determining music tags and music metadata of an original music track associated with each of the plurality of videos and the candidate music tracks corresponding to each of the plurality of videos; and
generating simulated music recommendation conversations based on the music tags and the music metadata using a generative pre-training transformer (GPT).

### Claim 12 (depends on 11)

The system of claim 11, wherein each of the simulated music recommendation conversation begins with recommending one of the candidate music tracks with associated reasons, and then recommends the original music track with reasonings based on recognizing musical differences between the one of the candidate music tracks and the original music track and in response to a simulated input indicating a preference for the original music track.

### Claim 13 (depends on 11)

The system of claim 11, the operations further comprising:
generating simulated music recommendation training data in which each sample contains information indicating a video, an original music track associated with the video, at least one candidate music track corresponding to the video, and simulated conversation text.

### Claim 14 (depends on 10)

The system of claim 10, wherein the first sub-model is trained to pair at least one music track from a music pool to fit an overall video, and wherein the first sub-model is trained to alter music recommendation from a previously recommended music track to a currently recommended music track.

### Claim 15 (depends on 10)

The system of claim 10, wherein the second sub-model is trained to express music recommendations and recommendation reasons in natural languages.

### Claim 16

A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations, the operation comprising:
inputting a video into a machine learning model, wherein the machine learning model comprises a first sub-model and a second sub-model, and the machine learning model is configured to implement the dialog-based music recommendations for the input videos;
identifying at least one music track by the first sub-model based on the input video;
generating a first set of sentences by the second sub-model, the first set of sentences depicting the at least one music track and recommendation reasons in natural languages;
causing to display the first set of sentences on a computing device associated with a user;
in response to receiving input indicating that the user prefers music different from the at least one music track, identifying at least one other music track based on the input video, previously recommended music track, and the input indicative of the user's preference; and
generating a second set of sentences depicting the at least one other music track in natural languages for display on the computing device.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the machine learning model is trained on conversational music recommendation data, and wherein the conversational music recommendation data is generated by:
calculating a cosine similarity between each of a plurality of videos and each of a set of music tracks;
selecting candidate music tracks corresponding to each of the plurality of videos based on cosine similarities between each of the plurality of videos and the set of music tracks;
determining music tags and music metadata of an original music track associated with each of the plurality of videos and the candidate music tracks corresponding to each of the plurality of videos; and
generating simulated music recommendation conversations based on the music tags and the music metadata using a generative pre-training transformer (GPT).

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein each of the simulated music recommendation conversation begins with recommending one of the candidate music tracks with associated reasons, and then recommends the original music track with reasonings based on recognizing musical differences between the one of the candidate music tracks and the original music track and in response to a simulated input indicating a preference for the original music track.

### Claim 19 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the first sub-model is trained to pair at least one music track from a music pool to fit an overall video, and wherein the first sub-model is trained to alter music recommendation from a previously recommended music track to a currently recommended music track.

### Claim 20 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the second sub-model is trained to express music recommendations and recommendation reasons in natural languages.
