# US 2023/0410155 A1 · Systems and methods for gating-enhanced multi-task neural networks with feature interaction learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410155 A1
- Document ID: US20230410155A1
- Status: Published application
- Kind: A1
- Application number: 18/250,525
- Filed: 2021-07-07
- Published: 2023-12-21
- Earliest claimed: 2021-07-07
- Assignees: Baidu USA LLC; Baidu.Com Times Technology (Beijing) Co., Ltd.
- Applicants: Baidu USA, LLC; Baidu.com Times Technology (Beijing) Co., Ltd.
- Inventors: Hongliang FEI; Jingyuan ZHANG; Xingxuan ZHOU; Junhao ZHAO; Banghu YIN; Ping LI
- Web page: https://patentgrep.com/patent/US20230410155A1

## CPC

- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06Q 30/0269: {based on user profile or attribute}
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Deep neural network (DNN) models have been widely used for user-relevance content prediction. Presented herein is a new user-relevance framework, embodiments of which may be referred as Gating-Enhanced Multi-task Neural Networks (GemNN). In one or more, neural network-based multi-task learning model embodiments herein predict user engagement with content in a coarse-to-fine manner, which gradually reduces content candidates and allows parameter sharing from upstream tasks to downstream tasks to improve the training efficiency. Also, in one or more embodiments, a gating mechanism was introduced between embedding layers and multi-layer perceptions to learn feature interactions and control the information flow fed to MLP layers. Tested embodiments demonstrated considerable improvements over prior approaches.

## Claims

### Claim 1

A computer-implemented method for delivering content to a user, the method comprising:
given a user query comprising one or more words and a first set of candidate contents, using a user-content ranking task model of a gating-enhanced multi-task neural network (GemNN) model, in which the user-content ranking task model of the GemNN model receives the user query and user-related data as inputs, to obtain a set of top candidate contents from the set of candidate contents and a projected user-selection rate for each of the top candidate contents in the set of top candidate contents;
using a content-style matching task model of the GemNN model and the set of top candidate contents, the projected user-selection rate for each of the top candidate contents, and potential style types for the top candidate contents as inputs to the content-style matching task model of the GemNN model to obtain a set of ranked style type information for at least some of the candidate contents in the set of top candidate contents based upon a predicted probability of being selected by the user;
using a user-content-style ranking task model of the GemNN model and the set of top candidate contents, the projected user-selection rate for each of the top candidate contents, and the set of ranked style type information for at least some of the candidate contents in the set of top candidate as inputs to the user-content-style ranking task model of the GemNN model to obtain a final set of candidate contents with style type information based at least in part on predicted probability of being selected by the user; and
outputting the final set of candidate contents with style type information.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein at least one of the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model comprises:
a gating layer that is positioned between an embedding layer and a Multi-Layer Perception (MLP) and performs gating on concatenated features instead of each feature field individually.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2 wherein at least one of the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model comprises:
the embedding layer that receives input features and embeds them into embeddings;
a concatenation layer that concatenates the embeddings;
a batch normalization layer that receives an output from the concatenation layer;
the gating layer that receives the concatenated features from the normalization layer and performs gating on the concatenated feature neuron units; and
the Multi-Layer Perception (MLP) that receives an output of the gating layer.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein:
parameters of a first layer of a Multi-Layer Perception (MLP) from the user-content ranking task model of the GemNN and the projected user-selection rate for each of the top candidate contents in the set of top candidate contents from the user-content ranking task model of the GemNN are shared with:
the content-style matching task model of the GemNN model; and
the user-content-style ranking task model of the GemNN model.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4 wherein the shared parameters are concatenated with:
an embedding layer of the content-style matching task model of the GemNN model; and
an embedding layer of the user-content-style ranking task model of the GemNN model.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4 wherein:
the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model are jointly optimized during training, which comprises updating the parameters of the first layer of the Multi-Layer Perception (MLP) from the user-content ranking task model while freezing the shared projected user-selection rate for each of the top candidate contents in the set of top candidate contents from the user-content ranking task model.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1 wherein the content-style matching task model of the GemNN model comprises:
a two-tower neural network model to match content and possible style types, in which style-type feature embeddings are pre-computed and indexed.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1 wherein the contents are advertisements.

### Claim 9

A system comprising:
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
given a user query comprising one or more words and a first set of candidate contents, using a user-content ranking task model of a gating-enhanced multi-task neural network (GemNN) model, in which the user-content ranking task model of the GemNN model receives the user query and user-related data as inputs, to obtain a set of top candidate contents from the set of candidate contents and a projected user-selection rate for each of the top candidate contents in the set of top candidate contents;
using a content-style matching task model of the GemNN model and the set of top candidate contents, the projected user-selection rate for each of the top candidate contents, and potential style types for the top candidate contents as inputs to the content-style matching task model of the GemNN model to obtain a set of ranked style type information for at least some of the candidate contents in the set of top candidate contents based upon a predicted probability of being selected by the user;
using a user-content-style ranking task model of the GemNN model and the set of top candidate contents, the projected user-selection rate for each of the top candidate contents, and the set of ranked style type information for at least some of the candidate contents in the set of top candidate as inputs to the user-content-style ranking task model of the GemNN model to obtain a final set of candidate contents with style type information based at least in part on predicted probability of being selected by the user; and
outputting the final set of candidate contents with style type information.

### Claim 10 (depends on 9)

The system of claim 9 wherein at least one of the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model comprises:
a gating layer that is positioned between an embedding layer and a Multi-Layer Perception (MLP) and performs gating on concatenated features instead of each feature field individually.

### Claim 11 (depends on 10)

The system of claim 10 wherein at least one of the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model comprises:
the embedding layer that receives input features and embeds them into embeddings;
a concatenation layer that concatenates the embeddings;
a batch normalization layer that receives an output from the concatenation layer;
the gating layer that receives the concatenated features from the normalization layer and performs gating on the concatenated feature neuron units; and
the Multi-Layer Perception (MLP) that receives an output of the gating layer.

### Claim 12 (depends on 9)

The system of claim 9 wherein:
parameters of a first layer of a Multi-Layer Perception (MLP) from the user-content ranking task model of the GemNN and the projected user-selection rate for each of the top candidate contents in the set of top candidate contents from the user-content ranking task model of the GemNN are shared with:
the content-style matching task model of the GemNN model; and
the user-content-style ranking task model of the GemNN model.

### Claim 13 (depends on 12)

The system of claim 12 wherein the shared parameters are concatenated with:
an embedding layer of the content-style matching task model of the GemNN model; and
an embedding layer of the user-content-style ranking task model of the GemNN model.

### Claim 14 (depends on 12)

The system of claim 12 wherein:
the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model are jointly optimized during training, which comprises updating the parameters of the first layer of the Multi-Layer Perception (MLP) from the user-content ranking task model while freezing the shared projected user-selection rate for each of the top candidate contents in the set of top candidate contents from the user-content ranking task model.

### Claim 15 (depends on 9)

The system of claim 9 wherein the content-style matching task model of the GemNN model comprises:
a two-tower neural network model to match content and possible style types, in which style-type feature embeddings are pre-computed and indexed.

### Claim 16

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
given a user query comprising one or more words and a first set of candidate contents, using a user-content ranking task model of a gating-enhanced multi-task neural network (GemNN) model, in which the user-content ranking task model of the GemNN model receives the user query and user-related data as inputs, to obtain a set of top candidate contents from the set of candidate contents and a projected user-selection rate for each of the top candidate contents in the set of top candidate contents;
using a content-style matching task model of the GemNN model and the set of top candidate contents, the projected user-selection rate for each of the top candidate contents, and potential style types for the top candidate contents as inputs to the content-style matching task model of the GemNN model to obtain a set of ranked style type information for at least some of the candidate contents in the set of top candidate contents based upon a predicted probability of being selected by the user;
using a user-content-style ranking task model of the GemNN model and the set of top candidate contents, the projected user-selection rate for each of the top candidate contents, and the set of ranked style type information for at least some of the candidate contents in the set of top candidate as inputs to the user-content-style ranking task model of the GemNN model to obtain a final set of candidate contents with style type information based at least in part on predicted probability of being selected by the user; and
outputting the final set of candidate contents with style type information.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 wherein at least one of the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model comprises:
a gating layer that is positioned between an embedding layer and a Multi-Layer Perception (MLP) and performs gating on concatenated features instead of each feature field individually.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium or media of claim 17 wherein at least one of the user-content ranking task model, the content-style matching task model, and the user-content-style ranking task model comprises:
the embedding layer that receives input features and embeds them into embeddings;
a concatenation layer that concatenates the embeddings;
a batch normalization layer that receives an output from the concatenation layer;
the gating layer that receives the concatenated features from the normalization layer and performs gating on the concatenated feature neuron units; and
the Multi-Layer Perception (MLP) that receives an output of the gating layer.

### Claim 19 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 wherein:
parameters of a first layer of a Multi-Layer Perception (MLP) from the user-content ranking task model of the GemNN and the projected user-selection rate for each of the top candidate contents in the set of top candidate contents from the user-content ranking task model of the GemNN are shared with:
the content-style matching task model of the GemNN model; and
the user-content-style ranking task model of the GemNN model.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium or media of claim 19 wherein the shared parameters are concatenated with:
an embedding layer of the content-style matching task model of the GemNN model; and
an embedding layer of the user-content-style ranking task model of the GemNN model.
