# US 2024/0193399 A1 · Learning language representation with logical inductive bias

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0193399 A1
- Document ID: US20240193399A1
- Status: Published application
- Kind: A1
- Application number: 18/077,723
- Filed: 2022-12-08
- Published: 2024-06-13
- Earliest claimed: 2022-12-08
- Assignees: Tencent America LLC
- Applicants: Tencent America LLC
- Inventors: Jianshu CHEN
- Web page: https://patentgrep.com/patent/US20240193399A1

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method including receiving input comprising natural language texts; pre-training a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, the FOLNet neural network model comprising of a plurality of layers; processing the input through the plurality of layers of the FOLNet neural network model; encoding a logical inductive bias using the FOLNet neural network model; outputting one or more tensors based on the logical inductive bias; and predicting an outcome using the one or more tensors.

## Claims

### Claim 1

A method executed by at least one processor, the method comprising:
receiving input comprising natural language texts;
pre-training a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, the FOLNet neural network model comprising of a plurality of layers;
processing the input through the plurality of layers of the FOLNet neural network model;
encoding a logical inductive bias using the FOLNet neural network model;
outputting one or more tensors based on the logical inductive bias; and
predicting an outcome using the one or more tensors.

### Claim 2 (depends on 1)

The method according to claim 1, wherein outputting the one or more tensors comprises preprocessing the input into one or more tokens to be converted into the one or more tensors.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the FOLNet neural network model further comprises a first interacting branch for unary relational reasoning and a second interacting branch for binary relational reasoning.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising constructing a set of neural logic operators as learnable Horn clauses.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the set of neural logic operators are forward-chained into the FOLNet neural network model.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the FOLNet neural network model is a fully differentiable neural architecture.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the pre-training the neural network model on unlabeled texts comprises training the neural network model to solve one or more downstream tasks.

### Claim 8

An apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
receiving code configured to cause the at least one processor to receive input comprising natural language texts;
pre-training code configured to cause the at least one processor to pre-train a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, the FOLNet neural network model comprising of a plurality of layers;
processing code configured to cause the at least one processor to process the input through the plurality of layers of the FOLNet neural network model;
encoding code configured to cause the at least one processor to encode a logical inductive bias using the FOLNet neural network model;
outputting code configured to cause the at least one processor to output one or more tensors based on the logical inductive bias; and
predicting code configured to cause the at least one processor to predict an outcome using the one or more tensors.

### Claim 9 (depends on 8)

The apparatus according to claim 8, wherein the outputting code is further configured to cause the at least one processor to preprocess the input into one or more tokens to be converted into the one or more tensors.

### Claim 10 (depends on 8)

The apparatus according to claim 8, wherein the FOLNet neural network model further comprises a first interacting branch for unary relational reasoning and a second interacting branch for binary relational reasoning.

### Claim 11 (depends on 8)

The apparatus according to claim 8, wherein the program code further comprises constructing code configured to cause the at least one processor to construct a set of neural logic operators as learnable Horn clauses.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the set of neural logic operators are forward-chained into the FOLNet neural network model.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the FOLNet neural network model is a fully differentiable neural architecture.

### Claim 14 (depends on 8)

The apparatus according to claim 8, wherein the pre-training code is further configured to cause the at least one processor to train the neural network model to solve one or more downstream tasks.

### Claim 15

A non-transitory computer-readable storage medium, storing instructions, which, when executed by at least one processor, cause the at least one processor to:
receive input comprising natural language texts;
pre-train a First-Order Logic Network (FOLNet) neural network model on unlabeled texts included in the natural language texts, the FOLNet neural network model comprising of a plurality of layers;
process the input through the plurality of layers of the FOLNet neural network model;
encode a logical inductive bias using the FOLNet neural network model;
output one or more tensors based on the logical inductive bias; and
predict an outcome using the one or more tensors.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the instructions further cause the at least one processor to preprocess the input into one or more tokens to be converted into the one or more tensors.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the FOLNet neural network model further comprises a first interacting branch for unary relational reasoning and a second interacting branch for binary relational reasoning.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium according to claim 15, wherein the instructions further cause the at least one processor to construct a set of neural logic operators as learnable Horn clauses.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the set of neural logic operators are forward-chained into the FOLNet neural network model.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium according to claim 19, wherein the FOLNet neural network model is a fully differentiable neural architecture.
