# US 2022/0277015 A1 · Processing Queries using an Attention-Based Ranking System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277015 A1
- Document ID: US20220277015A1
- Status: Published application
- Kind: A1
- Application number: 17/187,449
- Filed: 2021-02-26
- Published: 2022-09-01
- Earliest claimed: 2021-02-26
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Chiqun ZHANG; Michael Robert EVANS; Maksim LEPIKHIN; Dragomir Dimitrov YANKOV
- Web page: https://patentgrep.com/patent/US20220277015A1

## CPC

- G06F 16/24578: {using ranking}
- G06F 16/43: Querying
- G06F 16/29: Geographical information databases
- G06F 16/338: Presentation of query results
- G06F 16/90335: {Query processing}
- G06F 16/90344: {by using string matching techniques}
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06F 16/387: using geographical or spatial information, e.g. location
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Technology is described herein for ranking candidate result items in at least two stages. In a first stage, the technology uses a first attention-based neural network to determine an extent of attention that each token of an input query should pay to the tokens of each candidate result item. In a second stage, the technology uses a ranking subsystem to perform listwise inference on output results provided by the first stage, to generate a plurality of ranking scores that establish an order of relevance of the candidate results items. The ranking subsystem may use a second attention-based neural network to perform the listwise inference. According to some implementations, the technology is configured to process queries and candidate result items having different kinds and combinations of features. For instance, one kind of input query may include text-based features, structure-based features, and geographic-based features.

## Claims

### Claim 1

A method, using one more computing devices, for ranking a plurality of candidate result items in response to a query, comprising:
obtaining the query and the plurality of candidate result items that match the query, the query comprising one or more tokens, and each candidate result item comprising one or more tokens;
generating, using a first attention-based neural network, a plurality of instances of query-to-result similarity information respectively corresponding to the plurality of candidate result items, each instance of query-to-result similarity information expressing an extent of attention that each token of the query should pay to each token of a particular candidate result item;
generating, using a ranking subsystem, plural ranking scores based on the plurality of instances of query-to-result information, the plural ranking scores identifying an order of relevance of the plurality of candidate result items to the query; and
identifying at least a top-ranking candidate result item to be presented to a user based on the plural scores.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the first attention-based neural network uses a set of original features to describe the query and the plurality of candidate result items,
wherein the plurality of instances of query-to-result information generated by the first attention-based neural network are instances of original query-to-result information,
wherein processing performed by the ranking subsystem comprises:
combining the plurality of instances of original query-to-result similarity information with supplemental feature information pertaining to the query, to generate a plurality of respective instances of enhanced similarity information;
generating, using a second attention-based neural network, a plurality of instances of result-to-result context information using self-attention, each instance of result-to-result context information expressing an extent of attention that a particular instance of enhanced similarity information should pay to other instances of enhanced similarity information; and
mapping, using a scoring neural network, the plurality of instances of result-to-result context information into the plural respective ranking scores.

### Claim 3 (depends on 2)

The method of claim 2, wherein the set of original features includes at least text-based features that describe text associated with the query and each candidate result item.

### Claim 4 (depends on 3)

The method claim 3, wherein the set of original features also includes at least structure-based features that describe structural composition of the query and each candidate result item.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first attention-based neural network operates on: one or more instances of query embedding information associated with the query; and one or more instances of result embedding information associated with each candidate result item.

### Claim 6 (depends on 5)

The method of claim 5, wherein the method generates said one or more instances of query embedding information by:
identifying said one or more tokens of the query, to overall generate one or more instances of query token information;
identifying a type associated which each token of the query, to overall generate one or more instances of query type information;
mapping said one or more instances of query token information to respective one or more query token embeddings;
mapping said one or more instances of query type information to respective one or more query type embeddings; and
combining said one or more query token embeddings and said one or more query type embeddings to generate said one or more instances of query embedding information.

### Claim 7 (depends on 5)

The method of claim 5, wherein the first attention-based neural network generates preliminary output results, and wherein the first attention-based neural network generates the plurality of instances of query-to-result similarity information by generating an element-by-element difference of the preliminary output results and said one or more instances of query embedding information.

### Claim 8 (depends on 7)

The method of claim 7, wherein the first attention-based neural network forms each instance of query-to-result information for a particular pairing of the query and a specified candidate result item by summing a square of the differences associated with the particular pairing.

### Claim 9 (depends on 2)

The method of claim 2, wherein the supplemental feature information describes a geographical context in which the query was made.

### Claim 10 (depends on 9)

The method of claim 9, wherein the geographical context refers to a position associated with the user who submitted the query.

### Claim 11 (depends on 9)

The method of claim 9, wherein the geographical context refers to a portion of a digital map specified by the user.

### Claim 12 (depends on 2)

The method of claim 2, wherein the second attention-based neural network operates by:
mapping the plurality of instances of enhanced similarity information to a plurality of respective instances of projected feature information;
performing self-attention on the plurality of instances of projected feature information to generate a plurality of respective instances of attention feature information; and
combining the plurality of instances of projected feature information with the plurality of instances of attention feature information to respectively generate the plurality of instances of result-to-result context information.

### Claim 13 (depends on 2)

The method of claim 2, wherein the scoring neural network includes at least two or more fully-connected neural network layers.

### Claim 14

A computing system for ranking a plurality of candidate result items in response to a query, the query comprising one or more tokens, and each candidate result item comprising one or more tokens, the computing system comprising:
hardware logic circuitry that implements a first attention-based neural network, a second attention-based neural network, and a scoring neural network,
the first attention-based neural network being configured to generate a plurality of instances of original query-to-result similarity information respectively corresponding to the plurality of candidate result items, each instance of original query-to-result similarity information expressing an extent of attention that each token of the query should pay to each token of a particular candidate result item;
the first attention-based neural network being configured to use a set of original features to describe the query and the plurality of candidate result items,
the plurality of instances of original query-to-result similarity information being combined with supplemental feature information pertaining to the query, to generate a plurality of respective instances of enhanced similarity information;
the second attention-based neural network being configured to generate a plurality of instances of result-to-result context information using self-attention, each instance of result-to-result context information expressing an extent of attention that a particular instance of enhanced similarity information should pay to other instances of enhanced similarity information; and
the scoring neural network being configured to map the plurality of instances of result-to-result context information into plural respective ranking scores, the plural ranking scores identifying an order of relevance of the plurality of candidate result items to the query.

### Claim 15 (depends on 14)

The computing system of claim 14, wherein the set of original features includes at least text-based features that describe text associated with the query and each candidate result item, and wherein the supplemental feature information describes a geographical context in which the query was made.

### Claim 16 (depends on 14)

The computing system of claim 14, wherein the first attention-based neural network is configured to operate on: one or more instances of query embedding information associated with the query; and one or more instances of instances of result embedding information associated with each candidate result item.

### Claim 17 (depends on 16)

The computing system of claim 16, wherein the first attention-based neural network generates preliminary output results, and wherein the first attention-based neural network generates the plurality of instances of query-to-result similarity information by generating an element-by-element difference of the preliminary output results and said one or more instances of query embedding information.

### Claim 18 (depends on 14)

The computing system of claim 14, wherein the second attention-based neural network is configured to:
map the plurality of instances of enhanced similarity information to a plurality of respective instances of projected feature information;
perform self-attention on the plurality of instances of projected feature information to generate a plurality of respective instances of attention feature information; and
combine the plurality of instances of projected feature information with the plurality of instances of attention feature information to respectively generate the plurality of instances of result-to-result context information.

### Claim 19

A computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more hardware processors, performing a method that comprises:
obtaining a query and a plurality of candidate result items that match the query, the query comprising one or more tokens, and each candidate result item comprising one or more tokens;
generating, using a first attention-based neural network implemented by the computer-readable instructions, a plurality of instances of query-to-result similarity information respectively corresponding to the plurality of candidate result items, each instance of query-to-result similarity information expressing an extent of attention that each token of the query should pay to each token of a particular candidate result item,
wherein the first attention-based neural network generates preliminary output results, and wherein the first attention-based neural network generates the plurality of instances of query-to-result similarity information by generating an element-by-element difference of the preliminary output results and one or more instances of query embedding information produced from the query; and
generating plural ranking scores based on the plurality of instances of query-to-result information, the plural ranking scores identifying an order of relevance of the plurality of candidate result items to the query.

### Claim 20 (depends on 19)

The computer-readable storage medium of claim 19,
wherein the first attention-based neural network uses a set of original text-based and structure-based features to describe the query and the plurality of candidate result items,
wherein the query-to-result information generated by the first attention-based neural network is original query-to-result information,
wherein processing performed by said generating plural ranking scores comprises:
combining the plurality of instances of original query-to-result similarity information with supplemental feature information pertaining to a geographical context in which the query was made, to generate a plurality of respective instances enhanced similarity information;
generating, using a second attention-based neural network implemented by the computer-readable instructions, a plurality of instances of result-to-result context information using self-attention, each instance of result-to-result context information expressing an extent of attention that a particular instance of enhanced similarity information should pay to other instances of enhanced similarity information; and
mapping, using a scoring neural network implemented by the computer-readable instructions, the plurality of instances of result-to-result context information into the plural respective ranking scores.
