# US 11,030,209 B2 · Unsupervised learning to fuse for information retrieval

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,030,209 B2
- Document ID: US11030209B2
- Status: Granted
- Kind: B2
- Application number: 16/234,824
- Filed: 2018-12-28
- Published: 2020-07-02
- Granted: 2021-06-08
- Earliest claimed: 2018-12-28
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Haggai Roitman; Bar Weiner; Shai Erera
- Web page: https://patentgrep.com/patent/US11030209B2

## CPC

- G06F 16/248: Presentation of query results
- G06F 16/3346: {using probabilistic model}
- G06F 16/24578: {using ranking}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for generating and evaluating fused query lists. A query on a corpus of documents is evaluated using a plurality of retrieval methods and a ranked list for each of the plurality of retrieval methods is obtained. A plurality of fused ranked lists is sampled, each fusing said ranked lists for said plurality of retrieval methods, and the sampled fused ranked lists are sorted. In an unsupervised manner, an objective comprising a likelihood that a fused ranked list, fusing said ranked lists for each of said plurality of retrieval methods, is relevant to a query and a relevance event, is optimized to optimize the sampling, until convergence is achieved. Documents of the fused ranked list are determined based on the optimization.

## Claims

### Claim 1

A method comprising:
evaluating, using a hardware processor exercising a plurality of rankers, a query on a corpus of electronic documents stored in a document database using a plurality of retrieval methods corresponding to the plurality of rankers;
obtaining, using the hardware processor, a ranked list for each of the plurality of retrieval methods;
generating, using the hardware processor, a plurality of fused ranked lists;
sampling, using the hardware processor, the plurality of fused ranked lists, each fused ranked list fusing said ranked lists for said plurality of retrieval methods;
sorting, using the hardware processor, said sampled fused ranked lists;
in an unsupervised manner, optimizing, using the hardware processor, an objective comprising a likelihood that a fused ranked list, fusing said ranked lists for each of said plurality of retrieval methods, is relevant to a query and a relevance event, to optimize said sampling, until convergence is achieved;
determining, using the hardware processor, documents of the fused ranked list based on the optimization; and
generating a user interface for displaying an identification of the documents of the fused ranked list.

### Claim 2 (depends on 1)

The method of claim 1, where each fused rank list is generated based on the relationship:
p
⁡
(
d
|
L
F
)
⁢
=
def
⁢
p
⁡
(
d
|
q
,
r
)
∑
d
∈
L
F
⁢
p
⁡
(
d
|
q
,
r
)
;
and where:
q denotes the query;
C denotes the corpus;
d denotes a given document of the corpus of documents;
r denotes a relevance event;
LF denotes a given one of the fused ranked lists;
p(d|q, r) denotes a likelihood that document d is relevant to query q; and where each of the fused ranked lists includes k documents having a highest value of p(d|LF).

### Claim 3 (depends on 2)

The method of claim 2, wherein the optimizing comprises:
computing a relevance model for each of the ranked lists, each of the rankled lists being designated as Li;
computing a list entropy for each of the ranked lists Li;
calculating an information gain based on the computed list entropy for each document d that is an element of the given fused list LF;
calculating an estimated quality;
determining a list similarity between the fused list LF and the ranked list Li; and
estimating the fused list quality based on the calculated information gain, the estimated quality, and the list similarity in response to sampling the ranked list Li.

### Claim 4 (depends on 3)

The method of claim 3, further comprising revising the estimated fused list quality by processing a next ranked list Li.

### Claim 5 (depends on 3)

The method of claim 3, further comprising revising the estimated fused list quality in response to an addition of a supplemental retrieval method to the plurality of retrieval methods.

### Claim 6 (depends on 1)

The method of claim 1, further comprising re-optimizing the objective in response to an addition of a supplemental retrieval method to the plurality of retrieval methods.

### Claim 7

An apparatus comprising:
a memory; and
at least one processor, coupled to said memory, and operative to perform operations comprising:
evaluating, using a plurality of rankers, a query on a corpus of documents using a plurality of retrieval methods corresponding to the plurality of rankers;
obtaining a ranked list for each of the plurality of retrieval methods;
generating a plurality of fused ranked lists;
sampling the plurality of fused ranked lists, each fused ranked list fusing said ranked lists for said plurality of retrieval methods;
sorting said sampled fused ranked lists;
in an unsupervised manner, optimizing an objective comprising a likelihood that a fused ranked list, fusing said ranked lists for each of said plurality of retrieval methods, is relevant to a query and a relevance event, to optimize said sampling, until convergence is achieved;
determining documents of the fused ranked list based on the optimization; and
generating a user interface for displaying an identification of the documents of the fused ranked list.

### Claim 8 (depends on 7)

The apparatus of claim 7, where each fused rank list is generated based on the relationship:
p
⁡
(
d
|
L
F
)
⁢
=
def
⁢
p
⁡
(
d
|
q
,
r
)
∑
d
∈
L
F
⁢
p
⁡
(
d
|
q
,
r
)
;
and where:
q denotes the query;
C denotes the corpus;
d denotes a given document of the corpus of documents;
r denotes a relevance event;
LF denotes a given one of the fused ranked lists;
p(d|q, r) denotes a likelihood that document d is relevant to query q; and where each of the fused ranked lists includes k documents having a highest value of p(d|LF).

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the optimizing comprises:
computing a relevance model for each of the ranked lists, each of the rankled lists being designated as Li;
computing a list entropy for each of the ranked lists Li;
calculating an information gain based on the computed list entropy for each document d that is an element of the given fused list LF;
calculating an estimated quality;
determining a list similarity between the fused list LF and the ranked list Li; and
estimating the fused list quality based on the calculated information gain, the estimated quality, and the list similarity in response to sampling the ranked list Li.

### Claim 10 (depends on 9)

The apparatus of claim 9, the operations further comprising revising the estimated fused list quality by processing a next ranked list Li.

### Claim 11 (depends on 9)

The apparatus of claim 9, the operations further comprising revising the estimated fused list quality in response to an addition of a supplemental retrieval method to the plurality of retrieval methods.

### Claim 12 (depends on 7)

The apparatus of claim 7, the operations further comprising re-optimizing the objective in response to an addition of a supplemental retrieval method to the plurality of retrieval methods.

### Claim 13

A non-transitory computer readable medium comprising computer executable instructions which when executed by a computer cause the computer to perform operations comprising:
evaluating, using a plurality of rankers, a query on a corpus of documents using a plurality of retrieval methods corresponding to the plurality of rankers;
obtaining a ranked list for each of the plurality of retrieval methods;
generating a plurality of fused ranked lists;
sampling the plurality of fused ranked lists, each fused ranked list fusing said ranked lists for said plurality of retrieval methods;
sorting said sampled fused ranked lists;
in an unsupervised manner, optimizing an objective comprising a likelihood that a fused ranked list, fusing said ranked lists for each of said plurality of retrieval methods, is relevant to a query and a relevance event, to optimize said sampling, until convergence is achieved;
determining documents of the fused ranked list based on the optimization; and
generating a user interface for displaying an identification of the documents of the fused ranked list.

### Claim 14 (depends on 13)

The non-transitory computer readable medium of claim 13, where each fused rank list is generated based on the relationship:
p
⁡
(
d
|
L
F
)
⁢
=
def
⁢
p
⁡
(
d
|
q
,
r
)
∑
d
∈
L
F
⁢
p
⁡
(
d
|
q
,
r
)
;
and where:
q denotes the query;
C denotes the corpus;
d denotes a given document of the corpus of documents;
r denotes a relevance event;
LF denotes a given one of the fused ranked lists;
p(d|q, r) denotes a likelihood that document d is relevant to query q; and where each of the fused ranked lists includes k documents having a highest value of p(d|LF).

### Claim 15 (depends on 14)

The non-transitory computer readable medium of claim 14, wherein the optimizing comprises:
computing a relevance model for each of the ranked lists, each of the rankled lists being designated as Li;
computing a list entropy for each of the ranked lists Li;
calculating an information gain based on the computed list entropy for each document d that is an element of the given fused list LF;
calculating an estimated quality;
determining a list similarity between the fused list LF and the ranked list Li; and
estimating the fused list quality based on the calculated information gain, the estimated quality, and the list similarity in response to sampling the ranked list Li.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, the operations further comprising revising the estimated fused list quality by processing a next ranked list Li.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, the operations further comprising revising the estimated fused list quality in response to an addition of a supplemental retrieval method to the plurality of retrieval methods.

### Claim 18 (depends on 13)

The non-transitory computer readable medium of claim 13, the operations further comprising re-optimizing the objective in response to an addition of a supplemental retrieval method to the plurality of retrieval methods.