# US 2025/0094923 A1 · Systems and methods for obtaining data annotations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094923 A1
- Document ID: US20250094923A1
- Status: Published application
- Kind: A1
- Application number: 18/794,923
- Filed: 2024-08-05
- Published: 2025-03-20
- Earliest claimed: 2019-05-17
- Assignees: Allstate Insurance Company
- Applicants: Allstate Insurance Company
- Inventors: Glenn Shapiro
- Web page: https://patentgrep.com/patent/US20250094923A1

## CPC

- G06Q 10/10: Office automation; Time management
- G06F 16/55: Clustering; Classification
- G06Q 10/06311: {Scheduling, planning or task assignment for a person or group}
- G06Q 20/085: {involving remote charge determination or related payment systems}
- G06Q 30/0185: {Product, service or business identity fraud}
- G06Q 30/0278: {Product appraisal}
- G06Q 30/0282: Rating or review of business operators or products
- G06Q 40/08: Insurance
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods are provided for obtaining data annotations from a crowdsourced group of individuals. The individuals can be provided with a set of data describing damage to an item and a variety of annotations can be applied to the data. In a variety of embodiments, multiple individuals can review the same claim and a final claim outcome can be determined based on the multiple reviews. In many embodiments, machine classifiers can process the set of data to identify particular features within the data. Scoring data can be generated, based on annotations provided by other individuals and/or machine classifiers that reflects the adjuster's skill at identifying features within the data and annotating the data. Claims can be assigned to individuals based on the score assigned to the individual.

## Claims

### Claim 1

A method for providing data processing from a crowdsourced group, comprising:
communicating, by a data processing server system and to a particular adjuster device, job request data that comprises anonymized item data associated with one or more item features;
receiving, by the data processing server system and from the particular adjuster device, first annotation data associated with a first set of item features associated with the anonymized item data;
generating, by a machine learning model, second annotation data associated with a second set of features associated with the anonymized item data;
determining, based on a comparison of the first set of item features received from the particular adjuster device and the second set of item features generated by the machine learning model, one or more performance metrics associated with the particular adjuster device; and
updating, by the data processing server system, an adjuster score associated with the particular adjuster device based on the one or more performance metrics.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
communicating, by the data processing server system and to the particular adjuster device, the updated adjuster score.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
routing, by the data processing server system, second job request data to the particular adjuster device based at least on the updated adjuster score.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
obtaining, by the data processing server system and from a third-party server system, certification data that specifies one or more certifications associated with respective adjuster devices of a plurality of adjuster devices; and
determining, by the data processing server system and based on the certification data, the particular adjuster device to correspond to an adjuster device of the plurality of adjuster devices associated with one or more particular certifications.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
determining the one or more particular certifications based on an item described in the anonymized item data.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
generating, by the data processing server system, feedback data based on the one or more performance metrics; and
communicating, by the data processing server system, the feedback data to the particular adjuster device.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the one or more performance metrics associated with the particular adjuster device comprises at least one of:
determining a completeness of the first annotation data based on comparing a number of features identified in the first annotation data to a number of features identified in the second annotation data; and
determining an accuracy of the first annotation data based on comparing the set of features described in the first annotation data to features described in the second annotation data.

### Claim 8

A computing device comprising:
one or more processors; and
one or more storage devices in communicating with the one or more processors that store instruction code executable by the one or more processors to cause the computing device to:
communicate, to a particular adjuster device, job request data that comprises anonymized item data associated with one or more item features;
receive, from the particular adjuster device, first annotation data associated with a first set of item features associated with the anonymized item data;
generate, by a machine learning model, second annotation data associated with a second set of features associated with the anonymized item data;
determine, based on a comparison of the first set of item features received from the particular adjuster device and the second set of item features generated by the machine learning model, one or more performance metrics associated with the particular adjuster device; and
update an adjuster score associated with the particular adjuster device based on the one or more performance metrics.

### Claim 9 (depends on 8)

The computing device of claim 8, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
communicate, to the particular adjuster device, the updated adjuster score.

### Claim 10 (depends on 9)

The computing device of claim 9, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
route second job request data to the adjuster device based at least on the updated adjuster score.

### Claim 11 (depends on 8)

The computing device of claim 8, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
obtain, from a third-party server system, certification data that specifies one or more certifications associated with respective adjuster devices of a plurality of adjuster devices; and
determine, based on the certification data, the particular adjuster device to correspond to an adjuster device of the plurality of adjuster devices associated with one or more particular certifications.

### Claim 12 (depends on 11)

The computing device of claim 11, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
determine the one or more particular certifications based on an item described in the anonymized item data.

### Claim 13 (depends on 8)

The computing device of claim 8, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
generate feedback data based on the one or more performance metrics; and
communicate the feedback data to the particular adjuster device.

### Claim 14 (depends on 8)

The computing device of claim 8, wherein the instruction code that causes the computing device to determine the one or more performance metrics associated with the particular adjuster device comprises is executable by the one or more processors to cause the computing device to:
determine a completeness of the first annotation data based on comparing a number of features identified in the first annotation data to a number of features identified in the second annotation data; or
determine an accuracy of the first annotation data based on comparing the set of features described in the first annotation data to features described in the second annotation data.

### Claim 15

A non-transitory computer readable medium having stored thereon instruction code that, when executed by one or more processors of a computing device, causes the computing device to:
communicate to a particular adjuster device, job request data that comprises anonymized item data associated with one or more item features;
receive, from the particular adjuster device, first annotation data associated with a first set of item features associated with the anonymized item data;
generate, by a machine learning model, second annotation data associated with a second set of features associated with the anonymized item data;
determine, based on a comparison of the first set of item features received from the particular adjuster device and the second set of item features generated by the machine learning model, one or more performance metrics associated with the particular adjuster device; and
update an adjuster score associated with the particular adjuster device based on the one or more performance metrics.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
communicate, to the particular adjuster device, the updated adjuster score.

### Claim 17 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
route second job request data to the particular adjuster device based at least on the updated adjuster score.

### Claim 18 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
obtain, from a third-party server system, certification data that specifies one or more certifications associated with respective adjuster devices of a plurality of adjuster devices; and
determine, based on the certification data, the particular adjuster device to correspond to an adjuster device of the plurality of adjuster devices associated with one or more particular certifications.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
determine the one or more particular certifications based on an item described in the anonymized item data.

### Claim 20 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the instruction code is further executable by the one or more processors to cause the computing device to:
generate feedback data based on the one or more performance metrics; and
communicate the feedback data to the particular adjuster device.
