Back to results
US 2026/0228289 A1Published application
Human Evaluation of AI
OwnerMark Ograminventor
Life of this patent
Estimating the term…
Abstract
A system of computers having a central computer with a database of queries. A remote “proctor” computer repetitively withdraws these queries and presents them to a remote AI operating computer to obtain an AI response to the queries. These AI responses are sent by the “proctor” computer to several “polling” computers which use their human operators to gauge the accuracy of the AI responses. The human evaluations are returned to the central computer for evaluation/compilation as to the accuracy of the AI computer.
Claims17
A system comprising:
- a) a central computer having access to a memory having stored therein at least two queries;
- b) a remote proctor computer adapted to repetitively,
- 1) withdraw a selected query from the at least two queries from the memory via the central computer,
- 2) communicate the selected query to an AI operating computer,
- 3) receive an AI response to the selected query from the AI operating computer, and,
- 4) communicate the AI response to the central computer;
- and,
- c) a remote polling computer being operated by a human operator, said polling computer adapted to,
- 1) retrieve the AI response from the central computer,
- 2) present the AI response to the human operator of the remote polling computer,
- 3) receive a human evaluation of the AI response from the human operator of the remote polling computer, and,
- 4) communicate the human evaluation of the AI response to the central computer.
9 more elements
An AI rating system comprising:
- a) a proctor computer adapted to repetitively,
- 1) withdraw a selected query from a memory having at least two queries stored therein,
- 2) communicate the selected query to an AI operating computer, and,
- 3) receive an AI response to the selected query from the AI operating computer;
- b) at least two remote polling computers, each polling computer operated by a human operator, each of the polling computers adapted to,
- 1) retrieve a selected AI response from the proctor computer,
- 2) present the selected AI response to the human operator of the polling computer,
- 3) receive a human evaluation of the AI response from the human operator of the polling computer,
- 4) communicate the human evaluation to the proctor computer;
- and,
- c) wherein the proctor computer is adapted to,
- collectively summarize the human evaluation for all the at least two remoted polling stations to form.
9 more elements
A computer adapted to repetitively:
- a) withdraw a selected query from a memory having at least two queries therein;
- b) communicate the selected query to an AI operating computer;
- c) receive an AI response to the selected query from the AI operating computer;
- d) communicate the AI response to at least two polling computers;
- e) receive a human generated evaluation from each of the polling computer based on the AI response;
- f) compile a summary of human evaluations from the human generated evaluation from the at least two polling computers.
3 more elements
Dependencies are inferred from the claim text.