# US 2019/0361791 A1 · Computer programming assessment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0361791 A1
- Document ID: US20190361791A1
- Status: Published application
- Kind: A1
- Application number: 16/511,005
- Filed: 2019-07-15
- Published: 2019-11-28
- Earliest claimed: 2016-01-28
- Assignees: BrainFights, Inc.
- Applicants: Brainfights, Inc.
- Inventors: TIGRAN SLOYAN; ALBERT SAHAKYAN
- Web page: https://patentgrep.com/patent/US20190361791A1

## CPC

- G06F 8/77: Software metrics
- G06F 11/3612: {by runtime analysis (performance monitoring G06F11/3466)}
- G06F 8/41: Compilation
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 11/3636: {by tracing the execution of the program}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3692: {for test results analysis}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques are described for generating a programming score for a user based on source code submissions for a programming task set. In an embodiment, a programming system generates, at random, a programming task set for submission by a user. The programming system receives source codes submitted for the programming task set in a programming session with a client computer system of a user. The programming system executes the source codes to evaluate their accuracy. Based on the execution, the programming system determines that some source codes are evaluated to be accurate thereby solving corresponding programming tasks from the programming task set. Based on such an execution, programming system determines the programming score for the particular user for the programming task set.

## Claims

### Claim 1

A computer-implemented method comprising:
selecting a programming task set with a plurality of programming tasks at random, wherein each programming task of the programming task set corresponds to a task category of a plurality of task categories;
receiving a plurality of source codes submitted for the programming task set in one or more programming sessions with a client computer system of a particular user;
executing the plurality of source codes to evaluate accuracy of the plurality of source codes;
based on the executing, determining that one or more source codes of the plurality of source codes are evaluated to be accurate, wherein the one or more source codes are submissions for one or more solved programming tasks of the programming task set;
based on the determining that one or more source codes of the plurality of source codes are evaluated to be accurate, determining a programming score for the particular user for the programming task set.

### Claim 2 (depends on 1)

The method of claim 1, wherein selecting the programming task set with the plurality of programming tasks at random comprises selecting at random, from multiple programming tasks associated with a corresponding task category of the plurality of task categories, a random programming task for each programming task of the plurality of programming tasks in the programming task set.

### Claim 3 (depends on 1)

The method of claim 1, wherein selecting the programming task set with the plurality of programming tasks at random comprises selecting at random, from a plurality of pre-generated programming task sets, the programming task set, wherein each pre-generated programming task in the plurality of pre-generated programming task sets corresponds to a respective task category of the plurality of task categories.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
based on the executing:
determining that at least one source code of the plurality of source codes is evaluated to be inaccurate, wherein the at least one source code corresponds to an unsolved programming task from the programming task set, and
determining a task score fraction for the unsolved programming task from the programming task set;
determining a partial task score of the unsolved programming task based on complexity metrics of each programming task in the programming task set and a maximum score assigned to the programming task set;
based on the task score fraction and the partial task score of the unsolved programming task, determining the programming score for the particular user for the programming task set.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining a partial task score of a solved programming task of the one or more solved programming tasks in the programming task set, based on complexity metrics of each programming task in the programming task set and a pre-defined maximum programming score that can be assigned to a user for the programming task set;
based on the partial task score of the solved programming task, determining the programming score for the particular user for the programming task set.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
based on complexity metrics, determining one or more task scores for the one or more solved programming tasks of the programming task set;
based on the one or more task scores for the one or more solved programming tasks of the programming task set, determining a most difficult solved programming task from the programming task set having a maximal task score;
based on the maximal task score of the most difficult solved programming task, determining the programming score for the particular user for the programming task set.

### Claim 7 (depends on 1)

The method of claim 1, the method further comprising:
based on complexity metrics of the one or more solved programming tasks, determining one or more task scores for the one or more solved programming tasks of the programming task set;
based on the one or more task scores for the one or more solved programming tasks of the programming task set, determining a most difficult solved programming task from the programming task set having a maximal task score;
determining a partial task score of a solved programming task for at least one solved programming tasks in the programming task set that is not the most difficult solved programming task, based on complexity metrics of each programming task in the programming task set and a pre-defined maximum programming score that can be assigned to a user for the programming task set;
based on the partial task score of the solved programming task and the maximal task score for the most difficult solved programming task in the programming task set, determining the programming score for the particular user for the programming task set.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining one or more pre-compile metrics for the programming task set;
based on the one or more pre-compile metrics, calculating variation programming score;
adjusting the programming score for the particular user for the programming task set by the variation programming score.

### Claim 9 (depends on 8)

The method of claim 8, wherein the one or more pre-compile metrics include one or more of: programming duration, an editorial metric and a submission confidence metric

### Claim 10 (depends on 8)

The method of claim 8, wherein the variation programming score is based on a fraction of a maximum variation score for the programming task set.

### Claim 11 (depends on 1)

The method of claim 1, wherein a scoring paradigm determines a pre-defined maximum programming score that can be assigned to a user for the programming task set.

### Claim 12 (depends on 1)

The method of claim 1, wherein a first programming task of the one or more solved programming tasks in the programming task set is associated with a first task category, which is different than a second task category, with which a second programming task of the one or more solved programming tasks in the programming task set is associated.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first task category has a first task score range different from a second task score range of the second task category.

### Claim 14

A computer-implemented method comprising:
receiving a first plurality of source codes submitted for a corresponding first plurality of programming tasks of a first programming task set in one or more first programming sessions with a first client computer system of a first user;
wherein each first programming task in the corresponding first plurality of programming tasks corresponds to each task category of a plurality of task categories;
executing the first plurality of source codes to evaluate the accuracy of the first plurality of source codes;
based on the executing the first plurality of source codes, determining a first programming score for the first user for the first programming task set;
receiving a second plurality of source codes submitted for a corresponding second plurality of programming tasks of a second programming task set in one or more second programming sessions with a second client computer system of the first user;
wherein each second programming task in the corresponding second plurality of programming tasks corresponds to said each task category of the plurality of task categories but is different than corresponding said each first programming task in the corresponding first plurality of programming tasks;
executing the second plurality of source codes to evaluate the accuracy of the second plurality of source codes;
based on the executing the second plurality of source codes, determining a second programming score for the first user for the second programming task set;
wherein the second programming score for the first user for the second programming task set and the first programming score for the first user for the second programming task set correspond to a same user category of a plurality of user categories.

### Claim 15 (depends on 14)

The method of claim 14, wherein each of the plurality of user categories corresponds to each of the plurality of task categories.

### Claim 16 (depends on 14)

The method of claim 14, wherein each user category of the plurality of user categories differ from each other based on a programming score range assigned to said each user category.

### Claim 17 (depends on 14)

The method of claim 14, wherein each task category of the plurality of task categories differ from each other based on a task score range assigned to said each task category.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
receiving a third plurality of source codes submitted for a corresponding third plurality of programming tasks of a third programming task set in one or more third programming sessions with a third client computer system of a second user different from the first user;
wherein each third programming task in the corresponding third plurality of programming tasks corresponds to said each task category of the plurality of task categories but is different than corresponding said each first programming task in the corresponding first plurality of programming tasks and said each second programming task in the corresponding second plurality of programming tasks;
executing the third plurality of source codes to evaluate the accuracy of the third plurality of source codes;
based on the executing the third plurality of source codes, determining a third programming score for the second user for the second programming task set;
wherein the third programming score for the second user for the third programming task set corresponds to another user category of the plurality of user categories.

### Claim 19

A server computer system comprising:
one or more processors;
one or more storage media storing one or more computer programs for execution by the one or more processors, the one or more computer programs comprising instructions for:
receiving a first plurality of source codes submitted for a corresponding first plurality of programming tasks of a first programming task set in one or more first programming sessions with a first client computer system of a first user;
wherein each first programming task in the corresponding first plurality of programming tasks corresponds to each task category of a plurality of task categories;
executing the first plurality of source codes to evaluate the accuracy of the first plurality of source codes;
based on the executing the first plurality of source codes, determining a first programming score for the first user for the first programming task set;
receiving a second plurality of source codes submitted for a corresponding second plurality of programming tasks of a second programming task set in one or more second programming sessions with a second client computer system of the first user;
wherein each second programming task in the corresponding second plurality of programming tasks corresponds to said each task category of the plurality of task categories but is different than corresponding said each first programming task in the corresponding first plurality of programming tasks;
executing the second plurality of source codes to evaluate the accuracy of the second plurality of source codes;
based on the executing the second plurality of source codes, determining a second programming score for the first user for the second programming task set;
wherein the second programming score for the first user for the second programming task set and the first programming score for the first user for the second programming task set correspond to a same user category of a plurality of user categories.

### Claim 20 (depends on 19)

The system of claim 19, wherein each of the plurality of user categories corresponds to each of the plurality of task categories.
