# US 2019/0172368 A1 · Secure computer-implemented execution and evaluation of programming assignments for on demand courses > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172368 A1 - Document ID: US20190172368A1 - Status: Published application - Kind: A1 - Application number: 16/270,098 - Filed: 2019-02-07 - Published: 2019-06-06 - Earliest claimed: 2015-10-07 - Assignees: Coursera, Inc. - Applicants: Coursera, Inc. - Inventors: Shuang CHEN; Brennan SAETA; Marius DRAGUS; Mantas MATELIS - Web page: https://patentgrep.com/patent/US20190172368A1 ## CPC - G09B 19/0053: {Computers, e.g. programming} - G06Q 10/103: {Workflow collaboration or project management} - G06Q 50/20: Education - G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip - G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A computer system includes a course repository and a server. The course repository includes a video of an online course, an instructor provided output for a programming assignment associated with the online course, and a submission script for generating student provided output for the programming assignment. The server includes a submission module that receives and verifies the student provided output for the programming assignment. The server further includes one or more built-in grader applications that compare the student provided output with the instructor provided output to determine a grade for the programming assignment. ## Claims ### Claim 1 A computer system comprising: a course repository including: a video of the online course; instructor provided output for a programming assignment associated with the online course; and a submission script for generating student provided output for the programming assignment; and a server including: a submission module configured to: receive the student provided output for the programming assignment generated by the submission script; and verify the student provided output for the programming assignment; and one or more built-in grader applications configured to: compare the student provided output for the programming assignment to the instructor provided output for the programming assignment; and generate a grade for the programming assignment based on the comparison. ### Claim 2 (depends on 1) The system of claim 1, wherein the server is configured to provide the submission script to a computing device for execution in a local application included in the computing device, and wherein the received student provided output for the programming assignment generated by the submission script is received from the computing device. ### Claim 3 (depends on 1) The system of claim 1, wherein verifying the student provided output for the programming assignment includes determining that contents of the student provided output for the programming assignment are in a proper format and include an expected type of content for evaluation by the one or more built-in grader applications. ### Claim 4 (depends on 1) The system of claim 1, wherein the one or more built-in grader applications include a numeric grader application and an expression grader application. ### Claim 5 (depends on 4) The system of claim 4, wherein the built-in grader application is a numeric grader application; and wherein comparing the student provided output for the programming assignment to the instructor provided output for the programming assignment includes comparing numbers included in the student provided output for the programming assignment with numbers included in the instructor provided output for the programming assignment. ### Claim 6 (depends on 5) The system of claim 5, wherein the built-in grader application is an expression grader application, and wherein comparing the student provided output for the programming assignment to the instructor provided output for the programming assignment includes comparing text strings included in the student provided output for the programming assignment with text strings included in the instructor provided output for the programming assignment. ### Claim 7 A computer system comprising: a processor; a memory; and a repository including an online course, the processor and memory configured to: receive a student's output for a programming assignment in the online course; receive code for a custom grader application for grading the student's output; create a base custom grader container including the code for the custom grader application; associate the base custom grader container with the online course included in the repository; evaluate the base custom grader container by executing the custom grader application in the base custom grader container; determine that the custom grader application violates a predetermined criterion; modify the base custom grader container to prevent the executing of the custom grader application from violating the predetermined criterion; and store, in the repository and in association with the online course, the modified base custom grader container including the custom grader application as a custom grader container image. ### Claim 8 (depends on 7) The computer system of claim 7, wherein the processor and memory are further configured to: receive the programming assignment in the online course for grading; access the custom grader container image associated with the online course; use the custom grader container image as a basis for a custom grader container for grading the programming assignment; input the programming assignment to the custom grader container; execute the programming assignment using the custom grader application included the custom grader container; and compare the student's output to an output of the execution of the programming assignment by the custom grader application to determine a grade. ### Claim 9 (depends on 7) The computer system of claim 7, wherein the predetermined criterion is one of a plurality of security criteria. ### Claim 10 (depends on 9) The computer system of claim 9, wherein the plurality of security criteria include a user privilege access level, and a specified user identification (ID). ### Claim 11 (depends on 7) The computer system of claim 7, wherein the predetermined criterion is one of a plurality of network access criteria. ### Claim 12 (depends on 7) The computer system of claim 7, wherein the predetermined criterion is a file system quota. ### Claim 13 (depends on 7) The computer system of claim 7, wherein the predetermined criterion is a file access permission. ### Claim 14 (depends on 7) The computer system of claim 7, wherein the predetermined criterion is a maximum duration for the executing of the custom grader application. ### Claim 15 A computer system including a processor and a memory having instructions stored thereon, the instructions, when executed by the processor, causing the computing system to: receive code for a custom grader application; create a base custom grader container including the code for the custom grader application; associate the base custom grader container with an online course included in a repository; evaluate the base custom grader container by executing the custom grader application in the base custom grader container; determine that the custom grader application violates a predetermined criterion; modify the base custom grader container to prevent the executing of the custom grader application from violating the predetermined criterion; and store, in the repository and in association with the online course, the modified base custom grader container including the custom grader application as a custom grader container image. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the instructions, when executed by the processor, further cause the computing system to: receive a programming assignment for the online course for grading; access the custom grader container image associated with the online course; use the custom grader container image as a basis for a custom grader container for grading the programming assignment; input the programming assignment to the custom grader container; execute the programming assignment by the custom grader application included the custom grader container; and determine a grade for the programming assignment based on an output of the execution of the programming assignment by the custom grader application. ### Claim 17 (depends on 15) The computer system of claim 15, wherein the predetermined criterion is one of a plurality of security criteria or one of a plurality of network access criteria. ### Claim 18 (depends on 15) The computer system of claim 15, wherein the predetermined criterion is a maximum duration for the executing of the custom grader application or a file access permission. ### Claim 19 (depends on 15) The computer system of claim 15, wherein the predetermined criterion is a security criteria including a user privilege access level. ### Claim 20 (depends on 15) The computer system of claim 15, wherein the predetermined criterion is a security criteria including a specified user identification (ID).