# US 2018/0322796 A1 · A/b testing for massive open online courses > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0322796 A1 - Document ID: US20180322796A1 - Status: Published application - Kind: A1 - Application number: 15/969,241 - Filed: 2018-05-02 - Published: 2018-11-08 - Earliest claimed: 2017-05-03 - Assignees: Coursera, Inc. - Applicants: Coursera, Inc. - Inventors: Amory B. SCHLENDER - Web page: https://patentgrep.com/patent/US20180322796A1 ## CPC - G09B 5/08: providing for individual presentation of information to a plurality of student stations - G06F 7/588: {Random number generators, i.e. based on natural stochastic processes} - G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems} - G09B 7/077: different stations being capable of presenting different questions simultaneously - G09B 7/00: Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) - G09B 19/025: {with electrically operated apparatus or devices} - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques of randomized testing of massively open online courses (MOOCs) involve generating independent A/B tests on the plurality of individual sections of a MOOC. Along these lines, a MOOC may have many learning modules, with many students enrolled in the MOOC. A course instructor may wish to experiment with different variations of course content in order to discover whether any such variations may improve the MOOC. Rather than perform a single A/B test during the MOOC to obtain results for which the course instructor would have to wait weeks, the instructor submits variations of various individual learning modules of the MOOC to a A/B testing server. The A/B testing server may then assign students in each lecture to different versions of a learning module. The A/B testing server may also evaluate the results of the testing in order to provide a recommendation about the MOOC as a whole. ## Claims ### Claim 1 A method, comprising: obtaining, by processing circuitry of a computer, massive open online course (MOOC) data and data describing a population of students enrolled in the MOOC, the MOOC data including data describing a plurality of learning modules of the MOOC, each of the plurality of learning modules including respective first course content and second course content; for each of the plurality of learning modules: assigning a first portion of the population of students to experience that learning module with the respective first course content and a second portion of the population of students to experience that learning module with the respective second course content; and performing a short-timescale evaluation operation on that learning module based on a specified metric applied to the first portion of the population of students and the second portion of the population of students to produce evaluation results for that learning module. ### Claim 2 (depends on 1) The method as in claim 1, wherein assigning the first portion of the population of students includes performing a random number generating operation to produce random numbers identifying the first portion of the population of students. ### Claim 3 (depends on 2) The method as in claim 2, further comprising designating each student of the population of students as belonging to one of a plurality of specified groups; wherein assigning the first portion of the population of students further includes for each of the plurality of groups, performing the random number generating operation for that group; wherein the first portion of the population of students is identified by an aggregation of random numbers produced by the random number generation operation performed for each of the plurality of groups. ### Claim 4 (depends on 1) The method as in claim 1, wherein each of the plurality of learning modules includes a lecture provided over a single period of time; and wherein the respective first course content and the respective second course content of each of the plurality of learning modules includes video content. ### Claim 5 (depends on 4) The method as in claim 4, wherein the respective second course content of each of the plurality of learning modules has additional content not shown in the respective first course content of that learning module. ### Claim 6 (depends on 4) The method as in claim 4, wherein performing the short-timescale evaluation operation on each of the plurality of learning modules includes tracking a first number of students of the first portion of the population of students that are present in a subsequent lecture and a second number of students of the second portion of the population of students that are present in the subsequent lecture. ### Claim 7 (depends on 1) The method as in claim 1, further comprising performing a long-timescale evaluation operation on the MOOC based on the evaluation results for each of the learning modules of the MOOC to produce an evaluation result for the MOOC as a whole. ### Claim 8 (depends on 7) The method as in claim 7, wherein performing the long-timescale evaluation operation on the MOOC includes verifying whether each student of the population of students completed the MOOC. ### Claim 9 A computer program product comprising a nontransitive storage medium, the computer program product including code that, when executed by processing circuitry of a computer, causes the processing circuitry to perform a method, the method comprising: massive open online course (MOOC) data and data describing a population of students enrolled in the MOOC, the MOOC data including data describing a plurality of learning modules of the MOOC, each of the plurality of learning modules including respective first course content and second course content; for each of the plurality of learning modules: assigning a first portion of the population of students to experience that learning module with the respective first course content and a second portion of the population of students to experience that learning module with the respective second course content; and performing a short-timescale evaluation operation on that learning module based on a specified metric applied to the first portion of the population of students and the second portion of the population of students to produce evaluation results for that learning module. ### Claim 10 (depends on 9) The computer program product as in claim 9, wherein assigning the first portion of the population of students includes performing a random number generating operation to produce random numbers identifying the first portion of the population of students. ### Claim 11 (depends on 10) The computer program product as in claim 10, wherein the method further comprises designating each student of the population of students as belonging to one of a plurality of specified groups; wherein assigning the first portion of the population of students further includes for each of the plurality of groups, performing the random number generating operation for that group; wherein the first portion of the population of students is identified by an aggregation of random numbers produced by the random number generation operation performed for each of the plurality of groups. ### Claim 12 (depends on 9) The computer program product as in claim 9, wherein each of the plurality of learning modules includes a lecture provided over a single period of time; and wherein the respective first course content and the respective second course content of each of the plurality of learning modules includes video content. ### Claim 13 (depends on 12) The computer program product as in claim 12, wherein the respective second course content of each of the plurality of learning modules has additional content not shown in the respective first course content of that learning module. ### Claim 14 (depends on 12) The computer program product as in claim 12, wherein performing the short-timescale evaluation operation on each of the plurality of learning modules includes tracking a first number of students of the first portion of the population of students that are present in a subsequent lecture and a second number of students of the second portion of the population of students that are present in the subsequent lecture. ### Claim 15 (depends on 9) The method as in claim 9, wherein the method further comprises performing a long-timescale evaluation operation on the MOOC based on the evaluation results for each of the learning modules of the MOOC to produce an evaluation result for the MOOC as a whole. ### Claim 16 (depends on 15) The method as in claim 15, wherein performing the long-timescale evaluation operation on the MOOC includes verifying whether each student of the population of students completed the MOOC. ### Claim 17 An electronic apparatus, comprising: memory; and controlling circuitry coupled to the memory, the controlling circuitry being configured to: obtain massive open online course (MOOC) data and data describing a population of students enrolled in the MOOC, the MOOC data including data describing a plurality of learning modules of the MOOC, each of the plurality of learning modules including respective first course content and second course content; for each of the plurality of learning modules: assign a first portion of the population of students to experience that learning module with the respective first course content and a second portion of the population of students to experience that learning module with the respective second course content; and perform a short-timescale evaluation operation on that learning module based on a specified metric applied to the first portion of the population of students and the second portion of the population of students to produce evaluation results for that learning module. ### Claim 18 (depends on 17) The electronic apparatus as in claim 17, wherein the controlling circuitry configured to assign the first portion of the population of students is further configured to perform a random number generating operation to produce random numbers identifying the first portion of the population of students. ### Claim 19 (depends on 17) The electronic apparatus as in claim 17, wherein each of the plurality of learning modules includes a lecture provided over a single period of time; and wherein the respective first course content and the respective second course content of each of the plurality of learning modules includes video content. ### Claim 20 (depends on 17) The electronic apparatus as in claim 17, wherein the controlling circuitry is further configured to perform a long-timescale evaluation operation on the MOOC based on the evaluation results for each of the learning modules of the MOOC to produce an evaluation result for the MOOC as a whole.