# US 10,769,962 B1 · Systems and methods for generating a personalization score for a constructed response > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,962 B1 - Document ID: US10769962B1 - Status: Granted - Kind: B1 - Application number: 15/145,459 - Filed: 2016-05-03 - Granted: 2020-09-08 - Earliest claimed: 2015-05-04 - Assignees: Educational Testing Service - Applicants: Educational Testing Service - Inventors: Beata Beigman Klebanov; Jill Burstein - Web page: https://patentgrep.com/patent/US10769962B1 ## CPC - G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student - G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26) - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract Computer-implemented systems and methods for generating a personalization score for a constructed response are provided. A constructed response generated by a user is received. The constructed response is processed with a processing system to generate one or more feature values representative of one or more aspects of the constructed response. A personalization score for the constructed response is generated using the processing system by applying a computer scoring model to the one or more feature values. The personalization score provides a measure of a personalization of the constructed response. ## Claims ### Claim 1 A computer-implemented method of generating a personalization score for a constructed response, the method comprising: receiving, by a processing system having one or more data processors and one or more computer-readable medium encoded with instructions for commanding the processing system, a constructed response generated by a user; processing the constructed response with the processing system to generate one or more feature values representative of one or more aspects of the constructed response, wherein processing the constructed response comprises: identifying, using the processing system, words of the constructed response; extracting, using the processing system, features from the words of the constructed response, wherein the extracted features are indicative of a level of utility value articulated in the constructed response and comprise a log frequency per 1000 words of first person pronouns and a total number of indefinite pronouns in the constructed response; and generating, using the processing system, the one or more feature values based on the extracted features; generating a personalization score for the constructed response using the processing system by applying a computer scoring model to the one or more feature values, the personalization score providing a measure indicative of the level of utility value articulated in the constructed response; and displaying the personalization score. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the one or more feature values are indicative of pronoun usage in the constructed response. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the extracted features comprise: a total number of first person singular pronouns in the constructed response; a total number of first person plural pronouns in the constructed response; a total number of second person pronouns in the constructed response; and a total number of possessive determiners in the constructed response. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the constructed response is on a topic, and wherein the one or more feature values are indicative of a degree to which the constructed response describes a relevance of the topic to the user or to one or more other persons. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the topic relates to science, technology, engineering, or mathematics. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of the topic. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein processing the constructed response comprises: identifying words of the constructed response that are characteristic of the topic based on a set of words previously determined to be characteristic of the topic, wherein the one or more feature values are based on a proportion of words in the constructed response that are characteristic of the topic. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the one or more feature values are indicative of vocabulary usage in the constructed response. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the one or more feature values include: an average word length of the constructed response; a median word frequency of the constructed response; a value indicative of a degree to which the constructed response utilizes academic words; a value indicative of a degree to which the constructed response utilizes concrete words; a count of a number of different words used in the constructed response. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the computer scoring model comprises multiple weighted variables determined by training the computer scoring model relative to a plurality of training data. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that is characteristic of argumentative writing, and (ii) indicative of vocabulary usage in the constructed response that is characteristic of hedging or boosting. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of narrativity. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, further comprising: using a part-of-speech tagger to associate words of the constructed response with part-of speech tags; processing the part-of-speech tags with the processing system to generate a value indicative of a degree to which the constructed response utilizes past-tense verbs, wherein the one or more feature values include the value, and wherein the one or more feature values further include a value indicative of a degree to which the constructed response utilizes common verbs. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that expresses positive or negative affect, (ii) indicative of vocabulary usage in the constructed response that expresses social relations or social interactions, (iii) indicative of vocabulary usage in the constructed response that expresses cognitive engagement, (iv) indicative of vocabulary usage in the constructed response that refers to health or disease, (v) indicative of vocabulary usage in the constructed response that refers to risk, and (vi) indicative of vocabulary usage in the constructed response that refers to ingestion. ### Claim 15 A computer-implemented system for generating a personalization score for a constructed response, the system comprising: a processing system; and computer-readable memory in communication with the processing system encoded with instructions for commanding the processing system to execute steps comprising: receiving, by the processing system having one or more data processors and one or more computer-readable medium encoded with instructions for commanding the processing system, a constructed response generated by a user; processing the constructed response with the processing system to generate one or more feature values representative of one or more aspects of the constructed response, wherein processing the constructed response comprises: identifying, using the processing system, words of the constructed response; extracting, using the processing system, features from the words of the constructed response, wherein the extracted features are indicative of a level of utility value articulated in the constructed response and comprise a total number of first person pronouns and a log frequency per 1000 words of indefinite pronouns in the constructed response; and generating, using the processing system, the one or more feature values based on the extracted features; generating a personalization score for the constructed response by applying a computer scoring model to the one or more feature values, the personalization score providing a measure indicative of the level of utility value articulated in the constructed response; and displaying the personalization score. ### Claim 16 (depends on 15) The computer-implemented system of claim 15, wherein the one or more feature values are indicative of pronoun usage in the constructed response. ### Claim 17 (depends on 15) The computer-implemented system of claim 15, wherein the constructed response is on a topic, and wherein the one or more feature values are indicative of a degree to which the constructed response describes a relevance of the topic to the user or to one or more other persons. ### Claim 18 (depends on 15) The computer-implemented system of claim 15, wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of a topic. ### Claim 19 (depends on 15) The computer-implemented system of claim 15, wherein the one or more feature values include an average word length of the constructed response, a median word frequency of the constructed response, a value indicative of a degree to which the constructed response utilizes academic words, a value indicative of a degree to which the constructed response utilizes concrete words, and a count of a number of different words used in the constructed response. ### Claim 20 (depends on 15) The computer-implemented system of claim 15, wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that is characteristic of argumentative writing, and (ii) indicative of vocabulary usage in the constructed response that is characteristic of hedging or boosting. ### Claim 21 (depends on 15) The computer-implemented system of claim 15, wherein the one or more feature values are indicative of vocabulary usage in the constructed response that is characteristic of narrativity. ### Claim 22 (depends on 15) The computer-implemented system of claim 15, wherein the one or more feature values are (i) indicative of vocabulary usage in the constructed response that expresses positive or negative affect, (ii) indicative of vocabulary usage in the constructed response that expresses social relations or social interactions, (iii) indicative of vocabulary usage in the constructed response that expresses cognitive engagement, (iv) indicative of vocabulary usage in the constructed response that refers to health or disease, (v) indicative of vocabulary usage in the constructed response that refers to risk, and (vi) indicative of vocabulary usage in the constructed response that refers to ingestion. ### Claim 23 A non-transitory computer-readable storage medium for generating a personalization score for a constructed response, the computer-readable storage medium comprising computer executable instructions which, when executed, cause a processing system to execute steps including: receiving, by a processing system having one or more data processors for commanding the processing system, a constructed response at a tokenizer generated by a user; processing the constructed response with the processing system to generate one or more feature values representative of one or more aspects of the constructed response, wherein processing the constructed response comprises: identifying, using the processing system, words of the constructed response; tokenizing, using the tokenizer, the words of the constructed response into individual words; extracting, using the processing system, one or more features from the words of the constructed response, wherein the extracted features are indicative of a level of utility value articulated in the constructed response and comprise a log frequency per 1000 words of first person pronouns and a log frequency per 1000 words of indefinite pronouns in the constructed response; and generating the one or more feature values based on the one or more extracted features; generating a personalization score for the constructed response by applying a computer scoring model to the one or more feature values, the personalization score providing a measure indicative of the level of utility value articulated in the constructed response; and displaying the personalization score. ### Claim 24 (depends on 1) The computer-implemented method of claim 1, wherein utility value is indicative of the user's belief that a subject of the constructed response is useful and relevant to other aspects of the user's life.