# US 9,519,682 B1 · User trustworthiness > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,682 B1 - Document ID: US9519682B1 - Status: Granted - Kind: B1 - Application number: 13/117,037 - Filed: 2011-05-26 - Granted: 2016-12-13 - Earliest claimed: 2011-05-26 - Assignees: Yahoo! Inc. - Applicants: Jay Pujara; Vishwanath Tumkur Ramarao; Xiaopeng Xi; Martin Zinkevich; Anirban Dasgupta; Belle Tseng; Wei Chu; Jyh-Shin Gareth Shue - Inventors: Jay Pujara; Vishwanath Tumkur Ramarao; Xiaopeng Xi; Martin Zinkevich; Anirban Dasgupta; Belle Tseng; Wei Chu; Jyh-Shin Gareth Shue - Web page: https://patentgrep.com/patent/US9519682B1 ## CPC - G06F 16/24578: {using ranking} - G06Q 10/10: Office automation; Time management - H04H 20/28: Arrangements for simultaneous broadcast of plural pieces of information - H04H 60/82: the transmission system being the Internet - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N) ## Abstract Embodiments are directed towards generating a unified user account trustworthiness system through user account trustworthiness scores. A trusted group of user accounts may be identified for a given action by grouping a plurality of user accounts into tiers based on a trustworthiness score of each user account for the given action. The tiers and/or trustworthiness scores may be employed to classify an item, such as a message as spam or non-spam, based on input from the user accounts. The trustworthiness scores may also be employed to determine if a user account is a robot account or a human account. The trusted group for a given action may dynamically evolve over time by regrouping the user accounts based on modified trustworthiness scores. A trustworthiness score of an individual user account may be modified based on input received from the individual user account and input from other user accounts. ## Claims ### Claim 1 A method comprising: identifying, for a given action by one or more network computing devices, a trusted group of user accounts from a plurality of user accounts using a respective trustworthiness score for the given action assigned to each user account of the plurality, wherein the given action refers to an online activity that is performed by one or more users associated with respective user accounts, each user account's trustworthiness score being determined using inputs received from the each user account for the given action, the trustworthiness score being used to identify the trusted group of user accounts whose input is to be used to classify an item, wherein the item refers to an article on the internet that can have an action performed on it, and to identify other user accounts whose input is to be excluded from classifying the item, the identifying comprising generating an initial trustworthiness score for a user account of the plurality using a trained trustworthiness classifier and a feature set about the user account, the trained trustworthiness classifier comprising a number of machine-implemented algorithms used to evolve behaviors based on input data, the feature set comprising online user behavioral features and static profile features about the user account, wherein the online user behavioral features comprise online activity features and the static profile features comprise user registration features, the initial trustworthiness score for the user account is used at least initially to determine whether or not to include the user account in the trusted group of user accounts for the given action; monitoring, by the one or more network computing devices, inputs for the given action and about the item, the inputs about the item from the trusted group for the given action are used to classify the item as one of spam and non-spam, such that any input about the item from other than the trusted group of user accounts formed for the given action is excluded from being used to classify the item; and evolving, by the one or more network computing devices, the trusted group based on modified trustworthiness scores of the plurality of user accounts for the given action, each modified trustworthiness score is determined, in part, by a comparison of an input about the item from a corresponding user account and inputs about the item from other user accounts in the plurality of user accounts, wherein evolving the trusted group further comprises at least one of moving at least one user account into the trusted group that previously was not in the trusted group and moving at least one user account out of the trusted group that was previously in the trusted group. ### Claim 2 (depends on 1) The method of claim 1, wherein the online activity features of the online user behavioral features comprising at least one of online browsing activity, electronic mail transactions and address book entries, and the user registration features of the static profile features comprising at least one of user preferences, age, gender and occupation. ### Claim 3 (depends on 1) The method of claim 1, wherein the given action is one of a plurality of actions, participation in more than one action of the plurality of actions results in a user account of the plurality of user accounts being assigned multiple trustworthiness scores, each trustworthiness score assigned to a given user account is associated with a different action of the plurality of actions, and wherein a different trusted group is identified for each action in the plurality of actions. ### Claim 4 (depends on 1) The method of claim 1, further comprising: modifying the trustworthiness score assigned to each user account, comprising: increasing a trustworthiness score of the given user account when input from a given user account matches the input from the other user accounts; and decreasing the trustworthiness score of the given user account when input from a given user account does not match the input from the other user accounts. ### Claim 5 (depends on 1) The method of claim 1, further comprising: modifying the trustworthiness score assigned to each user account, comprising: increasing a trustworthiness score of the given user account when input from a given user account matches the input from the other user accounts and the input from the given user account matches a result from an anti-spam system; and decreasing the trustworthiness score of the given user account when the input from the given user account is different than the input from other user accounts and the input from the given user account is different than the result from the anti-spam system. ### Claim 6 (depends on 1) The method of claim 1, further comprising: when a trustworthiness score of a given user account is below a threshold value, identifying the given user account as a robot account; and identifying the given user account as a human account when a trustworthiness score of the given user account is not below the threshold value. ### Claim 7 (depends on 1) The method of claim 1, further comprising: modifying a spam filter based on the inputs from the trusted group. ### Claim 8 A network device comprising: a transceiver for communicating with at least one client device over a network; a processor; and a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic comprising: identifying logic executed by the processor for identifying, for a given action, a trusted group of user accounts from a plurality of user accounts using a respective trustworthiness score for the given action assigned to each user account of the plurality, wherein the given action refers to an online activity that is performed by one or more users associated with respective user accounts, each user account's trustworthiness score being determined using inputs received from the each user account for the given action, the trustworthiness score being used to identify the trusted group of user accounts whose input is to be used to classify an item and to identify other user accounts whose input is to be excluded from classifying the item, wherein the item refers to an article of the internet that can have an action performed on it, the identifying comprising generating an initial trustworthiness score for a user account of the plurality using a trained trustworthiness classifier and a feature set about the user account, the trained trustworthiness classifier comprising a number of machine-implemented algorithms used to evolve behaviors based on input data, the feature set comprising online user behavioral features and static profile features about the user account, wherein the online user behavioral features comprise online activity features and the static profile features comprise user registration features, the initial trustworthiness score for the user account is used at least initially to determine whether or not to include the user account in the trusted group of user accounts for the given action; monitoring logic executed by the processor for monitoring inputs for the given action and about the item, the inputs about the item from the trusted group for the given action are used to classify the item as one of spam and non-spam, such that any input about the item from other than the trusted group of user accounts formed for the given action is excluded from being used to classify the item; and evolving logic executed by the processor for evolving the trusted group based on modified trustworthiness scores of the plurality of user accounts for the given action, each modified trustworthiness score is determined, in part, by a comparison of an input about the item from a corresponding user account and inputs about the item from other user accounts in the plurality of user accounts, wherein evolving the trusted group further comprises at least one of moving at least one user account into the trusted group that previously was not in the trusted group and moving at least one user account out of the trusted group that was previously in the trusted group. ### Claim 9 (depends on 8) The network device of claim 8, wherein the given action is one of a plurality of actions, participation in more than one action of the plurality of actions results in a user account of the plurality of user accounts being assigned multiple trustworthiness scores, each trustworthiness score assigned to a given user account is associated with a different action of the plurality of actions, and wherein a different trusted group is identified for each action in the plurality of actions. ### Claim 10 (depends on 8) The network device of claim 8, wherein the program logic for execution by the processor further comprises: modifying logic executed by the processor for modifying the trustworthiness score assigned to each user account, comprising: increasing logic executed by the processor for increasing a trustworthiness score of the given user account when input from a given user account matches the input from the other user accounts; and decreasing logic executed by the processor for decreasing the trustworthiness score of the given user account when the input from the given user account does not match the input from the other user accounts. ### Claim 11 (depends on 8) The network device of claim 8, wherein the program logic for execution by the processor further comprising: identifying logic executed by the processor for identifying the given user account as a robot account when a trustworthiness score of a given user account is below a threshold value, identifying the given user account as a robot account; and identifying logic executed by the processor for identifying the given user account as a human account when the trustworthiness score of the given user account is not below the threshold value. ### Claim 12 (depends on 8) The network device of claim 8, wherein the online activity features of the online user behavioral features comprising at least one of online browsing activity, electronic mail transactions and address book entries, and the user registration features of the static profile features comprising at least one of user preferences, age, gender and occupation. ### Claim 13 A non-transitory computer-readable storage device having computer-executable instructions stored thereon such that when the storage device is accessed by a computing device, the instructions are executable by the computing device to perform actions, comprising: identifying, for a given action, a trusted group of user accounts from a plurality of user accounts using a respective trustworthiness score for the given action assigned to each user account of the plurality, wherein the given action refers to an online activity that is performed by one or more users associated with respective user accounts, each user account's trustworthiness score being determined using inputs received from the each user account for the given action, the trustworthiness score being used to identify the trusted group of user accounts whose input is to be used to classify an item, wherein the item refers to an article of the internet that can have an action performed on it, and to identify other user accounts whose input is to be excluded from classifying the item the identifying comprising generating an initial trustworthiness score for a user account of the plurality using a trained trustworthiness classifier and a feature set about the user account, the trained trustworthiness classifier comprising a number of machine-implemented algorithms used to evolve behaviors based on input data, the feature set comprising online user behavioral features and static profile features about the user account, wherein the online user behavioral features comprise online activity features and the static profile features comprise user registration features the initial trustworthiness score for the user account is used at least initially to determine whether or not to include the user account in the trusted group of user accounts for the given action; monitoring inputs for the given action and about an item, the inputs about the item from the one or more trusted groups for the given action are used to classify the item as one of spam and non-spam, such that any input about the item from other than the trusted group of user accounts formed for the given action is excluded from being used to classify the item; and evolving the trusted group based on modified trustworthiness scores of the plurality of user accounts for the given action, each modified trustworthiness score is determined, in part, by a comparison of an input about the item from a corresponding user account and inputs about the item from other user accounts in the plurality of user accounts, wherein evolving the trusted group further comprises at least one of moving at least one user account into the trusted group that previously was not in the trusted group and moving at least one user account out of the trusted group that was previously in the trusted group. ### Claim 14 (depends on 13) The non-transitory computer-readable storage device of claim 13, wherein the online activity features of the online user behavioral features comprising at least one of online browsing activity, electronic mail transactions and address book entries, and the user registration features of the static profile features comprising at least one of user preferences, age, gender and occupation. ### Claim 15 (depends on 13) The non-transitory computer-readable storage device of claim 13, wherein the given action is one of a plurality of actions, participation in more than one action of the plurality of actions results in a user account of the plurality of user accounts being assigned multiple trustworthiness scores, each trustworthiness score assigned to a given user account is associated with a different action of the plurality of actions, and wherein a different trusted group is identified for each action in the plurality of actions. ### Claim 16 (depends on 13) The non-transitory computer-readable storage device of claim 13, wherein the one or more other network devices further enables actions, the actions comprising: modifying the trustworthiness score assigned to each user account, comprising: when input from a given user account matches the input from the other user accounts, increasing a trustworthiness score of the given user account; and decreasing the trustworthiness score of the given user account when input from the given user account does not match the input from the other user accounts. ### Claim 17 (depends on 13) The non-transitory computer-readable storage device of claim 13, wherein the one or more other network devices further enables actions, the actions comprising: when a trustworthiness score of a given user account is below a threshold value, identifying the given user account as a robot account; and identifying the given user account as a human account when the trustworthiness score of the given user account is not below the threshold value.