# US 2022/0284884 A1 · Offensive chat filtering using machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0284884 A1 - Document ID: US20220284884A1 - Status: Published application - Kind: A1 - Application number: 17/190,530 - Filed: 2021-03-03 - Published: 2022-09-08 - Earliest claimed: 2021-03-03 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Monica TONGYA - Web page: https://patentgrep.com/patent/US20220284884A1 ## CPC - H04L 51/046: Interoperability with other network applications or services - H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)} - A63F 13/35: Details of game servers - A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use - A63F 13/75: Enforcing rules, e.g. detecting foul play or generating lists of cheating players - A63F 13/87: Communicating with other players during game play, e.g. by e-mail or chat - G06F 40/216: using statistical methods - G06F 40/242: Dictionaries - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/30: Semantic analysis - G06F 40/35: Discourse or dialogue representation - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/091: Active learning - G06N 20/20: Ensemble learning - H04L 51/212: using filtering or selective blocking - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A server system is provided that includes one or more processors configured to execute a platform for an online multi-user chat service that communicates with a plurality of client devices of users of the online multi-user chat service that exchanges user chat data between the plurality of client devices. The one or more processors are configured to execute a user chat filtering program that performs filter actions for user chat data exchanged on the platform for the online multi-user chat service. The user chat filtering program includes a plurality of trained machine learning models and a filter decision service that determines a filter action to be performed for target portions of user chat data based on output of the plurality of trained machine learning models for those target portions of user chat data. ## Claims ### Claim 1 A server system comprising: one or more processors configured to: execute a platform for an online multi-user chat service that communicates with a plurality of client devices of users of the online multi-user chat service, the platform being configured to exchange user chat data between the plurality of client devices; execute a user chat filtering program that performs filter actions for user chat data exchanged on the platform for the online multi-user chat service, the user chat filtering program including a plurality of trained machine learning models and a filter decision service that determines a filter action to be performed for target portions of user chat data based on output of the plurality of trained machine learning models for those target portions of user chat data; wherein the user chat filtering program is configured to: receive user chat data that includes a first user chat data and a second user chat data; process the first user chat data and the second user chat data using the plurality of trained machine learning models; determine that the first user chat data that includes a target word or phrase is offensive language based on output of the plurality of trained machine learning models for the first user chat data; determine that the second user chat data that includes the same target word or phrase is not offensive language based on output of the plurality of trained machine learning models for the second user chat data; and determine a filter action to be performed for the first user chat data. ### Claim 2 (depends on 1) The server system of claim 1, wherein the plurality of trained machine learning models includes at least one trained transformer machine learning model that includes a transformer encoder and a plurality of fully connected deep learning layers, wherein the transformer encoder is configured to receive an output encoding from a bidirectional encoder representations from transformers (BERT) model that processed the target portion of user chat data from the platform, and wherein the fully connected deep learning layers are configured to output a predicted label for a type of offensive language for the target portion of user chat data. ### Claim 3 (depends on 2) The server system of claim 2, wherein the at least one trained transformer machine learning model is one of a plurality of trained transformer machine learning models; and each of the plurality of trained transformer machine learning models is respectively trained to predict labels for a different respective type of offensive language. ### Claim 4 (depends on 3) The server system of claim 3, wherein the different respective types of offensive language are selected from the group consisting of a profanity type, a bullying type, an adult language type, and a hate speech type; and wherein the hate speech type includes at least one of racist hate speech type and/or sexist hate speech type. ### Claim 5 (depends on 2) The server system of claim 2, wherein the user chat filtering program further includes: a speech to text model that generates the target portion of the user chat data based on a target portion of user voice data received from the platform; and an acoustic model that generates a set of acoustic features for the target portion of user voice data. ### Claim 6 (depends on 5) The server system of claim 5, wherein the at least one trained transformer machine learning model is further configured to predict a label for the type of offensive language for the target portion of user chat data based on receiving a concatenated input that includes: the output encoding from the BERT model that processed the target portion of user chat data generated by the speech to text model based on the target portion of user voice data, and the set of acoustic features generated by the acoustic model for the target portion of user voice data. ### Claim 7 (depends on 1) The server system of claim 1, wherein the plurality of trained machine learning models includes at least one trained convolutional neural network (CNN) machine learning model configured to receive input embeddings from a Word2Vec model that processed the target portion of user chat data from the platform, and output a predicted label for an offensive language presence for the target portion of user chat data. ### Claim 8 (depends on 7) The server system of claim 7, wherein the at least one trained CNN machine learning model is further configured to receive input embeddings from a gaming language dictionary encoder that is cross-entropy trained on a corpus of user chat data. ### Claim 9 (depends on 1) The server system of claim 1, wherein the user chat filtering program further includes a heuristic offensive language checking module that includes a mapping between words or phrases and predicted labels for a type of offensive language, and wherein the filter decision service is configured to determine the filter action to be performed for the target portion of user chat data further based on an output of the heuristic offensive language checking module that processed the target portion of user chat data. ### Claim 10 (depends on 1) The server system of claim 1, wherein the user chat filtering program is further configured to: send a feedback package to a moderator user computer device, the feedback package including the target portion of the user chat data and the output of the plurality of trained machine learning models for the target portion of user chat data; receive a feedback response from the moderator user computer device that indicates a moderator selected label of offensive language for the target portion of the user chat data; and perform batched feedback training on the plurality of trained machine learning models based on the received feedback response from the moderator user computer device. ### Claim 11 (depends on 1) The server system of claim 1, wherein the user chat filtering program is configured to determine the filter action to be performed for target portions of user chat data based on output of the plurality of trained machine learning models for those target portions of user chat data in real-time as the target portions of user chat data are received from the plurality of client devices at the platform. ### Claim 12 A method comprising: at one or more processors of a computer device: executing a platform for an online multi-user chat service that communicates with a plurality of client devices of users of the online multi-user chat service and exchange user chat data between the plurality of client devices; executing a user chat filtering program for performing filter actions for user chat data exchanged on the platform for the online multi-user chat service, the user chat filtering program including a plurality of trained machine learning models and a filter decision service for determining a filter action to be performed for target portions of user chat data based on output of the plurality of trained machine learning models for those target portions of user chat data; receiving user chat data that includes a first user chat data and a second user chat data; processing the first user chat data and the second user chat data using the plurality of trained machine learning models; determining that the first user chat data that includes a target word or phrase is offensive language based on output of the plurality of trained machine learning models for the first user chat data; determining that the second user chat data that includes the same target word or phrase is not offensive language based on output of the plurality of trained machine learning models for the second user chat data; and determining a filter action to be performed for the first user chat data. ### Claim 13 (depends on 12) The method of claim 12, wherein the plurality of trained machine learning models includes at least one trained transformer machine learning model that includes a transformer encoder and a plurality of fully connected deep learning layers, wherein the transformer encoder receives an output encoding from a bidirectional encoder representations from transformers (BERT) model that processed the target portion of user chat data from the platform, and wherein the fully connected deep learning layers output a predicted label for a type of offensive language for the target portion of user chat data. ### Claim 14 (depends on 13) The method of claim 13, wherein the at least one trained transformer machine learning model is one of a plurality of trained transformer machine learning models; and each of the plurality of trained transformer machine learning models is respectively trained to predict labels for a different respective type of offensive language. ### Claim 15 (depends on 12) The method of claim 12, wherein the plurality of trained machine learning models includes at least one trained convolutional neural network (CNN) machine learning model that receives input embeddings from a Word2Vec model that processed the target portion of user chat data from the platform, and outputs a predicted label for an offensive language presence for the target portion of user chat data. ### Claim 16 (depends on 15) The method of claim 15, wherein the at least one trained CNN machine learning model further receives input embeddings from a gaming language dictionary encoder that is cross-entropy trained on a corpus of user chat data. ### Claim 17 A computer device comprising: a processor configured to: execute a client-side chat service program configured to communicate with a platform for an online multi-user chat service, the client-side chat service program including a graphical user interface (GUI) that presents user chat data received from the platform; receive a user selected filter preference that indicates to filter out offensive language from the user chat data presented to a user of the computer device via the GUI of the client-side chat service program; send the user selected filter preference to the platform; receive user chat data from the platform that is filtered based on the user selected filter preference; present the filtered user chat data via the GUI of the client-side chat service program, wherein the received user chat data is filtered such that a first user chat data that includes a target word or phrase that is labeled as offensive language based on output of a plurality of trained machine learning models executed by the platform is filtered out of the user chat data, and a second user chat data that includes the same target word or phrase that is labeled as not offensive language based on output from the plurality of trained machine learning models executed by the platform is not filtered out of the user chat data. ### Claim 18 (depends on 17) The computer device of claim 17, wherein the user selected filter preference is selected from a list of types of offensive language that includes a profanity type, a bullying type, an adult language type, and a hate speech type; and wherein the hate speech type includes at least one of racist hate speech type and/or sexist hate speech type. ### Claim 19 (depends on 17) The computer device of claim 17, wherein the user chat data is filtered by the platform before being sent to the computer device. ### Claim 20 (depends on 17) The computer device of claim 17, wherein the client-side chat service program is configured to: receive the first user chat data and the second user chat data in the user chat data; receive the output from the plurality of trained machine learning models executed by the platform for the first user chat data and the second user chat data; and determine a filter action to be performed on the user chat data based on the output from the plurality of trained machine learning models and the user selected filter preference.