# US 2025/0166633 A1 · Generating transcriptions of audio data for presentation at a client device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0166633 A1 - Document ID: US20250166633A1 - Status: Published application - Kind: A1 - Application number: 18/511,095 - Filed: 2023-11-16 - Published: 2025-05-22 - Earliest claimed: 2023-11-16 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Dimitri Kanevsky; Sharlene Yuan; Artem Dementyev; Sagar Savla; Vinton Gray Cerf - Web page: https://patentgrep.com/patent/US20250166633A1 ## CPC - G10L 17/06: Decision making techniques; Pattern matching strategies - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - H04H 20/86: Arrangements characterised by the broadcast information itself - G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)} - G10L 15/063: {Training} - G10L 15/18: using natural language modelling - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N) - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract Systems and methods for generating transcriptions of audio data for presentation at a client device are provided. One or more audio streams, provided by one or more audio sources of one or more client devices of a plurality of users, are received by a broadcasting system. Sensory modality information comprising one or more of auditory, visual, or haptic characteristics of a first user is determined. First audio data from the one or more audio streams corresponding to the first user and additional audio data from the one or more audio streams corresponding to other users are determined using one or more machine learning models. At least one of a first transcription of the first audio data or one or more additional transcription of one or more of the additional audio data are provided for presentation at a first client device according to the auditory, visual, or haptic characteristics. ## Claims ### Claim 1 A method comprising: receiving, by a broadcasting system, one or more audio streams provided by one or more audio sources of one or more client devices of a plurality of client devices of a plurality of users of the broadcasting system; determining sensory modality information pertaining to a first user of the plurality of users, wherein the sensory modality information comprises one or more of auditory, visual, or haptic characteristics of the first user; determining, using one or more machine learning models, first audio data from the one or more audio streams corresponding to the first user and additional audio data from the one or more audio streams corresponding to other users of the plurality of users; and providing at least one of a first transcription of the first audio data or one or more additional transcriptions of one or more of the additional audio data for presentation at a first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one of the first transcription of the first audio data or the one or more additional transcriptions of the one or more of the additional audio data is presented on a user interface (UI) of the first client device, wherein the UI comprises one or more display areas, each to display one of the at least one of the first transcription or the one or more additional transcriptions, and one or more UI elements selectable to (i) display the at least one of the first transcription or the one or more additional transcriptions, and (ii) display descriptive information pertaining to the sensory modality information pertaining to the first user. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one of the first transcription of the first audio data or the one or more additional transcriptions of the one or more of the additional audio data is broadcasted to a conference meeting, wherein each of the plurality of users is a participant of a plurality of participants of the conference meeting. ### Claim 4 (depends on 1) The method of claim 1, wherein the one or more audio streams is provided by a public address (PA) system connected to the one or more client devices. ### Claim 5 (depends on 1) The method of claim 1, wherein the first client device comprises a hearing device, a computer device, a mobile device, an audio assistive device, or a haptic assistive device. ### Claim 6 (depends on 1) The method of claim 1, further comprising: providing one or more sign language visual aids for the at least one of a first transcription or the one or more additional transcriptions for presentation at the first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information. ### Claim 7 (depends on 1) The method of claim 1, further comprising: providing one or more sound notifications pertaining to one or more of the first audio data or the additional audio data for presentation at the first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more audio streams provided by the one or more audio sources of the one or more client devices are provided as one or more inputs to the one or more machine learning models. ### Claim 9 (depends on 8) The method of claim 8, wherein the one or more machine learning models comprises a first machine learning model trained, based on speech of the first user, to identify the first audio data from the one or more audio streams corresponding to the first user, and a second machine learning model trained, based on speech of the second user, to identify second audio data from the one or more audio streams corresponding to the second user, the method further comprising: providing the one or more audio streams provided by the one or more audio sources of the one or more client devices as an input to the second machine learning model; and providing a second transcription of the second audio data for presentation at a second client device according to the auditory, visual, or haptic characteristics of the second user comprised by the sensory modality information. ### Claim 10 A broadcasting system comprising: a memory device; and a processing device coupled to the memory device, the processing device to perform operations comprising: receiving one or more audio streams provided by one or more audio sources of one or more client devices of a plurality of client devices of a plurality of users of the broadcasting system; determining sensory modality information pertaining to a first user of the plurality of users, wherein the sensory modality information comprises one or more of auditory, visual, or haptic characteristics of the first user; determining, using one or more machine learning models, first audio data from the one or more audio streams corresponding to the first user and additional audio data from the one or more audio streams corresponding to other users of the plurality of users; and providing at least one of a first transcription of the first audio data or one or more additional transcriptions of one or more of the additional audio data for presentation at a first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information. ### Claim 11 (depends on 10) The broadcasting system of claim 10, wherein the at least one of the first transcription of the first audio data or the one or more additional transcriptions of the one or more of the additional audio data is presented on a user interface (UI) of the first client device, wherein the UI comprises one or more display areas, each to display one of the at least one of the first transcription or the one or more additional transcriptions, and one or more UI elements selectable to (i) display the at least one of the first transcription or the one or more additional transcriptions, and (ii) display descriptive information pertaining to the sensory modality information pertaining to the first user. ### Claim 12 (depends on 10) The broadcasting system of claim 10, wherein the at least one of the first transcription of the first audio data or the one or more additional transcriptions of the one or more of the additional audio data is broadcasted to a conference meeting, wherein each of the plurality of users is a participant of a plurality of participants of the conference meeting. ### Claim 13 (depends on 10) The broadcasting system of claim 10, wherein the one or more audio streams is provided by a public address (PA) system connected to the one or more client devices. ### Claim 14 (depends on 10) The broadcasting system of claim 10, wherein the first client device comprises a hearing device, a computer device, a mobile device, an audio assistive device, or a haptic assistive device. ### Claim 15 (depends on 10) The broadcasting system of claim 10, wherein the processing device is to perform operations further comprising: providing one or more sign language visual aids for the at least one of a first transcription or the one or more additional transcriptions for presentation at the first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information. ### Claim 16 (depends on 10) The broadcasting system of claim 10, wherein the processing device is to perform operations further comprising: providing one or more sound notifications pertaining to one or more of the first audio data or the additional audio data for presentation at the first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information. ### Claim 17 A non-transitory computer readable storage medium comprising instructions for a server that, when executed by a processing device, cause the processing device to perform operations comprising: receiving, by a broadcasting system, one or more audio streams provided by one or more audio sources of one or more client devices of a plurality of client devices of a plurality of users of the broadcasting system; determining sensory modality information pertaining to a first user of the plurality of users, wherein the sensory modality information comprises one or more of auditory, visual, or haptic characteristics of the first user; determining, using one or more machine learning models, first audio data from the one or more audio streams corresponding to the first user and additional audio data from the one or more audio streams corresponding to other users of the plurality of users; and providing at least one of a first transcription of the first audio data or one or more additional transcriptions of one or more of the additional audio data for presentation at a first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the at least one of the first transcription of the first audio data or the one or more additional transcriptions of the one or more of the additional audio data is presented on a user interface (UI) of the first client device, wherein the UI comprises one or more display areas, each to display one of the at least one of the first transcription or the one or more additional transcriptions, and one or more UI elements selectable to (i) display the at least one of the first transcription or the one or more additional transcriptions, and (ii) display descriptive information pertaining to the sensory modality information pertaining to the first user. ### Claim 19 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the at least one of the first transcription of the first audio data or the one or more additional transcriptions of the one or more of the additional audio data is broadcasted to a conference meeting, wherein each of the plurality of users is a participant of a plurality of participants of the conference meeting. ### Claim 20 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the processing device is to perform operations further comprising: providing one or more sign language visual aids for the at least one of a first transcription or the one or more additional transcriptions for presentation at the first client device according to the auditory, visual, or haptic characteristics of the first user comprised by the sensory modality information.