# US 2024/0205368 A1 · Methods and Apparatus for Displaying, Compressing and/or Indexing Information Relating to a Meeting > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0205368 A1 - Document ID: US20240205368A1 - Status: Published application - Kind: A1 - Application number: 18/589,677 - Filed: 2024-02-28 - Published: 2024-06-20 - Earliest claimed: 2019-04-12 - Assignees: Pinch Labs Pty Ltd - Applicants: Pinch Labs Pty Ltd. - Inventors: Christopher RAETHKE; Saxon FLETCHER; Jaco DU PLESSIS; Andrew CUPPER; Iain McCowan - Web page: https://patentgrep.com/patent/US20240205368A1 ## CPC - H04N 7/155: {involving storage of or access to video conference sessions (tracking arrangements for later retrieval of a computer conference content or participants activities H04L12/1831)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 18/23213: with fixed number of clusters, e.g. K-means clustering - G06F 18/25: Fusion techniques - G06F 18/295: {Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models} - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06Q 10/10: Office automation; Time management - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 10/82: using neural networks - G06V 10/85: {Markov-related models; Markov random fields} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - H04L 12/1831: {Tracking arrangements for later retrieval, e.g. recording contents, participants activities or behavior, network status} - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/295: Named entity recognition - G06F 40/30: Semantic analysis - G06N 3/04: Architecture, e.g. interconnection topology - G06N 5/047: Pattern matching networks; Rete networks - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06V 20/44: {Event detection} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 25/63: for estimating an emotional state - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method of visualising a meeting between one or more participants on a display includes, in an electronic processing device, the steps of: determining a plurality of signals, each of the plurality of signals being at least partially indicative of the meeting; generating a plurality of features using the plurality of signals, the features being at least partially indicative of the signals; generating at least one of: at least one phase indicator associated with the plurality of features, the at least one phase indicator being indicative of a temporal segmentation of at least part of the meeting; and at least one event indicator associated with the plurality of features, the at least one event indicator being indicative of an event during the meeting. The method also includes the step of causing a representation indicative of the at least one phase indicator and/or the at least one event indicator to be displayed on the display to thereby provide visualisation of the meeting. ## Claims ### Claim 1 A method for summarizing a conversation, the method comprising, in an electronic processing device, the steps of: determining phases of the conversation from data representative of the conversation; generating textual phase indicators that describe the phases of the conversation; detecting moments within the conversation from the data representative of the conversation; generating textual moment indicators that describe the moments within the conversation; generating text summaries from audio data associated with the phases and/or the detected moments; and providing a textual representation of the conversation, which includes: the phases and moments of the conversation described by the textual phase indicators and the textual moment indicators; and the text summaries of the associated phases and/or detected moments. ### Claim 2 (depends on 1) The method according to claim 1, wherein the step of generating the textual moment indicators includes the step of classifying the moments by training a supervised classifier on a labelled dataset using a neural network. ### Claim 3 (depends on 1) The method as claimed in claim 1, wherein the method includes, in an electronic processing device, using multi-modal signals that are indicative of the conversation, the data representative of the conversation being included in the multi-modal signals. ### Claim 4 (depends on 1) The method according to claim 1, wherein the method includes, in an electronic processing device, the steps of: determining at least one moment type associated with each moment; and generating textual indicators that describe the moment types, wherein the step of providing the representation of the phases and moments of the conversation includes the step of providing a representation of the moment types with the textual indicators. ### Claim 5 (depends on 4) The method according to claim 4, wherein the step of determining at least one moment type includes the step of determining any one of more of: a. an action; b. a mood; c. a sentiment; d. a highlight; e. a question; f. recapitulation; g. a milestone; and h. an event type determined in accordance with user input. ### Claim 6 (depends on 1) The method according to claim 1, wherein the method includes, in an electronic processing device, the steps of: generating graphical phase indicators that represent the phases of the conversation; and generating graphical moment indicators that represent the moments within the conversation. ### Claim 7 (depends on 1) The method according to claim 1, wherein the step of determining phases of the conversation includes the step of generating temporal indicators, and the step of providing a representation of the phases includes the step of providing a graphical representation of the phases using the temporal indicators. ### Claim 8 (depends on 1) The method according to claim 1, wherein the step of determining phases of the conversation includes the step of using the detected moments to determine the phases of the conversation. ### Claim 9 (depends on 1) The method according to claim 1, wherein the step of detecting moments within the conversation includes the step of anomaly detection. ### Claim 10 (depends on 3) The method according to claim 3, wherein the step of determining phases of the conversation includes the steps of: generating a plurality of feature vectors wherein each feature vector is generated using a respective one of the multi-modal signals; generating a plurality of multi-modal feature vectors using the feature vectors; and classifying the multi-modal feature vectors. ### Claim 11 (depends on 3) The method according to claim 3, wherein the step of detecting moments within the conversation includes the steps of: generating a plurality of feature vectors wherein each feature vector is generated using a respective one of the multi-modal signals; generating a plurality of multi-modal feature vectors using the feature vectors; classifying the multi-modal feature vectors. ### Claim 12 (depends on 1) The method according to claim 1, wherein the method includes, in the electronic processing device, determining at least one of: a. at least one phase parameter; and b. at least one event parameter, wherein the phase and moment indicators are in accordance with at least one of the phase and event parameters. ### Claim 13 (depends on 1) The method according to claim 1, wherein the step of generating textual phase indicators that describe the phases of the conversation includes the step of incorporating summary keywords relating to the phases. ### Claim 14 (depends on 1) The method according to claim 1, wherein the step of generating textual moment indicators that describe the moments within the conversation includes the step of incorporating summary keywords relating to the moments. ### Claim 15 (depends on 1) The method according to claim 1, wherein the steps of determining phases of the conversation and detecting moments within the conversation includes the step of extracting cues from a variety of sources. ### Claim 16 (depends on 1) The method according to claim 1, wherein the steps of determining phases of the conversation and detecting moments within the conversation includes the step of extracting cues from information relating to a context of the conversation and generating the textual phase and moment indicators from those cues. ### Claim 17 (depends on 16) The method according to claim 16, wherein the information relating to the context of the conversation includes information that is of significance to a user, including, but not limited to industries, companies, professions, geographic location, and meeting types. ### Claim 18 (depends on 1) The method according to claim 1, wherein the method includes, in the electronic processing device, the steps of: receiving an input selection associated with at least one of the phase indicator and the moment indicator; selectively updating the representation to include an indication of one or more features, respectively, associated with the selected one of the phase indicator and/or the event indicator; and causing the update representation to be displayed. ### Claim 19 (depends on 1) The method according to claim 1, wherein the representation includes at least one temporal indicator indicative of a time range of the meeting. ### Claim 20 An apparatus for summarizing a conversation, the apparatus including an electronic processing device that is configured for carrying out the steps of: determining phases of the conversation from data representative of the conversation; generating textual phase indicators that describe the phases of the conversation; detecting moments within the conversation from the data representative of the conversation; generating textual moment indicators that describe the moments within the conversation, respectively; generating text summaries from audio data associated with the phases and/or detected moments, respectively; and providing a textual representation of the conversation, which includes: the phases and moments of the conversation described by the textual phase indicators and the textual moment indicators; and the text summaries of the associated phases and/or detected moments.