A system for using video content of a video stream of a first recorded teleconference among a plurality of participants to diarize speech, the system comprising:
- one or more processors;
- and
- non-transitory computer-readable memory operatively connected to the one or more processors, the non-transitory computer-readable memory including machine readable instructions that, when executed by the one or more processors cause the one or more processors to perform steps of:
- (a) obtaining components of the first recorded teleconference among the plurality of participants conducted over a network, wherein the components include:
- (1) an audio component including utterances of respective participants that spoke during the first recorded teleconference;
- (2) a video component including a video feed comprising video of respective participants that spoke during the first recorded teleconference;
- (3) teleconference metadata associated with the first recorded teleconference and including a first plurality of timestamp information and respective speaker identification information associated with each respective timestamp information;
- and,
- (4) transcription data associated with the first recorded teleconference, wherein said transcription data is indexed by timestamps;
- (b) parsing the audio component into a plurality of speech segments in which one or more participants were speaking during the first recorded teleconference, wherein each respective speech segment is associated with a respective time segment including a start timestamp indicating a first time in the first recorded teleconference when the respective speech segment begins, and a stop timestamp associated with a second time in the first recorded teleconference when the respective speech segment ends;
- (c) tagging each respective speech segment with the respective speaker identification information based on the teleconference metadata associated with the respective time segment;
- and
- (d) diarizing the first recorded teleconference in a process comprising:
- (1) indexing the transcription data in accordance with respective speech segments and the respective speaker identification information to generate a segmented transcription data set for the first recorded teleconference;
- (2) identifying respective spoken dialogue information associated with respective speech segments using a neural network with at least a portion of the video feed comprising video of at least one participant among the respective participants corresponding in time to at least a portion of the segmented transcription data set determined according to the indexing as an input, and spoken dialogue indication information as an output and a training set including a plurality of videos of persons tagged with indications of what spoken dialogue the respective persons are speaking, wherein the portion of the video feed includes a first artificial visual representation not including a face generated by telephone conferencing software in the visual content associated with a first participant that spoke during a first speech segment of the first recorded teleconference, and the portion of the video feed does not include any artificial visual representation associated with a second participant that did not speak during the first speech segment of the first recorded teleconference, and the speaker identification information is based at least on presence of the first artificial visual representation;
- and
- (3) updating the transcription data based on the identified respective spoken dialogue information associated with the respective speech segment.