Generating short videos from live video streams

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining a live stream of video content that comprises a creator content stream and one or more viewer content streams, wherein the creator content stream comprises visual data and audio data provided by a content creator, and wherein each viewer content stream comprises action data that characterizes actions performed by a content viewer while observing the visual data and audio data provided by the content creator;
• for each time interval in a plurality of successive time intervals during the live stream of the video content:
• generating a candidate short video using a multi-modal neural network system to process a multi-modal input that comprises (i) a portion of the creator content stream that corresponds to the time interval and (ii) for each viewer content stream, a portion of the action data that characterizes actions performed by the content viewer within the time interval, wherein the candidate short video comprises a subset of video frames selected from video frames included in the creator content stream within the time interval and a subset of audio frames selected from audio frames included in the creator content stream within the time interval;
• selecting, from the candidate short videos generated for the plurality of successive time intervals during the live stream of the video content, one or more short videos;
• and
• providing the one or more selected short videos for presentation to the content creator.

One or more non-transitory computer readable medium storing instructions, that when executed by an artificial intelligence system, causes the artificial intelligence system to perform operations comprising:

• obtaining a live stream of video content that comprises a creator content stream and one or more viewer content streams, wherein the creator content stream comprises visual data and audio data provided by a content creator, and wherein each viewer content stream comprises action data that characterizes actions performed by a content viewer while observing the visual data and audio data provided by the content creator;
• for each time interval in a plurality of successive time intervals during the live stream of the video content:
• generating a candidate short video using a multi-modal neural network system to process a multi-modal input that comprises (i) a portion of the creator content stream that corresponds to the time interval and (ii) for each viewer content stream, a portion of the action data that characterizes actions performed by the content viewer within the time interval, wherein the candidate short video comprises a subset of video frames selected from video frames included in the creator content stream within the time interval and a subset of audio frames selected from audio frames included in the creator content stream within the time interval;
• selecting, from the candidate short videos generated for the plurality of successive time intervals during the live stream of the video content, one or more short videos;
• and
• providing the one or more selected short videos for presentation to the content creator.