Back to results
US 2022/0067384 A1Published application
Multimodal game video summarization
AssigneeSony Interactive Entertainment Inc.
InventorsLakshmish Kaushik, Saket Kumar, Jaekwon Yoo, Kevin Zhang, Soheil Khorram, Sharath Rao, Chockalingam Ravi Sundaram
Life of this patent
Estimating the term…
Abstract
Video and audio from a computer simulation are processed by a machine learning engine to identify candidate segments of the simulation for use in a video summary of the simulation. Text input is then used to reinforce whether a candidate segment should be included in the video summary.
Claims24
An apparatus comprising:
- at least one processor programmed with instructions to:
- receive audio-video (AV) data;
- provide a video summary of the AV data that is shorter than the AV data at least in part by:
- input to a machine learning (ML) engine first modality data;
- input to the ML engine second modality data;
- and
- receive the video summary of the AV data from the ML engine responsive to the inputting of the first and second modality data.
4 more elements
A method comprising:
- identifying an audio-video (AV) entity;
- using audio from the AV entity, identifying plural first candidate segments of the AV entity for establishing a summary of the entity;
- using video from the AV entity, identifying plural second candidate segments of the AV entity for establishing a summary of the entity;
- identifying at least one parameter associated with chat related to the AV entity;
- selecting at least some of the plural first and second candidate segments based at least in part on the parameter;
- and
- using the at least some of the plural first and second candidate segments, generating a video summary of the AV entity that is shorter than the AV entity.
4 more elements
An assembly, comprising:
- at least one display apparatus configured to present an audio-video (AV) computer game;
- at least one processor associated with the display apparatus and configured with instructions to execute a machine learning (ML) engine to generate a video summary of the computer game that is shorter than the computer game, the ML engine comprising:
- an acoustic event ML model trained to identify events in audio of the computer game;
- a speech pitch and power ML model trained to identify pitch and power in speech of the audio;
- a speech emotion ML model trained to identify emotion in the audio;
- a scene change detector ML model trained to identify scene changes in video of the computer game;
- a text sentiment detector model trained to identify sentiment in text associated with chat related to the computer game;
- a text emotion detector model trained to identify emotion in text associated with the chat;
- a text topic detector model trained to identify at least one topic of text associated with the chat;
- and
- an event relevance detector (ERD) module configured to receive input from the acoustic event ML model, speech pitch and power ML model, speech emotion ML model, and scene change detector ML model to identify plural candidate segments of the computer game and to select a subset of the plural candidate segments to establish the video summary based at least in part on input from one or more of the text sentiment detector model, text emotion detector model, and text topic detector model.
8 more elements
Dependencies are inferred from the claim text.