Scene annotation using machine learning
AssigneeSony Interactive Entertainment Inc.
InventorsSudha Krishnamurthy, Justice Adams, Arindam Jati, Masanori Omote, Jian Zheng
Life of this patent
Estimating the term…
Abstract
A system enhances existing audio-visual content with audio describing the setting of the visual content. A scene annotation module classifies scene elements from an image frame received from a host system and generates a caption describing the scene elements. A text to speech synthesis module may then convert the caption to synthesized speech data describing the scene elements within the image frame
Claims13
A system for enhancing the accessibility of Audio Visual content, the system comprising:
- a scene annotation module configured to classify scene elements from an image frame received from a host system and generate a caption describing the scene elements wherein the scene annotation module includes a first neural network configured to generate a feature vector from the image frame and a second neural network configured to generate a caption describing elements within the image frame from the feature vector;
- a controller coupled to the scene annotation module wherein the controller is configured to discard the image frame without the image frame being processed by the scene annotation module unless an activation input is received by the controller.
A method for enhancing the accessibility of Audio Visual content, comprising:
- classifying scene elements from an image frame received from a controller with a scene annotation module wherein classifying scene elements includes generating a feature vector from the image with a first neural network of the scene annotation module and generating a caption describing the scene elements with the scene annotation module wherein generating a caption describing elements within the image frame from the feature vector with a second neural network of the scene annotation module, wherein the controller discards the image frame without the frame being processed by the scene annotation module unless an activation input is received by the controller.
A non-transitory computer-readable medium having computer readable instructions embodied therein, the computer-readable instructions being configured, when executed to implement a method for enhancing the accessibility of Audio Visual content, the method comprising:
- classifying scene elements from an image frame received from a controller with a scene annotation module and generating a caption describing the scene elements with the scene annotation module wherein the scene annotation module includes a first neural network configured to generate a feature vector from the image frame and a second neural network configured to generate a caption describing elements within the image frame from the feature vector, wherein the controller discards the image frame without the image frame being processed by the scene annotation module unless an activation input is received by the controller.
Dependencies are inferred from the claim text.