Method and electronic device for intelligently reading displayed contents

An electronic device for intelligently reading displayed contents, the electronic device comprising:

• a screen
• a memory storing one or more instruction;
• and
• at least one processor configured to execute the one or more instructions stored in the memory to:
• obtain a screen representation based on a plurality of contents displayed on a screen of the electronic device;
• extract a plurality of insights comprising at least one of intent, importance, emotion, sound representation and information sequence of the plurality of contents from the plurality of contents based on the screen representation;
• and
• generate audio emulating the extracted plurality of insights,
• wherein the one or more instructions to extract the plurality of insights are further configured to:
• obtain multimodal features comprising a text and at least one emoji based on the screen representation;
• generate multimodal embeddings based on the multimodal features;
• and
• classify the multimodal embeddings into a sound label belonging to the sound representation using a deep neural network (DNN).

A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:

• obtain a screen representation based on a plurality of contents displayed on a screen of an electronic device;
• extract a plurality of insights comprising at least one of intent, importance, emotion, sound representation and information sequence of the plurality of contents from the plurality of contents based on the screen representation;
• and
• generate audio emulating the extracted plurality of insights,
• wherein the extracting of the plurality of insights includes:
• obtaining multimodal features comprising a text and at least one emoji based on the screen representation;
• generating multimodal embeddings based on the multimodal features;
• and
• classifying the multimodal embeddings into a sound label belonging to the sound representation using a deep neural network (DNN).