Information retrieval system and method of information retrieval

A method of retrieving information from an instructional document, the method comprising:

• receiving, by at least one processor, a query from a user;
• comparing, by the at least one processor, the query with one or more text sections in the instructional document;
• obtaining, by the at least one processor and from the one or more text sections, top x text sections relevant to the query using a pre-trained encoder;
• comparing, by the at least one processor, the query with one or more images in the instructional document;
• obtaining, by the at least one processor and from the one or more images, top y images relevant to the query;
• generating, by the at least one processor, top y image-text sections based on the top y images;
• obtaining, by the at least one processor, top k sections from the top x text sections and the top y image-text sections;
• obtaining, by the at least one processor, one or more most relevant sections from the top k sections using a domain-specific pre-trained encoder and a sequential classification model;
• obtaining, by the at least one processor, device context information;
• and
• generating, by the at least one processor, an answer based on the one or more most relevant sections and the device context information using the domain-specific pre-trained encoder.

A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an electronic device, cause the one or more processors to:

• receive a query from a user;
• compare the query with one or more text sections in an instructional document;
• obtain, from the one or more text sections, top x text sections relevant to the query using a pre-trained encoder;
• compare the query with one or more images in the instructional document;
• obtain, from the images, top y images relevant to the query;
• generate top y image-text sections based on the top y images;
• obtain top k sections from the top x text sections and the top y image-text sections;
• obtain one or more most relevant sections from the top k sections using a domain-specific pre-trained encoder and a sequential classification model;
• obtain device context information;
• and
• generate an answer based on the one or more most relevant sections and the device context information using the domain-specific pre-trained encoder.