System

A system comprising:

• circuitry configured to:
• receive, via a communication interface coupled to a packet-switched network, image data from a terminal device, and decode the image data into a normalized image tensor;
• apply a convolutional neural network to the normalized image tensor to extract character information from text regions using optical character recognition and pattern information including product category probabilities and logo identifiers, and compute a composite score for candidate identifiers using a weighted combination of the character information and the pattern information to generate identification information and feature information of a product;
• query an information storage device to retrieve document information associated with a target product identified by the identification information, extract document segments relevant to a natural language inquiry, select and truncate the document segments to fit within a token capacity of a generative neural network model, and construct a prompt sentence incorporating the selected document segments and the natural language inquiry;
• transmit the prompt sentence to the generative neural network model and receive answer text, and perform content verification processing on the answer text to evaluate a reliability score, and when the reliability score is below a threshold, transmit the prompt sentence and answer text to a human response support apparatus and incorporate supplementary information from the human response support apparatus into response data;
• and
• estimate an emotional state of the user from at least one of audio data, image data, and text data from the terminal device, adjust a description style of the prompt sentence and an expression style of the response data based on the emotional state, and transmit the response data to the terminal device via the communication interface as text data or audio data generated by speech synthesis software.

A method comprising:

• receiving, via a communication interface coupled to a packet-switched network, image data from a terminal device, and decoding the image data into a normalized image tensor using image processing software;
• applying a convolutional neural network to the normalized image tensor to locate text regions and executing optical character recognition on the located text regions to extract character information including candidate identifier strings and confidence scores, and applying a pattern recognition model to the normalized image tensor to extract pattern information including predicted product categories and logo identifiers;
• normalizing the candidate identifier strings and computing a composite score for each candidate identifier using a weighted combination of the character information and the pattern information to generate identification information and feature information of a product;
• querying an information storage device using a query language to retrieve document information associated with a target product identified by the identification information, extracting from the document information one or more document segments relevant to a natural language inquiry received from the terminal device, and selecting and truncating the document segments to fit within a token capacity of a generative neural network model;
• generating a prompt sentence incorporating the selected document segments and the natural language inquiry, transmitting the prompt sentence to the generative neural network model, and receiving answer text generated by the generative neural network model;
• performing content verification processing on the answer text to evaluate a reliability score, and when the reliability score is below a predetermined threshold, transmitting the prompt sentence and the answer text to a human response support apparatus and receiving supplementary information;
• and
• generating response data by applying formatting processing to the answer text and any supplementary information, and transmitting the response data to the terminal device via the communication interface.