System
AssigneeSoftBank Group Corp.
InventorShinji OWAKI
Life of this patent
Estimating the term…
Abstract
A system includes a processor that is configured to access restaurant information within a map application and acquire one or more images of a menu included in the restaurant information, extract text from the acquired image or images by using optical character recognition, extract and summarize information relating to the menu from the extracted text, generate a prompt for instructing input of the extracted and summarized information to a generative artificial intelligence model, and recommend a restaurant or a menu based on a user emotion by using an emotion engine that recognizes the user emotion.
Claims20
A system comprising:
- circuitry configured to:
- receive, via a communication interface coupled to a packet-switched network, image data comprising provision information from a facility information source, and store the image data in a storage device;
- apply character recognition processing comprising optical character recognition to the image data to extract character string data from the provision information;
- analyze the extracted character string data to extract item name information and price information, and apply rule-based expression processing to convert the extracted information into structured provision data in a predefined schema;
- generate a prompt sentence for causing a generative AI model to process input data comprising at least one of the extracted character string data and the structured provision data, and transmit the prompt sentence and the input data to the generative AI model;
- obtain response data from the generative AI model comprising at least one of classification information and summary information related to the provision information, and integrate the structured provision data and the response data to generate integrated provision data;
- and
- receive user state data from a terminal device, apply an emotion recognition model to the user state data to generate an emotional state representation, and transmit provision recommendation data based on the integrated provision data and the emotional state representation to the terminal device via the communication interface.
A system comprising:
- circuitry configured to:
- receive, via a communication interface coupled to a packet-switched network, image data comprising provision information from a facility information source, apply character recognition processing to the image data to extract character string data, and apply rule-based expression processing to convert the extracted character string data into structured provision data;
- apply a region detection model and a layout analysis model to the image data to identify structural elements of the provision information, apply a convolutional neural network to generate image feature vectors, and generate a multimodal input comprising the structured provision data and the image feature vectors;
- generate a prompt sentence incorporating the multimodal input, transmit the prompt sentence to a generative AI model, and integrate structured provision data with response data received from the generative AI model to generate integrated provision data;
- receive user state data from a terminal device, apply an emotion recognition model to the user state data to generate an emotional state representation with a confidence level, and apply a ranking model to the integrated provision data using the emotional state representation as a weighting parameter;
- and
- transmit provision recommendation data comprising ranked integrated provision data and explanation information to the terminal device via the communication interface.
A method comprising:
- receiving, via a communication interface coupled to a packet-switched network, image data comprising provision information from a facility information source, and storing the image data in a storage device;
- applying character recognition processing comprising optical character recognition to the image data to extract character string data from the provision information;
- analyzing the extracted character string data to extract item name information and price information, and applying rule-based expression processing to convert the extracted information into structured provision data in a predefined schema;
- generating a prompt sentence for causing a generative AI model to process input data comprising at least one of the extracted character string data and the structured provision data, and transmitting the prompt sentence and the input data to the generative AI model;
- obtaining response data from the generative AI model comprising at least one of classification information and summary information related to the provision information, and integrating the structured provision data and the response data to generate integrated provision data;
- and
- receiving user state data from a terminal device, applying an emotion recognition model to the user state data to generate an emotional state representation, and transmitting provision recommendation data based on the integrated provision data and the emotional state representation to the terminal device via the communication interface.
Dependencies are inferred from the claim text.