# US 2026/0290061 A1 · System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0290061 A1
- Document ID: US20260290061A1
- Status: Published application
- Kind: A1
- Application number: 19/568,805
- Filed: 2026-03-17
- Published: 2026-09-24
- Earliest claimed: 2025-03-19
- Assignees: SoftBank Group Corp.
- Applicants: SoftBank Group Corp.
- Inventors: Shinji OWAKI
- Web page: https://patentgrep.com/patent/US20260290061A1

## CPC

- G06V 30/19173: {Classification techniques}
- G06F 16/901: Indexing; Data structures therefor; Storage structures (for retrieval from the web G06F16/951)
- G06F 16/90335: {Query processing}
- G06F 40/30: Semantic analysis
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/0475: Generative networks
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/40: Extraction of image or video features
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 10/98: Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
- G06V 30/147: {Determination of region of interest}
- G06V 30/18: Extraction of features or characteristics of the image
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system includes a processor that is configured to access restaurant information within a map application and acquire one or more images of a menu included in the restaurant information, extract text from the acquired image or images by using optical character recognition, extract and summarize information relating to the menu from the extracted text, generate a prompt for instructing input of the extracted and summarized information to a generative artificial intelligence model, and recommend a restaurant or a menu based on a user emotion by using an emotion engine that recognizes the user emotion.

## Claims

### Claim 1

A system comprising:
circuitry configured to:
receive, via a communication interface coupled to a packet-switched network, image data comprising provision information from a facility information source, and store the image data in a storage device;
apply character recognition processing comprising optical character recognition to the image data to extract character string data from the provision information;
analyze the extracted character string data to extract item name information and price information, and apply rule-based expression processing to convert the extracted information into structured provision data in a predefined schema;
generate a prompt sentence for causing a generative AI model to process input data comprising at least one of the extracted character string data and the structured provision data, and transmit the prompt sentence and the input data to the generative AI model;
obtain response data from the generative AI model comprising at least one of classification information and summary information related to the provision information, and integrate the structured provision data and the response data to generate integrated provision data; and
receive user state data from a terminal device, apply an emotion recognition model to the user state data to generate an emotional state representation, and transmit provision recommendation data based on the integrated provision data and the emotional state representation to the terminal device via the communication interface.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the circuitry is configured to apply a natural language processing model to the integrated provision data to identify semantic categories of provision items, and generate a structured index mapping semantic categories to corresponding provision data records in the storage device.

### Claim 3 (depends on 2)

The system according to claim 2, wherein the circuitry is configured to receive a search query from the terminal device, apply a text matching model to the search query and the structured index to retrieve matching provision data records, and transmit search results to the terminal device.

### Claim 4 (depends on 3)

The system according to claim 3, wherein the circuitry is configured to apply a ranking model to the retrieved matching provision data records using the emotional state representation as a weighting parameter, and return a ranked list of provision data records to the terminal device.

### Claim 5 (depends on 4)

The system according to claim 4, wherein the circuitry is configured to generate explanation information for each provision data record in the ranked list using the generative AI model, and transmit the explanation information to the terminal device together with the ranked list.

### Claim 6 (depends on 1)

The system according to claim 1, wherein the circuitry is configured to apply a context analysis model to contextual information associated with the user state data, and generate context information comprising user preference parameters and situational parameters for input to the generative AI model together with the provision recommendation data.

### Claim 7 (depends on 6)

The system according to claim 6, wherein the circuitry is configured to cause the generative AI model to generate recommended provision items suitable for the emotional state representation based on the context information and the integrated provision data.

### Claim 8 (depends on 1)

The system according to claim 1, wherein the circuitry is configured to apply a region detection model to the image data to identify text regions within the provision information, apply optical character recognition to the identified text regions to extract character strings, and apply an error correction model to the extracted character strings to improve recognition accuracy.

### Claim 9 (depends on 8)

The system according to claim 8, wherein the circuitry is configured to apply a layout analysis model to the image data to identify structural elements of the provision information comprising table structures, section headers, and item groupings, and use the identified structural elements to guide the rule-based expression processing.

### Claim 10 (depends on 1)

The system according to claim 1, wherein the circuitry is configured to receive audio data from the terminal device, apply a speech recognition model to the audio data to generate a text query, and use the text query as a search query for retrieval of matching provision data records from the storage device.

### Claim 11 (depends on 10)

The system according to claim 10, wherein the circuitry is configured to apply a natural language understanding model to the text query to extract search intent and constraint parameters, and apply the search intent and constraint parameters to filter matching provision data records.

### Claim 12 (depends on 1)

The system according to claim 1, wherein the circuitry is configured to apply a multimodal feature extraction model to the image data to extract visual features of the provision information in addition to the character string data, and incorporate the visual features as additional input data in the prompt sentence transmitted to the generative AI model.

### Claim 13 (depends on 12)

The system according to claim 12, wherein the circuitry is configured to apply a convolutional neural network to the image data to generate image feature vectors, and combine the image feature vectors with the structured provision data as a multimodal input for the generative AI model.

### Claim 14 (depends on 1)

The system according to claim 1, wherein the circuitry is configured to store the integrated provision data in a structured database indexed by item name information, price information, and semantic category, and update the structured database upon receipt of new image data from the facility information source.

### Claim 15 (depends on 14)

The system according to claim 14, wherein the circuitry is configured to apply a change detection model to compare newly received image data with stored image data to identify updated provision information, and apply optical character recognition and structured data processing only to changed regions of the image data.

### Claim 16 (depends on 1)

The system according to claim 1, wherein the circuitry is configured to accumulate emotional state representations and corresponding provision recommendation data in the storage device, apply a preference learning model to the accumulated data to identify user preference patterns, and update recommendation parameters based on the identified preference patterns.

### Claim 17 (depends on 16)

The system according to claim 16, wherein the circuitry is configured to generate updated prompt sentences incorporating the updated recommendation parameters, and transmit the updated prompt sentences to the generative AI model to refine provision recommendation data.

### Claim 18

A system comprising:
circuitry configured to:
receive, via a communication interface coupled to a packet-switched network, image data comprising provision information from a facility information source, apply character recognition processing to the image data to extract character string data, and apply rule-based expression processing to convert the extracted character string data into structured provision data;
apply a region detection model and a layout analysis model to the image data to identify structural elements of the provision information, apply a convolutional neural network to generate image feature vectors, and generate a multimodal input comprising the structured provision data and the image feature vectors;
generate a prompt sentence incorporating the multimodal input, transmit the prompt sentence to a generative AI model, and integrate structured provision data with response data received from the generative AI model to generate integrated provision data;
receive user state data from a terminal device, apply an emotion recognition model to the user state data to generate an emotional state representation with a confidence level, and apply a ranking model to the integrated provision data using the emotional state representation as a weighting parameter; and
transmit provision recommendation data comprising ranked integrated provision data and explanation information to the terminal device via the communication interface.

### Claim 19 (depends on 18)

The system according to claim 18, wherein the circuitry is configured to accumulate emotional state representations and corresponding provision recommendation data in the storage device, apply a preference learning model to the accumulated data to identify user preference patterns, update recommendation parameters, and generate updated prompt sentences incorporating the updated parameters for input to the generative AI model.

### Claim 20

A method comprising:
receiving, via a communication interface coupled to a packet-switched network, image data comprising provision information from a facility information source, and storing the image data in a storage device;
applying character recognition processing comprising optical character recognition to the image data to extract character string data from the provision information;
analyzing the extracted character string data to extract item name information and price information, and applying rule-based expression processing to convert the extracted information into structured provision data in a predefined schema;
generating a prompt sentence for causing a generative AI model to process input data comprising at least one of the extracted character string data and the structured provision data, and transmitting the prompt sentence and the input data to the generative AI model;
obtaining response data from the generative AI model comprising at least one of classification information and summary information related to the provision information, and integrating the structured provision data and the response data to generate integrated provision data; and
receiving user state data from a terminal device, applying an emotion recognition model to the user state data to generate an emotional state representation, and transmitting provision recommendation data based on the integrated provision data and the emotional state representation to the terminal device via the communication interface.
