# US 2026/0311446 A1 · Method for fine-tuning a multimodal large language model machine to generate educational level-specific feedback in scene description tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0311446 A1
- Document ID: US20260311446A1
- Status: Published application
- Kind: A1
- Application number: 19/169,024
- Filed: 2025-04-03
- Published: 2026-10-08
- Earliest claimed: 2025-04-03
- Assignees: The Education University of Hong Kong
- Applicants: The Education University of Hong Kong
- Inventors: Leung Ho Philip YU; Zhiwei XIE
- Web page: https://patentgrep.com/patent/US20260311446A1

## CPC

- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06Q 50/20: Education
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

The present invention provides a method for fine-tuning a multimodal large language model (MLLM) machine, based on a vision encoder, a vision-language connector and a foundational large language model device, to autonomously generate educational level-specific feedback in scene description tasks with respect to different educational stages. The method comprises: constructing a multimodal training dataset comprising image-text-feedback samples, wherein each image-text-feedback sample includes an image sample, scene-description text sample matched with the image sample, and educational level-specific feedback sample on the descriptive text sample with respect to an educational stage; and fine-tuning the MLLM machine with the constructed multimodal training dataset. The fine-tuned MLLM machine can maximize the pre-trained capabilities of language and visual models to generate precise and pedagogically appropriate educational level-specific feedback while ensuring scalability in terms of data and model size, therefore marking a significant step forward in AI-assisted writing support for education.

## Claims

### Claim 1

A method for fine-tuning a multimodal large language model machine to autonomously generate educational level-specific feedback in scene description tasks with respect to different educational stages, wherein the multimodal large language model machine is based on a vision encoder, a vision-language connector and a foundational large language model device, and the method comprising:
constructing a multimodal training dataset comprising image-text-feedback samples, wherein each image-text-feedback sample includes an image sample, scene-description text sample matched with the image sample, and a corresponding educational level-specific feedback sample with respect to an educational stage; and
fine-tuning the multimodal large language model machine with the constructed multimodal training dataset to generate educational level-specific feedback in scene description tasks by:
training the vision encoder to extract visual features of an input image;
training the vision-language connector to project the visual features extracted by the vision encoder into word embedding space of the foundational large language model device to obtain a sequence of visual tokens; and
training the foundational large language model device to generate educational level-specific feedback based on the obtained sequence of visual tokens.

### Claim 2 (depends on 1)

The method of claim 1, wherein the multimodal training dataset is constructed by:
selecting one or more scene-description text samples for each image sample to form one or more image-text sample pairs respectively; and
generating one or more educational level-specific feedback samples corresponding to each image-text sample pair with respect to one or more educational stages; and
mapping the generated educational level-specific feedback samples with the image-text sample pairs for each educational stage to form the image-text-feedback samples.

### Claim 3 (depends on 2)

The method of claim 2, wherein the one or more scene-description text samples are selected from description written by students at different educational stages.

### Claim 4 (depends on 3)

The method of claim 3, each educational level-specific feedback sample is generated by using a retrieval-augmented generation network comprising:
a retriever configured to retrieves educationally appropriate vocabulary from a predefined corpus structured into multiple educational stages; and
a generator configured to generate the educational level-specific feedback sample based on the retrieved educationally appropriate vocabulary.

### Claim 5 (depends on 1)

The method of claim 1, further comprising designing an instruction prompt template for configuring the generator to generate the educational level-specific feedback sample with respect to a designated educational stage across multiple metrics including comprehensiveness, vividness, mechanics, vocabulary choice, and sentence structure.

### Claim 6 (depends on 1)

The method of claim 1, wherein
the vision encoder is a vision transformer pre-trained for visual feature extraction; and
the vision-language connector is a two-layer multilayer perceptron projector with gaussian error linear unit (GELU) activation pretrained for enhancing alignment of visual and textual features.

### Claim 7 (depends on 1)

The method of claim 1, wherein the multimodal large language model machine is fine-tuned by adopting low-rank adaptation (LoRA) to update a weight matrix of the multimodal large language model machine.

### Claim 8 (depends on 7)

The method of claim 7, wherein the low-rank adaptation is set to have a rank of 128 and a scaling factor of 256.

### Claim 9 (depends on 1)

The method of claim 1, further the multimodal large language model machine is fine-tuned using a visual instruction tuning approach.

### Claim 10 (depends on 1)

The method of claim 1, wherein the vision encoder is trained to extract visual features of image samples by:
segmenting each image sample into patches;
performing independent encoding on each patch to extract visual feature from the patch; and
concatenating visual features extracted from the patches to obtain a unified feature map of for the image sample.

### Claim 11 (depends on 1)

An artificial intelligence-based method of using a multimodal large language model machine fine-tuned with the method of claim 1 to autonomously generate educational level-specific feedback on scene description tasks with respect to different educational stages.

### Claim 12 (depends on 11)

The artificial intelligence-based method of claim 11, wherein the multimodal training dataset is constructed by:
selecting one or more scene-description text samples for each image sample to form one or more image-text sample pairs respectively; and
generating one or more educational level-specific feedback samples corresponding to each image-text sample pair with respect to one or more educational stages; and
mapping the generated educational level-specific feedback samples with the image-text sample pairs for each educational stage to form the image-text-feedback samples.

### Claim 13 (depends on 12)

The artificial intelligence-based method of claim 12, wherein the one or more scene-description text samples are selected from description written by students at different educational stages.

### Claim 14 (depends on 13)

The artificial intelligence-based method of claim 13, wherein each educational level-specific feedback sample is generated by using a retrieval-augmented generation network comprising:
a retriever configured to retrieves educationally appropriate vocabulary from a predefined corpus structured into multiple educational stages; and
a generator configured to generate the educational level-specific feedback sample based on the retrieved educationally appropriate vocabulary.

### Claim 15 (depends on 11)

The artificial intelligence-based method of claim 11, further comprising designing an instruction prompt template for configuring the generator to generate the educational level-specific feedback sample with respect to a designated educational stage across multiple metrics including comprehensiveness, vividness, mechanics, vocabulary choice, and sentence structure.

### Claim 16 (depends on 11)

The artificial intelligence-based method of claim 11, wherein
the vision encoder is a vision transformer pre-trained for visual feature extraction; and
the vision-language connector is a two-layer multilayer perceptron projector with gaussian error linear unit (GELU) activation pretrained for enhancing alignment of visual and textual features.

### Claim 17 (depends on 11)

The artificial intelligence-based method of claim 11, wherein the multimodal large language model machine is fine-tuned by adopting low-rank adaptation (LoRA) to update a weight matrix of the multimodal large language model machine.

### Claim 18 (depends on 17)

The artificial intelligence-based method of claim 17, wherein the low-rank adaptation is set to have a rank of 128 and a scaling factor of 256.

### Claim 19 (depends on 11)

The artificial intelligence-based method of claim 11, further the multimodal large language model machine is fine-tuned using a visual instruction tuning approach.

### Claim 20 (depends on 11)

The artificial intelligence-based method of claim 11, wherein the vision encoder is trained to extract visual features of image samples by:
segmenting each image sample into patches;
performing independent encoding on each patch to extract visual feature from the patch; and
concatenating visual features extracted from the patches to obtain a unified feature map of for the image sample.
