# US 2026/0311048 A1 · Vision fine-tuning for machine learning model and application programming interface (api)

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0311048 A1
- Document ID: US20260311048A1
- Status: Published application
- Kind: A1
- Application number: 19/240,232
- Filed: 2025-06-17
- Published: 2026-10-08
- Earliest claimed: 2025-03-26
- Assignees: OpenAI OpCo, LLC
- Applicants: OpenAI OpCo, LLC.
- Inventors: John Allard; James Blomo; William Hang; Steven Heidel; Todd Hendry; Radu Kopetz; Ravi Teja Mullapudi; Damian Mrowca; Sherwin Wu
- Web page: https://patentgrep.com/patent/US20260311048A1

## CPC

- G06N 20/00: Machine learning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/11: Region-based segmentation
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present technology pertains to a fine-tuned generative response engine that accepts multi-modal inputs include text and one or more images. A training set for fine-tuning the engine/model (e.g., a machine learning model) can be received via an application programming interface (API) call to an API. The training set includes inputs having input images associated with input text. Further, for supervised learning the training sets can include output text (e.g., completions that are the desired generated responses). The API call can indicate a base model (or previously fine-tuned model) to be fine-tuned. The API can cause a fine-tuned model to be trained based on the base model and the training set and can return a name for the fine-tuned model. To use the fine-tuned model to generate a response to a prompt, a subsequent API call includes the name and the prompt.

## Claims

### Claim 1

A method for using machine learning (ML) and models created thereby, comprising:
receiving, via an application programming interface (API) call, a training set for fine-tuning an ML model, the training set comprising a plurality of images associated with respective text strings;
receiving, via the API call, information indicating a first model that is to be fine-tuned using the training set;
causing a fine-tuned ML model to be trained based on the first model and the training set by performing supervised fine-tuning in which, for at least one layer of the first model having a pretrained weight matrix, pretrained weighting coefficients of the pretrained weight matrix are maintained constant while weighting coefficients of a first trainable matrix and a second trainable matrix of a low-rank adaptation layer are updated; and
returning, in response to the API call, a first identifier that identifies the fine-tuned ML model, which has been fine-tuned using the training set,
wherein the first identifier is used to call the fine-tuned ML model, and the first model is a base model or a previously fine-tuned model.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining values for one or more progress metrics representing a state of the fine-tuned ML model at respective epochs of the fine-tuning, wherein a first progress metric of the one or more progress metrics depends on validation data associated with the training set.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the fine-tuned ML model has been fine-tuned using the training set to perform supervised learning, and
the training set comprises inputs associated with outputs, each of the inputs including input text and at least one image of the plurality of images, and the outputs including, as the respective text strings of the training set, supervised-learning labels that are respectively associated with the inputs.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
providing feedback regarding the training set before the ML model is fine-tuned using the training set; and
receiving, in response to providing the feedback, modifications to the training set to provide a modified training set,
wherein the fine-tuned ML model has been fine-tuned using the modified training set as the training set.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
preventing the training set from being used to fine-tune when the training set violates or fails to satisfy one or more predefined safety criteria, and
signaling to a user an indication why the training set violates or fails to satisfy the one or more predefined safety criteria.

### Claim 6

(canceled)

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
analyzing the fine-tuned ML model based on one or more predefined safety criteria, after the fine-tuned ML model has been trained using the first model and the training set; and
signaling to a user an indication that the fine-tuned ML model failed a safety test, when the fine-tuned ML model fails to satisfy one or more predefined safety criteria.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
restricting images in the training set to a predefined maximum size; and
receiving user input indicating whether oversized images that exceed the predefined maximum size are resized or subdivided into respective tiles.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
resizing an image of the training set that exceeds the predefined maximum size, when the user input indicates that the oversized images are resized, wherein resizing the image includes scaling down the image to generate a reduced-sized image that does not exceed the predefined maximum size; and
subdividing the image into a plurality of image tiles, when the user input indicates that the oversized images are subdivided into the respective tiles, wherein subdividing the image into the plurality of image tiles includes maintaining a scale of the image and determining as the plurality of image tiles an order array of patches that individually do not exceed the predefined maximum size and that collectively span the image.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
providing progress indicia for fine-tuning of the ML model,
wherein the ML model is fine-tuned by updating the weighting coefficients of the first trainable matrix and the second trainable matrix to optimize one or more metrics when the training set is applied the ML model, and
wherein the progress indicia represents values at respective steps of the fine-tuning for the one or more metrics being optimized.

### Claim 11

(canceled)

### Claim 12 (depends on 1)

The method of claim 1,
wherein the first model is automatically trained in response to the API call.

### Claim 13 (depends on 1)

The method of claim 1, wherein:
the fine-tuned ML model includes an image embedding layer, the first model having fixed weighting coefficients, and a fine-tuned part having adjustable weighting coefficients, and
the first model lacks the image embedding layer and the first model cannot receive image inputs.

### Claim 14 (depends on 13)

The method of claim 13, wherein the image embedding layer comprises a contrastive language-image pre-training (CLIP) model that maps image tokens to image embeddings.

### Claim 15 (depends on 1)

The method of claim 1, comprising:
receiving a second API call requesting that the fine-tuned ML model be used to generate a response to a multi-modal input that includes input text and at least one input image, the second API call referring to the first identifier of the fine-tuned ML model and including the multi-modal input;
causing the multi-modal input to be applied to the fine-tuned ML model to generate the response; and
returning the response including generated text that is generated by the fine-tuned ML model in response to the multi-modal input.

### Claim 16

A method of using machine learning (ML) and models created thereby, the method comprising:
sending, to an application programming interface (API), a first API call including a training request and information of a training set, the training set including input images and text outputs, and the training request requesting that a fine-tuned ML model be generated based on the training set and based on a specified first model, wherein the first API call includes an indication of a fine-tuning method selected by a user from among a plurality of available fine-tuning methods via a user interface; and
receiving, in response to the first API call, a first identifier that identifies the fine-tuned ML model, which has been fine-tuned using the training set,
wherein the specified first model is a base model or a previously fine-tuned model.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
sending, to the API, a second API call requesting a response from the fine-tuned ML model to a multi-modal input, the second API call including the first identifier and the multi-modal input, the multi-modal input including input text and at least one input image; and
receiving, in response to the second API call, the response that is generated by the fine-tuned ML model in response to the multi-modal input.

### Claim 18

A computing apparatus for using machine learning (ML) and models created thereby, the computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the computing apparatus to:
receive, via an application programming interface (API) call, a training set for fine-tuning an ML model, the training set comprising a plurality of images associated with respective text strings;
receive, via the API call, information indicating a first model that is to be fine-tuned using the training set;
cause a fine-tuned ML model to be trained based on the first model and the training set by performing supervised fine-tuning in which, for at least one layer of the first model having a pretrained weight matrix, pretrained weighting coefficients of the pretrained weight matrix are maintained constant while weighting coefficients of a first trainable matrix and a second trainable matrix of a low-rank adaptation layer are updated; and
return, in response to the API call, a first identifier that identifies the fine-tuned ML model, which has been fine-tuned using the training set,
wherein the first identifier is used to call the fine-tuned ML model; and the first model is a base model or a previously fine-tuned model.

### Claim 19 (depends on 18)

The computing apparatus of claim 18, wherein the instructions further configure the computing apparatus to:
restrict images in the training set to a predefined maximum size;
receive user input indicating whether oversized images that exceed the predefined maximum size are resized or subdivided into respective tiles;
resize an image of the training set that exceeds the predefined maximum size, when the user input indicates that the oversized images are resized, wherein resizing the image includes scaling down the image to generate a reduced-sized image that does not exceed the predefined maximum size; and
subdivide the image into a plurality of image tiles, when the user input indicates that the oversized images are subdivided into the respective tiles, wherein subdividing the image into the plurality of image tiles includes maintaining a scale of the image and determining as the plurality of image tiles an order array of patches that individually do not exceed the predefined maximum size and that collectively span the image.

### Claim 20

(canceled)

### Claim 21 (depends on 18)

The computing apparatus of claim 18, wherein the instructions further configure the computing apparatus to:
receive a second API call requesting that the fine-tuned ML model be used to generate a response to a multi-modal input that includes input text and at least one input image, the second API call referring to the first identifier of the fine-tuned ML model and including the multi-modal input;
cause the multi-modal input to be applied to the fine-tuned ML model to generate the response; and
return the response including generated text that is generated by the fine-tuned ML model in response to the multi-modal input.

### Claim 22 (depends on 18)

The computing apparatus of claim 18, wherein:
the fine-tuned ML model includes an image embedding layer, the first model having fixed weighting coefficients, and a fine-tuned part having adjustable weighting coefficients, and
the first model lacks the image embedding layer and the first model cannot receive image inputs.

### Claim 23 (depends on 1)

The method of claim 1, wherein the API call is received by an API mediating communication between a user or an application and a server.

### Claim 24 (depends on 1)

The method of claim 1, wherein the first trainable matrix and the second trainable matrix together define a low-rank decomposition of an update to the pretrained weight matrix and wherein an output of the at least one layer is generated based on a combination of the pretrained weight matrix and the update to the pretrained weight matrix.

### Claim 25 (depends on 16)

The method of claim 16, wherein the fine-tuning method selected by the user includes a low-rank adaptation and wherein the fine-tuned ML model has been fine-tuned using the training set by performing supervised fine-tuning in which, for at least one layer of the specified first model having a pretrained weight matrix, pretrained weighting coefficients of the pretrained weight matrix are maintained constant while weighting coefficients of a first trainable matrix and a second trainable matrix of a low-rank adaptation layer are updated.

### Claim 26 (depends on 18)

The computing apparatus of claim 18, wherein:
the training set includes inputs associated with outputs, the inputs including the plurality of images and input text, and the outputs including labels for supervised learning as the respective text strings that are associated with the plurality of images, and
the first trainable matrix and the second trainable matrix together define a low-rank decomposition of an update to the pretrained weight matrix and wherein an output of the at least one layer is generated based on a combination of the pretrained weight matrix and the update to the pretrained weight matrix.
