# US 2025/0095395 A1 · Method for associating natural language with digital images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095395 A1
- Document ID: US20250095395A1
- Status: Published application
- Kind: A1
- Application number: 18/889,936
- Filed: 2024-09-19
- Published: 2025-03-20
- Earliest claimed: 2023-09-19
- Assignees: Alchemie Solutions, Inc.
- Applicants: Alchemie Solutions, Inc.
- Inventors: Julia English WINTER; Joseph ENGALAN; Sarah WEGWERTH; Alexa URREA; Katherine ROCHON; Jacob ZYSNARSKI
- Web page: https://patentgrep.com/patent/US20250095395A1

## CPC

- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G06V 10/82: using neural networks
- G09B 21/001: {Teaching or communicating with blind persons (G09B21/02 - G09B21/06 take precedence)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

A method for associating natural language with digital images is provided. The method includes steps of receiving a digitized image; identifying elements in the digitized image as identified elements; associating a contextual label to each element that becomes content for each element; identifying predetermined relationships between the identified elements; describing the content for each element individually and relationships between the elements with a predetermined language; engineering a prompt to be sent to a Language Model (LM); and receiving a response from the LM. Characteristically, the prompt is configured to provide an LM input and instruct the LL regarding a manner and configuration for responding to the LM input.

## Claims

### Claim 1

A method comprising:
receiving a digitized image;
identifying elements in the digitized image as identified elements;
associating a contextual label to each element that becomes content for each element;
identifying predetermined relationships between the identified elements;
describing the content for each element individually and relationships between the elements with a predetermined language;
engineering a prompt to be sent to a Language Model (LM), the prompt being configured to provide an LM input and instruct the LM regarding a manner and configuration for responding to the LM input; and
receiving a response from the LM.

### Claim 2 (depends on 1)

The method of claim 1, further comprising training a neural network to identify the elements in the digitized image, wherein the training comprises:
(a) collecting a dataset of annotated digitized images, wherein each image includes pre-labeled elements,
(b) applying data augmentation techniques to the collected dataset, including rotation, scaling, cropping, and noise addition, to improve the network's generalization capabilities;
(c) designing and implementing a convolutional neural network (CNN) architecture with multiple convolutional layers to detect spatial relationships between elements;
(d) using annotated images to train the neural network by adjusting network weights based on the identified elements and their relationships through forward and backward propagation; and
(e) evaluating the neural network's accuracy and performance using validation sets.

### Claim 3 (depends on 2)

The method of claim 2, wherein the pre-labeled elements include chemical structures, electrical components, or blueprint symbols.

### Claim 4 (depends on 2)

The method of claim 2, wherein the neural network's accuracy and performance are further evaluated using precision, recall, F1-score, and/or a confusion matrix to ensure reliable element recognition in digitized images.

### Claim 5 (depends on 1)

The method of claim 1, wherein the prompt is further refined by a plurality of links in the LM chain.

### Claim 6 (depends on 5)

The method of claim 5, wherein a custom knowledge base specifies phrases and keywords found and manipulated in a context of a predetermined goal.

### Claim 7 (depends on 1)

The method of claim 1, wherein the language model is a small language model or a large language model.

### Claim 8 (depends on 1)

The method of claim 1, wherein the response includes a detailed answer, guidance for a specific task, a detailed description of the digitized image, a detailed description of part of the digitized image, and/or alt text.

### Claim 9 (depends on 1)

The method of claim 1, wherein the response includes an LM preamble, an LM response template, and textual descriptions that are sent to a Language Model Chain as the prompt.

### Claim 10 (depends on 1)

The method of claim 1, wherein the elements are identified by a trained machine learning algorithm.

### Claim 11 (depends on 10)

The method of claim 10, wherein the elements are identified by a trained neural network.

### Claim 12 (depends on 1)

The method of claim 1, wherein the predetermined language is created by subject matter experts (SME).

### Claim 13 (depends on 1)

The method of claim 1, wherein subject matter experts train a rules-based or AI system that provides contextual labels and expected relationships in a subject area.

### Claim 14 (depends on 1)

The method of claim 1, wherein a rubric is provided by subject matter experts to filter content and relationships.

### Claim 15 (depends on 1)

The method of claim 1, wherein the relationships are based on proximity of the elements in the digitized image.

### Claim 16 (depends on 1)

The method of claim 1, wherein the digitized image is a raster image.

### Claim 17 (depends on 1)

The method of claim 1, wherein the digitized image is a vector image.

### Claim 18 (depends on 1)

The method of claim 1, wherein the digitized image is a 3D image.

### Claim 19 (depends on 1)

The method of claim 1, wherein the predetermined language is specific for a category of the digitized image.

### Claim 20 (depends on 19)

The method of claim 19, wherein the digitized image is an image of a chemical compound or chemical reaction.

### Claim 21 (depends on 20)

The method of claim 20, wherein the elements include representations of atoms and chemical bonds.

### Claim 22 (depends on 21)

The method of claim 21, wherein the elements further include representations of electron movement.

### Claim 23 (depends on 19)

The method of claim 19, wherein the digitized image is an image for electrical diagram analysis.

### Claim 24 (depends on 19)

The method of claim 19, wherein the digitized image is an image for blueprint analysis.

### Claim 25 (depends on 19)

The method of claim 19, wherein the digitized image is an image for a plumbing system.

### Claim 26 (depends on 19)

The method of claim 19, wherein the digitized image is an image for a plumbing system a mechanical system.

### Claim 27 (depends on 19)

The method of claim 19, the digitized image is an image for building standards.

### Claim 28 (depends on 1)

The method of claim 1, wherein one or more steps are executed by a computer.

### Claim 29

A method for providing interactive STEM education tools accessible to blind and low-vision individuals, comprising:
generating, in real-time, alt text descriptions for diagrams based on configuration data from a STEM interactive tool, wherein said alt text descriptions comprise a contextual overview, component details, and relationships between components;
allowing user interaction with the tool through a form-driven, keyboard-accessible control interface, enabling the selection and manipulation of components in the diagram without the use of a mouse;
providing an artificial intelligence-driven learning assistant, configured to answer user queries based on the alt text descriptions, wherein the learning assistant offers personalized responses to clarify and guide the user in understanding and manipulating the diagram; and
embedding said alt text descriptions into the metadata of an exportable image generated from the STEM interactive tool, wherein said image file retains its accessibility for future use.

### Claim 30 (depends on 29)

The method of claim 29, wherein the alt text generation engine applies a drill-down organization method to create the descriptions, wherein the overview provides a high-level summary, and additional layers of detail are available upon user request.

### Claim 31 (depends on 29)

The method of claim 29, wherein the control interface includes drop-down menus, buttons, and selection options corresponding to diagram components, configured to present real-time updates and feedback as the user interacts with the tool.

### Claim 32

A system for generating alt text for interactive STEM diagrams for blind and low-vision individuals, comprising:
an alt text generation engine configured to receive configuration data from a digital interactive system, wherein the engine automatically generates alt text descriptions based on the diagram's components, said alt text following a structured format of overview, detailed description, and component relationships;
a user interface, operable via keyboard controls, allowing the user to add, modify, and review components of the diagram, wherein feedback is provided through dynamically updated alt text that describes actions performed by the user; and
an artificial intelligence assistant integrated with the alt text generation engine, wherein the assistant responds to user inquiries, providing detailed or specific information about the diagram based on the alt text data.

### Claim 33 (depends on 2)

The system of claim 2, further comprising a rubric-based feedback system, wherein the artificial intelligence assistant provides guided suggestions to help users improve their construction of the diagram, based on specific learning objectives.
