# US 2026/0227951 A1 · Photo and image screen reader for the blind

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0227951 A1
- Document ID: US20260227951A1
- Status: Published application
- Kind: A1
- Application number: 19/047,274
- Filed: 2025-02-06
- Published: 2026-08-06
- Earliest claimed: 2025-02-06
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Gregory Ward RATAJIK
- Web page: https://patentgrep.com/patent/US20260227951A1

## CPC

- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 10/235: {based on user input or interaction}
- G06V 10/764: using classification, e.g. of video objects
- G09B 21/001: {Teaching or communicating with blind persons (G09B21/02 - G09B21/06 take precedence)}
- G06V 2201/10: Recognition assisted with metadata
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Techniques for assisting visually impaired individuals when working with images are disclosed. A service accesses an image of a scene. The image includes pixels representing an object included in the scene. The service generates a classification for the object pixels and a classification for the scene. The service receives user input directed to the image. The user input includes at least one of: a cursor hovering over one or more of the object pixels, a selection of the one or more object pixels, or a movement of the cursor over the one or more object pixels. In response to the user input, the service triggers playback of an audio output comprising audio details describing the object classification.

## Claims

### Claim 1

A computer system comprising:
one or more processors; and
one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
access an image of a scene, wherein the image includes pixels representing an object included in the scene;
generate, via a machine learning engine, a first classification for the pixels representing the object, wherein first metadata for the object is structured to include the first classification;
generate, via the machine learning engine, a second classification for the scene, as represented in the image, wherein second metadata for the image is structured to include the second classification;
receive user input directed to the image, the user input comprising at least one of:
a cursor hovering over one or more of the pixels representing the object in the scene, a selection of the one or more pixels representing the object, or a movement of the cursor over the one or more pixels representing the object; and
in response to the user input, trigger playback of an audio output comprising audio details describing the first classification for the object.

### Claim 2 (depends on 1)

The computer system of claim 1, wherein the user input is the cursor hovering over the one or more of the pixels representing the object in the scene.

### Claim 3 (depends on 1)

The computer system of claim 1, wherein the user input is the selection of the one or more pixels representing the object.

### Claim 4 (depends on 1)

The computer system of claim 1, wherein the user input is the movement of the cursor over the one or more pixels representing the object.

### Claim 5 (depends on 1)

The computer system of claim 1, wherein the audio output is generated by a large language model (LLM).

### Claim 6 (depends on 1)

The computer system of claim 1, wherein the first classification includes a determined type for the object.

### Claim 7 (depends on 1)

The computer system of claim 1, wherein the first metadata for the object further includes location data for the object.

### Claim 8 (depends on 7)

The computer system of claim 7, wherein the location data include image pixel coordinates for the pixels relative to the image.

### Claim 9 (depends on 7)

The computer system of claim 7, wherein the location data includes proximity information of the object represented by the image relative to another object that is also represented in the image.

### Claim 10 (depends on 1)

The computer system of claim 1, wherein the second metadata for the image includes descriptions of multiple different groups of objects, where one group includes said object.

### Claim 11

A method comprising:
accessing an image of a scene, wherein the image includes pixels representing an object included in the scene;
generating a first classification for the pixels representing the object, wherein first metadata for the object is structured to include the first classification;
generating a second classification for the scene, as represented in the image, wherein second metadata for the image is structured to include the second classification;
receiving user input directed to the image, the user input comprising at least one of: a cursor hovering over one or more of the pixels representing the object in the scene, a selection of the one or more pixels representing the object, or a movement of the cursor over the one or more pixels representing the object; and
in response to the user input, triggering playback of an audio output comprising audio details describing the first classification for the object.

### Claim 12 (depends on 11)

The method of claim 11, wherein the audio details further include location information for the object.

### Claim 13 (depends on 12)

The method of claim 12, wherein the location information includes a proximity of the object relative to a second object that is also represented in the image and that is included in the scene.

### Claim 14 (depends on 11)

The method of claim 11, wherein the object is a product, and wherein the first classification includes product details for the product, at least some of the product details being identified from recognizable package details for the object, as identified from within the image.

### Claim 15 (depends on 11)

The method of claim 11, wherein generating the first classification is performed via object segmentation of the image.

### Claim 16

One or more hardware storage devices that store instructions that are executable by one or more processors to cause the one or more processors to:
access an image of a scene, wherein the image includes pixels representing an object included in the scene;
generate a first classification for the pixels representing the object, wherein first metadata for the object is structured to include the first classification;
generate a second classification for the scene, as represented in the image, wherein second metadata for the image is structured to include the second classification;
receive user input directed to the image, the user input comprising at least one of: a cursor hovering over one or more of the pixels representing the object in the scene, a selection of the one or more pixels representing the object, or a movement of the cursor over the one or more pixels representing the object; and
in response to the user input, trigger playback of an audio output comprising audio details describing the first classification for the object.

### Claim 17 (depends on 16)

The one or more hardware storage devices of claim 16, wherein the second classification for the scene includes a description of multiple different objects that are included in the image and that are included in the scene, the multiple different objects including said object, and wherein the second classification includes the second classification including relative proximity information for the multiple different objects.

### Claim 18 (depends on 16)

The one or more hardware storage devices of claim 16, wherein the image includes second pixels representing a second object in the scene, and wherein the instructions are further executable to cause the one or more processors to:
generate a third classification for the second pixels representing the second object in the scene, wherein third metadata for the second object is structured to include the third classification;
wherein the audio details further include location information describing a proximity of the object relative to the second object in the scene.

### Claim 19 (depends on 18)

The one or more hardware storage devices of claim 18, wherein the instructions are further executable to cause the one or more processors to:
receive second user input, the second user input comprising the cursor hovering over one or more of the second pixels representing the second object; and
in response to the second user input, triggering playback of a second audio output comprising second audio details describing the second object, wherein the second audio details includes information describing the proximity of the object relative to the second object in the scene.

### Claim 20 (depends on 16)

The one or more hardware storage devices of claim 16, wherein a large language model (LLM) generates the first classification, the second classification, and the audio output.
