Back to results
US 12,718,793 B2Granted
Sonifying visual content for vision-impaired users
AssigneeGoogle Inc.
InventorsRamprasad Sedouram, Jaunani Sriramachandran
Life of this patent
Estimating the term…
Abstract
A method includes receiving, for presentation to a user of a user device, image data representing an image. The method also includes generating, using a textual story generative model, based on the image data, a textual story for the image, and generating, based on the textual story for the image, textual story audio data representing the textual story for the image. The method further includes providing, for audible output from the user device, the textual story audio data.
Claims16
A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
- receiving, for presentation to a user of a user device, image data representing an image;
- obtaining one or more personal factors associated with the user, the one or more personal factors comprising an elaborateness preference for textual stories and soundscapes;
- generating, using a textual story generative model, based on the image data and the one or more personal factors, a textual story for the image;
- generating, based on the textual story for the image, textual story audio data representing the textual story for the image;
- generating, based on the image data, a soundscape for the image by:
- processing, using a visual feature identification model, the image data to generate a description of the image;
- identifying, from a repository of images each paired with corresponding audio representing a visual environment of the image and a corresponding description of the image, one or more of the images that are paired with corresponding descriptions that match the description of the image;
- and
- generating, using a soundscape generative model, based on the corresponding audio representing the visual environment of the identified one or more images, the soundscape audio data;
- generating, based on the soundscape for the image, soundscape audio data representing the soundscape for the image;
- and
- providing, for audible output from the user device, the textual story audio data and the soundscape audio data.
9 more elements
A system comprising:
- data processing hardware;
- and
- memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
- receiving, for presentation to a user of a user device, image data representing an image;
- obtaining one or more personal factors associated with the user, the one or more personal factors comprising an elaborateness preference for textual stories and soundscapes;
- generating, using a textual story generative model, based on the image data and the one or more personal factors, a textual story for the image;
- generating, based on the textual story for the image, textual story audio data representing the textual story for the image;
- generating, based on the image data, a soundscape for the image by:
- processing, using a visual feature identification model, the image data to generate a description of the image;
- identifying, from a repository of images each paired with corresponding audio representing a visual environment of the image and a corresponding description of the image, one or more of the images that are paired with corresponding descriptions that match the description of the image;
- and
- generating, using a soundscape generative model, based on the corresponding audio representing the visual environment of the identified one or more images, the soundscape audio data;
- generating, based on the soundscape for the image, soundscape audio data representing the soundscape for the image;
- and
- providing, for audible output from the user device, the textual story audio data and the soundscape audio data.
12 more elements
Dependencies are inferred from the claim text.