Back to results
US 9,754,188 B2Granted
Tagging personal photos with deep networks
AssigneeMicrosoft Technology Licensing, LLC
InventorsTao Mei, Jianlong Fu, Kuiyuan Yang, Yong Rui
Life of this patent
Estimating the term…
Abstract
Techniques and constructs to facilitate automatic tagging can provide improvements in image storage and searching. The constructs may enable training a deep network using tagged source images and target images. The constructs may also train a top layer of the deep network using a personal photo ontology. The constructs also may select one or more concepts from the ontology for tagging personal digital images.
Claims20
A method comprising:
- providing an ontology for personal digital images, the ontology comprising a hierarchy including a plurality of concepts grouped into a plurality of nodes of the hierarchy, the plurality of concepts comprising a textual descriptor for association with one or more personal digital images;
- providing a plurality of source digital images from a source domain, the source digital images having at least one associated textual descriptor;
- providing a plurality of target digital images from a target domain;
- training, on a computing device, a deep network using the source digital images, the at least one associated textual descriptor, the target digital images, and the ontology;
- receiving, at the computing device, a plurality of personal digital images, the plurality of personal digital images having one or more attributes;
- applying, using the computing device, the deep network to the plurality of personal digital images to determine one or more of the concepts for association with the plurality of personal digital images;
- and
- commonly associating at least one of the one or more of the concepts to a subset of the plurality of personal digital images, the subset of the plurality of personal digital images having the one or more attributes that is within a predetermined threshold of each other.
5 more elements
A device comprising:
- one or more computer-readable media having thereon a plurality of modules, a source image repository, a target image repository, and an ontology repository, the ontology repository storing an ontology comprising a hierarchy including textual descriptors grouped into a plurality of nodes;
- and
- a processing unit operably coupled to the computer-readable media, the processing unit adapted to execute modules comprising:
- a convolutional autoencoder training module executable to train one or more convolutional autoencoders comprising a deep network using source images from the source image repository and target images from the target image repository;
- a top level training module executable to train a top level of the deep network using the ontology from the ontology repository;
- a scoring module for determining, for a first digital image, confidence scores for a plurality of the textual descriptors using the deep network and metadata associated with the first digital image;
- and
- a tagging module for selecting, based at least in part on the confidence scores, one or more of the plurality of textual descriptors for tagging the first digital image, and for tagging the first digital image and a second digital image commonly with the selected one or more of the plurality of textual descriptors, the second digital image having an associated metadata being within a predetermined threshold of the metadata associated with the first digital image.
5 more elements
A system comprising:
- one or more processors;
- and
- memory storing a plurality of computer-executable instructions, the computer-executable instructions, upon execution, configure the system to perform operations comprising:
- training a deep network comprising a stack of convolutional autoencoders in an unsupervised manner using tagged source images from a source image repository and target images from a target image repository to discover deep feature abstractions from raw pixels of the source images and the target images;
- fine-tuning the deep feature abstractions in a supervised manner using tags from the tagged source image data to obtain finely-tuned deep feature abstractions;
- training a fully connected top layer of the deep network using a personal photo ontology, the personal photo ontology comprising a hierarchy including textual descriptors grouped into a plurality of nodes;
- determining, for a plurality of digital images to be tagged, confidence scores for one or more of the textual descriptors using the deep network, the plurality of digital images having metadata;
- and
- based at least in part on the confidence scores, tagging the plurality of digital images with at least one of the textual descriptors,
- wherein individual ones of the plurality of digital images having metadata that are within a predetermined threshold of each other are tagged with at least one common textual descriptor.
7 more elements
Dependencies are inferred from the claim text.