# US 12,008,821 B2 · Machine-learned models for unsupervised image transformation and retrieval > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,821 B2 - Document ID: US12008821B2 - Status: Granted - Kind: B2 - Application number: 17/314,738 - Filed: 2021-05-07 - Published: 2022-11-24 - Granted: 2024-06-11 - Earliest claimed: 2021-05-07 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Wen-Sheng Chu; Abhishek Kumar; Min Jin Chong - Web page: https://patentgrep.com/patent/US12008821B2 ## CPC - G06V 20/64: Three-dimensional [3D] objects - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/23: Clustering techniques - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/094: Adversarial learning - G06N 20/00: Machine learning - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06T 7/73: using feature-based methods - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods of the present disclosure are directed to a computer-implemented method. The method can include obtaining a first image depicting a first object and a second image depicting a second object, wherein the first object comprises a first feature set and the second object comprises a second feature set. The method can include processing the first image with a machine-learned image transformation model comprising a plurality of model channels to obtain a first channel mapping indicative of a mapping between the plurality of model channels and the first feature set. The method can include processing the second image with the model to obtain a second channel mapping indicative of a mapping between the plurality of model channels and the second feature set. The method can include generating an interpolation vector for a selected feature. ## Claims ### Claim 1 A computer-implemented method for generation of interpolation vectors for image transformation, comprising: obtaining, by a computing system comprising one or more computing devices, a first image depicting a first object of an object type and a second image depicting a second object of the object type, wherein objects of the object type comprise a plurality of object features, and wherein the first object comprises a first feature set for the plurality of object features and the second object comprises a second feature set for the plurality of object features different than the first feature set; processing, by the computing system, the first image with a machine-learned image transformation model comprising a plurality of model channels to obtain a first channel mapping indicative of a mapping between the plurality of model channels and the first feature set for the plurality of object features; processing, by the computing system, the second image with the machine-learned image transformation model to obtain a second channel mapping indicative of a mapping between the plurality of model channels and the second feature set for the plurality of object features; and generating, by the computing system based at least in part on the first channel mapping and the second channel mapping, an interpolation vector for a selected feature of the plurality of object features, wherein the interpolation vector is configured to interpolate the selected feature of the plurality of object features from the first feature set for the plurality of object features to the second feature set for the plurality of object features; and processing, by the computing stem, the first image, the second image and the interpolation vector with the machine-learned image transformation model to obtain an interpolated image, wherein the interpolated image depicts a third object of the object type comprising a third feature set for the plurality of object features, wherein the third feature set comprises the selected feature from the first feature set and one or more second features for the second feature set for the plurality of object features. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein: the interpolation vector is indicative of a similarity between the selected feature from the first object and the selected feature from the second object. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the method further comprises: obtaining, by the computing system, a plurality of additional images, wherein the plurality of additional images depict a respective plurality of objects of the object type, wherein each of the respective plurality of objects comprises a feature set for the plurality of object features; and for each of the plurality of additional images: processing, by the computing system, a respective additional image of the plurality of additional images with the machine-learned image transformation model to obtain a respective channel mapping indicative of a mapping between the plurality of model channels and the feature set of the respective additional image for the plurality of object features; and generating, by the computing system based at least in part on the first channel mapping and the respective channel mapping, an additional interpolation vector for the selected feature of the plurality of object features. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the method further comprises: clustering, by the computing system, the first image, the second image, and the plurality of additional images based at least in part on the interpolation vector and each of the additional interpolation vectors to obtain one or more submembership clusters respectively associated with one or more aspects of the selected feature. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the selected feature comprises one or more selected features selected from the plurality of object features. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the object type comprises a face, and wherein the plurality of object features comprise one or more of: one or more eye features; one or more mouth features; one or more nose features; one or more features for a region comprising a portion of the face; one or more hair features; one or more pose features; and one or more head shape features. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein: the selected feature comprises the one or more pose features, the plurality of object features comprises the one or more nose features; and wherein a third object comprises the one or more pose features from the first object and the one or more nose features from the second object. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the object type comprises a room, and wherein the plurality of object features comprises a plurality of furniture objects. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the object type comprises a human body, and wherein the plurality of object features comprises one or more of: one or more clothing items; one or more body shape features; one or more facial features; or one or more body pose features. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the machine-learned image transformation model comprises an encoding portion, a decoding portion, and a discriminator portion. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein: the encoding portion of the machine-learned image transformation model is configured to encode an image; and wherein processing the first image with the machine-learned image transformation model comprises processing, by the computing system, the first image with the encoding portion of the machine-learned image transformation model to obtain an image encoding and the first channel mapping indicative of the mapping between the plurality of model channels and the first feature set for the plurality of object features. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein the machine-learned image transformation model comprises one or more neural networks pre-trained for image transformation tasks. ### Claim 13 A computing system for generation of interpolation vectors for image transformation and retrieval tasks, comprising: one or more processors; a machine-learned image transformation model trained to perform image transformation tasks, wherein the machine-learned image transformation model comprises an encoding portion, a decoding portion, and a plurality of model channels; one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising: obtaining a first image depicting a first object of an object type and a second image depicting a second object of the object type, wherein objects of the object type comprise a plurality of object features, and wherein the first object comprises a first feature set for the plurality of object features and the second object comprises a second feature set for the plurality of object features different than the first feature set; processing the first image with the encoding portion of the machine-learned image transformation model to obtain a first channel mapping indicative of a mapping between the plurality of model channels and the first feature set for the plurality of object features; processing the second image with the encoding portion of the machine-learned image transformation model to obtain a second channel mapping indicative of a mapping between the plurality of model channels and the second feature set for the plurality of object features; generating, based at least in part on the first channel mapping and the second channel mapping, an interpolation vector for a selected feature of the plurality of object features, wherein the interpolation vector is configured to interpolate the selected feature of the plurality of object features from the first feature set for the plurality of object features to the second feature set for the plurality of object features; and processing the first image, the second image, and the interpolation vector with the machine-learned image transformation model to obtain an interpolated image, wherein the interpolated image depicts a third object of the object type comprising a third feature set for the plurality of object features, wherein the third feature set comprises the selected feature from the first feature set and one or more second features for the second feature set for the plurality of object features. ### Claim 14 (depends on 13) The computing system of claim 13, wherein: the interpolation vector is indicative of a similarity between the selected feature from the first object and the selected feature from the second object; and wherein the operations further comprise: obtaining a plurality of additional images, wherein each of the plurality of additional images depicts an object of a respective plurality of objects of the object type, wherein each of the plurality of additional images comprises a plurality of channels; and for each of the plurality of additional images: processing a respective additional image of the plurality of additional images with the machine-learned image transformation model to obtain a respective channel mapping indicative of a mapping between the plurality of model channels and the feature set of the respective additional image for the plurality of object features; and generating, based at least in part on the first channel mapping and the respective channel mapping, an additional interpolation vector for the selected feature of the plurality of object features. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the operations further comprise: clustering the first image, the second image, and the plurality of additional images based at least in part on the interpolation vector and each of the additional interpolation vectors to obtain one or more submembership clusters respectively associated with one or more aspects of the selected feature. ### Claim 16 (depends on 13) The computing system of claim 13, wherein the object type comprises a face, and wherein the plurality of object features comprise one or more of: one or more eye features; one or more mouth features; one or more nose features; one or more features for a region comprising a portion of the face; one or more hair features; one or more pose features; and one or more head shape features. ### Claim 17 A computer-implemented method for generation of interpolation vectors for image transformation and retrieval tasks, comprising: obtaining, by a computing system comprising one or more computing devices, a first image depicting a first object of an object type and a second image depicting a second object of the object type, wherein objects of the object type comprise a plurality of object features, and wherein the first object comprises a first feature set for the plurality of object features and the second object comprises a second feature set for the plurality of object features different than the first feature set; processing, by the computing system, the first image with a machine-learned image transformation model comprising a plurality of model channels to obtain a first channel mapping indicative of a mapping between the plurality of model channels and the first feature set for the plurality of object features; processing, by the computing system, the second image with the machine-learned image transformation model to obtain a second channel mapping indicative of a mapping between the plurality of model channels and the second feature set for the plurality of object features; generating, by the computing system based at least in part on the first channel mapping and the second channel mapping, an interpolation vector for a selected feature of the plurality of object features; obtaining; by the computing system, a plurality of additional images, wherein the plurality of additional images depict a respective plurality of objects of the object type, wherein each of the respective plurality of objects comprises a feature set for the plurality of object features; and for each of the plurality of additional images: processing, by the computing system, a respective additional image of the plurality of additional images with the machine-learned image transformation model to obtain a respective channel mapping indicative of a mapping between the plurality of model channels and the feature set of the respective additional image for the plurality of object features; and generating, by the computing system based at least in part on the first channel mapping and the respective channel mapping, an additional interpolation vector for the selected feature of the plurality of object features. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, wherein the method further comprises: clustering, by the computing system, the first image, the second image, and the plurality of additional images based at least in part on the interpolation vector and each of the additional interpolation vectors to obtain one or more submembership clusters respectively associated with one or more aspects of the selected feature.