US 2026/0017846 A1Published application

Method and system for multi-subject personalization with selective u-net influence

AssigneeTata Consultancy Services Limited

InventorsAshutosh RANJAN, Vivek SRIVASTAVA, Shirish Subhash KARANDE

Life of this patent

Estimating the term…

Loading patent…

Abstract

Recently, an uptick in the interest in providing interpretability to foundational models has been observed. However, the significant effort is limited to the large language models. Diffusion models have proven significant for the generative AI landscape and the interpretability of these models The present disclosure presents a novel selective U-Net influence (SelUT) technique for greater control and interpretability of DreamBooth for multi-subject personalization task. Specifically, the influence of the trained U-Net block(s) is controlled in the present disclosure during model inference. It provides a greater handle on interpreting the contribution of individual U-Net blocks across quality aspects such as identity disentanglement, image aesthetic, human preference, etc. Furthermore, we present an ensemble selection strategy to incorporate the dynamicity between base DreamBooth and the models trained with selective influence of U-Net block(s) which significantly improve the capability of DreamBooth for multi subject personalization.

Claims12

Dependent claims

A processor-implemented method, the method comprising:

receiving, via one or more hardware processors, an input comprising a plurality of subject images, a text description pertaining to each of the plurality of subject images and an associated text prompt explaining action to be performed by each of the plurality of subjects;
finetuning, via the one or more hardware processors, a trained diffusion model based on the received input using a fine-tuning technique;
generating, via the one or more hardware processors, an updated U-Net state dictionary by selecting a plurality of U-Net blocks with associated weights using a finetuned diffusion model, wherein the plurality of U-Net blocks comprises a plurality of down blocks, a plurality of mid blocks and a plurality of up blocks;
generating, via the one or more hardware processors, a plurality of influenced blocks from among of a plurality of blocks associated with the trained diffusion model by updating the associated weights based on the plurality of U-Net blocks, wherein the plurality of blocks are selected from a state dictionary associated with the trained diffusion model;
generating, via the one or more hardware processors, a plurality of images based on an influenced plurality of influenced blocks using the pretrained diffusion model;
and
identifying, via the one or more hardware processors, an optimal image from the plurality of generated images using an ensemble-based image selection technique.

4 more elements

A system comprising:

at least one memory storing programmed instructions;
one or more Input/Output (I/O) interfaces;
and one or more hardware processors operatively coupled to the at least one memory, wherein the one or more hardware processors are configured by the programmed instructions to:
receive an input comprising a plurality of subject images, a text description pertaining to each of the plurality of subject images and an associated text prompt explaining action to be performed by each of the plurality of subjects;
finetune a trained diffusion model based on the received input using a fine-tuning technique;
generate an updated U-Net state dictionary by selecting a plurality of U-Net blocks with associated weights using a finetuned diffusion model, wherein the plurality of U-Net blocks comprises a plurality of down blocks, a plurality of mid blocks and a plurality of up blocks;
generate a plurality of influenced blocks from among of a plurality of blocks associated with the trained diffusion model by updating the associated weights based on the plurality of U-Net blocks, wherein the plurality of blocks are selected from a state dictionary associated with the trained diffusion model;
generate a plurality of images based on an influenced plurality of influenced blocks using the pretrained diffusion model;
and
identify an optimal image from the plurality of generated images using an ensemble-based image selection technique.

7 more elements

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

receiving, an input comprising a plurality of subject images, a text description pertaining to each of the plurality of subject images and an associated text prompt explaining action to be performed by each of the plurality of subjects;
finetuning, via the one or more hardware processors, a trained diffusion model based on the received input using a fine-tuning technique;
generating, via the one or more hardware processors, an updated U-Net state dictionary by selecting a plurality of U-Net blocks with associated weights using a finetuned diffusion model, wherein the plurality of U-Net blocks comprises a plurality of down blocks, a plurality of mid blocks and a plurality of up blocks;
generating, via the one or more hardware processors, a plurality of influenced blocks from among of a plurality of blocks associated with the trained diffusion model by updating the associated weights based on the plurality of U-Net blocks, wherein the plurality of blocks are selected from a state dictionary associated with the trained diffusion model;
generating, via the one or more hardware processors, a plurality of images based on an influenced plurality of influenced blocks using the pretrained diffusion model;
and
identifying, via the one or more hardware processors, an optimal image from the plurality of generated images using an ensemble-based image selection technique.

4 more elements

Dependencies are inferred from the claim text.