Visual prompting methods for GPT-4V based zero-shot graphic layout design generation
AssigneeFractal Analytics, Inc.
InventorsKunal Singh, Mukund Khanna, Ankan Biswas, Pradeep Moturi, Shivam
Life of this patent
Estimating the term…
Abstract
GPT-4-Vision (GPT-4V) large multimodal models (LMMs) may be used to do zero-shot graphic layout design generation in a versatile manner. Segmentation/superpixel methods may be used to identify and mark the key regions to visually augment the image to enhance GPT-4V's spatial reasoning capability. The results demonstrate the efficacy of these visual prompting methods, showing improvement over standard GPT-4V prompting methods and also performing at par and even better, for some techniques, when compared to the LayoutDetr model.
Claims20
A method for zero-shot graphic layout design generation using a Generative AI large multimodal model (LMM), the method comprising:
- using one or more segmentation strategies, dividing at least one image into a plurality of semantically or spatially distinct regions;
- using the one or more segmentation strategies, placing at least one visual marker on the plurality of semantically or spatially distinct regions that explicitly denote the plurality of semantically or spatially distinct regions;
- passing the at least one image that has been divided and visually marked and a prompt to the Generative AI LMM;
- and
- using pretrained, vision-language understanding in the Generative AI LMM, predicting coordinates by referring to the at least one visual marker, associating the at least one visual marker with corresponding ones of the plurality of semantically or spatially distinct regions providing a bounding box to predict placement of primary text on the image, wherein the bounding box is applied to a final image to place primary text on the final image.
A method for zero-shot graphic layout design generation using a Generative AI large multimodal model (LMM), the method comprising:
- using one or more segmentation strategies, dividing at least one image into a plurality of semantically or spatially distinct regions, the at least one image including a plurality of text elements;
- using the one or more segmentation strategies, placing at least one visual marker on the plurality of semantically or spatially distinct regions that explicitly denote the plurality of semantically or spatially distinct regions;
- passing the at least one image that has been divided and visually marked and a prompt to use pretrained, vision-language understanding in the Generative AI LMM to obtain bounding box coordinates for a position identified as best for placement of text within the at least one image based on positioning of other elements present in the at least one image and the prompt;
- and
- using pretrained, vision-language understanding in the Generative AI LMM, predicting coordinates by referring to the at least one visual marker, associating the at least one visual marker with corresponding ones of the plurality of semantically or spatially distinct regions providing a bounding box to predict placement of primary text on the image, wherein the bounding box is applied to a final image to place primary text on the final image.
Dependencies are inferred from the claim text.