Automatic digital content captioning using spatial relationships method and apparatus

A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:

• obtaining a digital content item;
• analyzing the digital content item to detect a plurality of objects depicted in the digital content item, the analysis comprising determining a bounding box for each object of the plurality;
• determining, for each object of the plurality, an n-dimensional appearance feature vector for the object, the appearance feature vector determination comprising analyzing content of the digital content item to generate the n-dimensional appearance feature vector;
• determining, for each object of the plurality, a set of geometry features for the object using the bounding box determined for the object;
• generating, using a trained transformer machine model, encoded output using an encoder of the trained transformer, the n-dimensional appearance feature vector and the set of geometry features determined for each object of the plurality, the encoded output generation comprising the encoder using spatial relationships among the plurality of objects identified using the set of geometry features determined for each object of the plurality;
• and
• automatically creating, using the trained transformer machine model, a caption comprising a sequence of words, the automatic caption creation comprising using a decoder component of the trained transformer to decode the encoded output to identify the sequence of words.

A computing device comprising:

• a processor;
• a non-transitory storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:
• obtaining logic executed by the processor for obtaining a digital content item;
• analyzing logic executed by the processor for analyzing the digital content item to detect a plurality of objects depicted in the digital content item, the analysis comprising determining a bounding box for each object of the plurality;
• determining logic executed by the processor for determining, for each object of the plurality, an n-dimensional appearance feature vector for the object, the appearance feature vector determination comprising analyzing content of the digital content item to generate the n-dimensional appearance feature vector;
• determining logic executed by the processor for determining, for each object of the plurality, a set of geometry features for the object using the bounding box determined for the object;
• generating logic executed by the processor for generating, using a trained transformer machine model, encoded output using an encoder of the trained transformer, the n-dimensional appearance feature vector and the set of geometry features determined for each object of the plurality, the encoded output generation comprising the encoder using spatial relationships among the plurality of objects identified using the set of geometry features determined for each object of the plurality;
• and
• creating logic executed by the processor for automatically creating, using the trained transformer machine model, a caption comprising a sequence of words, the automatic caption creation comprising using a decoder component of the trained transformer to decode the encoded output to identify the sequence of words.