System and method for efficiently amalgamated cnn-transformer architecture for mobile vision applications

A non-transitory computer readable storage medium storing computer program instructions, which when executed in an edge computing system, performs a method of object detection, wherein the edge computing system comprises processing circuitry including:

• at least one multi-code graphics processing unit (GPU) having GPU memory, and
• a multi-core central processing unit (CPU) connected to random access memory (RAM),
• wherein the processing circuitry is configured with a hybrid convolutional neural network (CNN) and vision transformer backbone network in an object detection neural network, the method comprising:
• inputting an image:
• in a first stage having a convolutional encoder, extracting local features from the input image:
• in following stages each having a split depth-wise transpose attention (SDTA) encoder, performing multi-headed self-attention by applying a dot product operation across channel dimensions in order to compute cross-covariance across channels to generate attention feature maps;
• and
• producing, in the object detection neural network, a fixed-size collection of bounding boxes and scores for a presence of object class instances in those boxes, followed by
• producing, by a non-maximum suppression layer, final object detections.

A method of object detection, with an edge computing system that comprises processing circuitry including:

• at least one multi-code graphics processing unit (GPU) having GPU memory, and
• a multi-core central processing unit (CPU) connected to random access memory (RAM),
• wherein the processing circuitry is configured with a hybrid convolutional neural network (CNN) and vision transformer backbone network in an object detection neural network, the method comprising:
• inputting an image;
• in a first stage of the backbone network having a convolutional encoder, extracting local features from the input image;
• in following stages of the backbone network, each following stage having a split depth-wise transpose attention (SDTA) encoder, performing multi-headed self-attention by applying a dot product operation across channel dimensions in order to compute cross-covariance across channels to generate attention feature maps;
• and
• producing, in the object detection neural network, a fixed-size collection of bounding boxes and scores for a presence of object class instances in those boxes, followed by
• producing, by a non-maximum suppression layer, final object detections.