US 2026/0087793 A1 · SYSTEM AND METHOD FOR MACHINE LEARNING MODELS FOR COMPUTER VISION ON DEVICES

Abstract

A system and method are provided for implementing transformer-based computer vision models on resource-constrained devices. An input image is divided into tokens, each corresponding to a patch. A background-aware vision transformer (BAViT) classifies tokens as foreground or background using a lightweight architecture without a class token and with a linear classifier for token-wise prediction. Training utilizes an accumulative cross entropy loss that aggregates token-level losses to improve accuracy. Tokens classified as background are pruned, thereby reducing computational complexity, runtime memory, and inference latency. Foreground tokens are processed in a downstream transformer-based object detection model, such as YOLOS, to generate detection outputs. The BAViT module operates as a pre-processing stage, facilitating integration with detection models without retraining. Configurations include BAViT-small with two transformer layers suitable for edge devices, supporting applications such as security and inventory tracking.

Claims20

Dependent claims

A method for computer vision on a device, the method comprising:

receiving an input image at a processor of the device;
dividing the input image into a plurality of non-overlapping patches, each patch representing an image token;
classifying each image token as either a token classified as foreground or a token classified as background by applying a background-aware vision transformer (BAViT), the BAViT comprising:
a vision transformer having at least two transformer layers without a class (CLS) token and configured to generate an embedding for each image token;
and
a linear output layer coupled to the vision transformer and configured to receive the embedding for each image token and output a classification score for foreground and background for that image token;
during training, computing an accumulative cross entropy loss function across the image tokens to train the background-aware vision transformer;
during inference, discarding at least a subset of image tokens classified as background by skipping computation for those image tokens in subsequent layers of a computer vision model;
and
processing the image tokens classified as foreground using an object detection transformer model to generate a detection output.

7 more elements

A device for performing computer vision, comprising:

a memory storing instructions;
and
a processor configured to execute the instructions to:
receive an input image;
partition the input image into a plurality of patches corresponding to image tokens;
classify each image token as a token classified as foreground or a token classified as background using a background-aware vision transformer (BAViT) having a transformer network without a class token and a linear classifier configured to output foreground/background scores for each image token;
identify image tokens classified as background as prunable tokens;
discard computation for image tokens classified as background prior to processing the image tokens in an object detection model;
and
process only the image tokens classified as foreground in a transformer-based object detection model to generate a detection output.

7 more elements

Dependencies are inferred from the claim text.

System and method for machine learning models for computer vision on devices

Life of this patent

System and method for machine learning models for computer vision on devices

Life of this patent

Abstract

Claims20