Recognizing social groups and identifying order in queues from RGB images

A computer program product for training a neural network to cluster multiple objects in an image into groups and estimate an activity of each group of the groups, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• extracting a feature of each object of the multiple objects in the image;
• receiving, via the neural network, an arbitrary number of features of the multiple objects with arbitrary cardinality to handle an ordering of the multiple objects in the image, the neural network being set-based;
• inputting the feature of each object of the multiple objects to the neural network;
• estimating, by the neural network based on the inputting, an affinity matrix A and the activity of each group of the groups, the estimated affinity matrix A being a n×n symmetric matrix where n is a number of separate objects in the image and each element ai represents that i-th separate objects and j-th separate objects belong to a same group from among the groups;
• calculating a first loss between the estimated affinity matrix A and a ground truth affinity matrix for the image, and a second loss between the estimated activity of each group of the groups and a ground truth activity of each group of the groups for the image, the first loss being calculated using Maximum Spanning Trees (MST) that penalizes entries in the MST within a same cluster to produce a sparser affinity matrix A, wherein
• the sparser affinity matrix A identifies an order of members of a queue without order labels, by finding a path graph that maximizes a sum of edge weights defined by the estimated affinity matrix A,
• the path graph includes a dummy node added to all nodes in the path graph, the order being direction agnostic, and
• the MST recomputes batches of images;
• training the neural network based on the first loss and the second loss, wherein a term in the first loss trains in-cluster entries by assigning a value of greater than 0.5 and less than 1 to the in-cluster entries;
• generating, based on the training of the neural network, by an Advanced Driver Assistance System (ADAS) of a vehicle, a prediction of an action taken by the vehicle to control the vehicle to avoid potential collisions according to the estimated activity;
• and
• controlling, by the ADAS, the action on the vehicle based on the prediction.

A computer processing system for training a neural network to cluster multiple objects in an image into groups and estimate an activity of each group of the groups, the computer processing system comprising:

• a memory device for storing program code;
• and
• a processor device operatively coupled to the memory device for running the program code to:
• extract a feature of each object of the multiple objects in the image;
• receive, via the neural network, an arbitrary number of features of the multiple objects with arbitrary cardinality to handle an ordering of the multiple objects in the image, the neural network being set-based;
• input the feature of each object of the multiple objects to the neural network;
• estimate, by the neural network based on the inputting, an affinity matrix A and the activity of each group of the groups, the estimated affinity matrix A being a n×n symmetric matrix where n is a number of separate objects in the image and each element aij represents that i-th separate objects and j-th separate objects belong to a same group from among the groups;
• calculate a first loss between the estimated affinity matrix A and a ground truth affinity matrix for the image, and a second loss between the estimated activity of each group of the groups and a ground truth activity of each group of the groups for the image, the first loss being calculated using Maximum Spanning Trees (MST) that penalizes entries in the MST within a same cluster to produce a sparser affinity matrix A, wherein
• the sparser affinity matrix A identifies an order of members of a queue without order labels, by finding a path graph that maximizes a sum of edge weights defined by the estimated affinity matrix A,
• the path graph includes a dummy node added to all nodes in the path graph, the order being direction agnostic, and
• the MST recomputes batches of images;
• train the neural network based on the first loss and the second loss, wherein a term in the first loss trains in-cluster entries unassociated with any of the MST by assigning a value of greater than 0.5 and less than 1 to the in-cluster entries;
• generate, based on the training of the neural network, by an Advanced Driver Assistance System (ADAS) of a vehicle, a prediction of an action taken by the vehicle to control the vehicle to avoid potential collisions according to the estimated activity;
• and
• control, by the ADAS, the action on the vehicle based on the prediction.