Generating segmentation masks for objects in digital videos using pose tracking data

A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• generating a down-sampled frame corresponding to a frame of a digital video;
• determining, using a global pose tracking neural network and from the down-sampled frame, a set of joint coordinates corresponding to a digital object portrayed in the frame of the digital video;
• generating, using the set of joint coordinates, a joint heat map corresponding to the digital object portrayed in the frame of the digital video, the joint heat map having values that distinguish between joint points corresponding to the set of joint coordinates and areas of the frame of the digital video that surround the joint points;
• determining, using one or more convolutional layers of a local segmentation neural network, visual features from the joint heat map and the frame of the digital video;
• generating, using at least one multi-perceptron mixer layer of the local segmentation neural network, mixer encodings that mix the visual features across feature channels by generating the mixer encodings from the visual features using layer normalization and an activation function of the at least one multi-perceptron mixer layer;
• and
• generating, using the local segmentation neural network and from the mixer encodings, a joint-based segmentation mask that corresponds to the digital object portrayed in the frame.

A system comprising:

• at least one memory device;
• and
• at least one server device configured to cause the system to:
• generate a down-sampled frame corresponding to a frame of a digital video;
• determine, utilizing a global pose tracking neural network and from the down-sampled frame, a set of joint coordinates corresponding to a digital object portrayed in the frame of the digital video;
• generate, using the set of joint coordinates, a joint heat map corresponding to the digital object portrayed in the frame of the digital video, the joint heat map having values that distinguish between joint points corresponding to the set of joint coordinates and areas of the frame of the digital video that surround the joint points;
• determine, utilizing an encoder of a segmentation neural network, mixer encodings for the digital object based on the joint heat map, the encoder comprising:
• a plurality of convolutional layers that determine a set of features using the joint heat map;
• and
• a multi-layer perceptron mixer layer that determines the mixer encodings by mixing the set of features using layer normalization and an activation function;
• and
• generate, utilizing a decoder of the segmentation neural network, a joint-based segmentation mask for the digital object portrayed in the frame of the digital video based on the mixer encodings.