Face-based frame packing for video calls

A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:

• capturing a video frame that includes a face of a person;
• detecting the face in the video frame;
• segmenting the video frame into a plurality of rectangles, the plurality of rectangles including a face rectangle with pixels corresponding to the face;
• packing the video frame based on the plurality of rectangles, wherein a greater number of pixels in the packed video frame are allocated to the face rectangle as compared to other rectangles of the plurality of rectangles;
• and
• encoding the packed video frame, wherein the encoded video frame includes metadata descriptive of the packing.

A system comprising:

• one or more processors;
• and
• a memory that stores instructions that, when executed by the one or more processors cause the one or more processors to perform operations comprising:
• capturing a video frame that includes a face of a person;
• detecting the face in the video frame;
• segmenting the video frame into a plurality of rectangles, the plurality of rectangles including a face rectangle with pixels corresponding to the face;
• packing the video frame based on the plurality of rectangles, wherein a greater number of pixels in the packed video frame are allocated to the face rectangle as compared to other rectangles of the plurality of rectangles;
• and
• encoding the packed video frame, wherein the encoded video frame includes metadata descriptive of the packing.