Deep learning apparatus and method for segmentation and survival prediction for head and neck tumors

A system for prognosis of head and neck cancer, comprising:

• processing circuitry configured to
• project clinical data of a patient into a first feature vector,
• project three-dimensional (3D) multimodal images of a head and neck area of the patient into a second feature vector, including:
• flattening the 3D multimodal images into a sequence of flattened two-dimensional (2D) patches, and
• projecting the flattened 2D patches into the second feature vector,
• generate a third feature vector based on the first feature vector and the second feature vector by:
• concatenating the first feature vector and the second feature vector into a concatenated feature vector, and
• generating the third feature vector based on positional encoding being added to the concatenated feature vector,
• encode the third feature vector using a transformer-based encoder that encodes the third feature vector based on a self-attention mechanism, and
• calculate a risk score of prognosis of head and neck cancer for the patient based on the encoded third feature vector.