Back to results
US 12,738,092 B2Granted
Learning domain and pose invariance for thermal-to-visible face recognition
AssigneeNUtech Ventures
InventorsBenjamin S. Riggan, Cedric Nimpa Fondje
Life of this patent
Estimating the term…
Abstract
A method and system for cross-spectrum face recognition is disclosed. The system for off-pose cross-spectrum to frontal visible face matching, which used modified base architectures to extract image representations along with a new sub-network to simultaneously learn pose and domain invariance using new joint-loss function that combines proposed pose-correction and cross-spectrum losses. The system and method demonstrate significant improvements over state-of-the-art domain adaptation methods not only on pose conditions, but also on baseline and varying expressions. Other aspects, embodiments, and features are also claimed and described.
Claims19
A method for cross-spectrum face recognition, comprising:
- receiving a visible image of a person;
- receiving a cross-spectrum image of the person;
- training a neural network based on both the visible image and the cross-spectrum image by:
- extracting a first representation from the visible image using a modified neural network architecture;
- extracting a second representation from the cross-spectrum image based on the modified neural network architecture;
- converting the second representation to a modified representation using a domain adaptation sub-network, the domain adaptation sub-network comprising:
- a first residual block and a second residual block being immediately subsequent to the first block, the first residual block having the second representation as an input, the second residual block having the second representation and an output of the first residual block as an input, the modified representation being based on the second representation, the output of the first block, and an output of the second block;
- minimizing a distance between the first representation and the modified representation using a pose-correction loss function, wherein the pose-correction loss function is defined as Lp=Σ∥Vi-{circumflex over (T)}i∥2, where V is the first representation, and {circumflex over (T)} is the modified representation;
- and
- updating parameters of the neural network based on the modified representation.
7 more elements
A method for cross-spectrum face recognition, comprising:
- receiving a cross-spectrum image of a person among people in a visible imagery;
- providing the cross-spectrum image to a trained neural network, the trained neural network having been trained using both a first representation based on visual training images and a modified representation based on cross-spectrum training images by minimizing a distance between the first representation and the modified representation using a pose-correction loss function, wherein the pose-correction loss function is defined as Lp=Σ∥Vi−{circumflex over (T)}i∥2, where V is the first representation, and {circumflex over (T)} is the modified representation, the trained neural network comprising:
- a trained feature extraction convolutional neural network (CNN);
- and
- a domain adaptation sub-network, the domain adaptation sub-network converting a cross-spectrum representation of the cross-spectrum image generated by the trained feature extraction CNN to the modified representation;
- receiving, from the trained neural network, a representation of the cross-spectrum image;
- predicting an identity of the person in the cross-spectrum image based on a comparison of the representation of the cross-spectrum image and a representation of a visible image of the person in the visible imagery;
- and
- presenting the identity the person in the cross-spectrum image.
6 more elements
A method for face recognition, comprising:
- receiving an off-pose visible spectrum image of a person among people in a visible imagery;
- providing the off-pose visible spectrum image to a trained neural network, the trained neural network having been trained using both a first representation based on visual training images and a modified representation based on cross-spectrum training images by minimizing a distance between the first representation and the modified representation using a pose-correction loss function, wherein the pose-correction loss function is defined as Lp=Σ∥Vi−{circumflex over (T)}i∥2, where V is the first representation, and T is the modified representation, the trained neural network comprising:
- a trained feature extraction convolutional neural network (CNN);
- and
- a domain adaptation sub-network, the domain adaptation sub-network converting an off-pose representation of the off-pose visible spectrum image generated by the trained feature extraction CNN to the modified representation;
- receiving, from the trained neural network, the modified representation of the off-pose visible spectrum image;
- predicting an identity of the person in the off-pose visible spectrum image based on a comparison of the representation of the off-pose visible spectrum image and a representation of a visible image of the person in the visible imagery;
- and
- presenting the identity the person in the off-pose visible image.
6 more elements
Dependencies are inferred from the claim text.