# US 11,308,313 B2 · Hybrid deep learning method for recognizing facial expressions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,313 B2 - Document ID: US11308313B2 - Status: Granted - Kind: B2 - Application number: 16/890,309 - Filed: 2020-06-02 - Published: 2020-09-17 - Granted: 2022-04-19 - Earliest claimed: 2018-04-25 - Assignees: Shutterfly, Inc. - Applicants: Shutterfly, LLC - Inventors: Leo Cyrus - Web page: https://patentgrep.com/patent/US11308313B2 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06F 18/2132: based on discrimination criteria, e.g. discriminant analysis - G06F 18/254: {of classification results, e.g. of results related to same input data} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data} - G06V 10/82: using neural networks - G06V 40/165: {using facial parts and geometric relationships} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/174: {Facial expression recognition} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A computer implemented method for recognizing facial expressions by applying feature learning and feature engineering to face images. The method includes conducting feature learning on a face image comprising feeding the face image into a first convolution neural network to obtain a first decision, conducting feature engineering on a face image, comprising the steps of automatically detecting facial landmarks in the face image, transforming the facial features into a two-dimensional matrix, and feeding the two-dimensional matrix into a second convolution neural network to obtain a second decision, computing a hybrid decision based on the first decision and the second decision, and recognizing a facial expression in the face image in accordance to the hybrid decision. ## Claims ### Claim 1 A computer-implemented method for recognizing facial expressions, comprising: feeding a face image into a first convolution neural network to obtain a first decision by one or more computer processors; automatically detecting facial landmarks in the face image; describing each of the facial landmarks by a set of facial features; transforming the facial features that describe the facial landmarks into a two-dimensional matrix comprising coordinates of the facial features by the one or more computer processors; feeding the two-dimensional matrix into a second convolution neural network to obtain a second decision; computing a hybrid decision based on the first decision and the second decision; and recognizing a facial expression in the face image in accordance to the hybrid decision. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: multiplying the first decision by a first weight to produce a first weighted decision; and multiplying the second decision by a second weight to produce a second weighted decision, wherein the hybrid decision is computed based on the first weighted decision and the second weighted decision. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, further comprising: updating the first weight and the second weight in the hybrid decision by backpropagation. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the hybrid decision is an average, a sum, or a root-mean square function of the first decision and the second decision. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the two-dimensional matrix comprises multiple pairs of coordinates each associated with one of the facial features. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: automatically detecting a face in a digital image by the one or more computer processors; extracting a face portion surrounding the face from the digital image by the one or more computer processors; and normalizing the face portion to obtain the face image. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the facial landmarks are respectively associated with at least a portion of an eye, an eyebrow, a mouth, a chin, an edge of a face, or a nose in the face image. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the facial landmarks include a plurality of groups each of which describes at least a portion of a facial feature. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the facial landmarks include an eye, an eyebrow, a mouth, a chin, an edge of a face, or a nose in the face image. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein each of the facial features is described by a pair of coordinates. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein transforming the facial features into a two-dimensional matrix comprises: describing each of the facial features by a pair of coordinates; and forming a first matrix using the facial features and their respective coordinates. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, further comprising: concatenating copied of the first matrix one or more times to produce the two-dimensional matrix. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the two-dimensional matrix is a square matrix.