# US 11,113,510 B1 · Virtual templates for facial recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,510 B1 - Document ID: US11113510B1 - Status: Granted - Kind: B1 - Application number: 16/427,948 - Filed: 2019-05-31 - Granted: 2021-09-07 - Earliest claimed: 2018-06-03 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Eslam A. Mostafa; Kelsey Y. Ho - Web page: https://patentgrep.com/patent/US11113510B1 ## CPC - G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints - G06F 18/23: Clustering techniques - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 40/166: {using acquisition arrangements} - G06V 40/168: {Feature extraction; Face representation} - G06V 40/172: {Classification, e.g. identification} - G06V 40/50: Maintenance of biometric data or enrolment thereof - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract When a device is successfully unlock using a facial recognition authentication process, feature vectors generated from images obtained during the facial recognition authentication process may be stored as temporary templates on the device. After a period of time, one of the temporary templates may be selected to be used as a “virtual” template for the device. For example, a median temporary template in the temporary templates may be selected as the virtual template. The performance of the virtual template may then be assessed over time and compared to the performance of templates generated from an enrollment process to determine if and how the virtual template is implemented on the device. ## Claims ### Claim 1 A method, comprising: obtaining a plurality of images of a face of a user using a camera located on a device, the device comprising a computer processor and a memory, wherein the plurality of images is obtained for user authentication attempts on the device using a facial recognition authentication process; for each obtained image: encoding the image to generate at least one feature vector for the image, wherein the at least one feature vector represents one or more facial features of the user in the image; comparing the at least one feature vector to one or more reference templates stored in the memory of the device to obtain a matching score for the at least one feature vector; storing the at least one feature vector as a temporary template in the memory of the device when the matching score for the at least one feature vector is above a threshold; after a selected number of temporary templates are stored in the memory of the device, selecting one of the temporary templates based on an assessment of the temporary templates; obtaining a plurality of additional images for additional user authentication attempts of the device using the facial recognition authentication process, assessing, using the plurality of additional images, a performance of the reference templates in the facial recognition authentication process during the additional user authentication attempts of the device; assessing, using the plurality of additional images, a performance of the selected temporary template in an additional facial recognition authentication process during the additional user authentication attempts of the device, comparing the performance of the reference templates and the performance of the selected temporary template during the additional user authentication attempts of the device; and adding the selected temporary template to the reference templates in response to the performance of the selected temporary template being higher than at least one of the reference templates during the additional user authentication attempts of the device. ### Claim 2 (depends on 1) The method of claim 1, wherein selecting the temporary template comprises selecting a median temporary template from the temporary templates. ### Claim 3 (depends on 1) The method of claim 1, wherein selecting the temporary template comprises assessing one or more clusters of the temporary templates in a feature space and selecting a median temporary template in a cluster of temporary templates. ### Claim 4 (depends on 1) The method of claim 1, wherein the threshold for storing the feature vector as the temporary template is an unlock threshold for the device. ### Claim 5 (depends on 1) The method of claim 1, wherein the selected temporary template replaces one of the reference templates when the performance of the selected temporary template is higher than at least one of the reference templates. ### Claim 6 (depends on 1) The method of claim 1, wherein the selected temporary template replaces all the reference templates when the performance of the selected temporary template is higher than each of the reference templates. ### Claim 7 (depends on 1) The method of claim 1, wherein the reference templates are generated by: obtaining a plurality of reference images of the face of the user using the camera; selecting reference images for encoding based on the selected reference images meeting selected criteria for the reference images; encoding a plurality of facial features of the user from the selected reference images to generate the plurality of reference feature vectors; and storing the plurality of reference feature vectors as the reference templates in the memory of the device. ### Claim 8 (depends on 1) The method of claim 1, wherein the additional facial recognition authentication process operates alongside the facial recognition authentication process with the selected temporary template replacing the reference templates in the additional facial recognition authentication process. ### Claim 9 A device, comprising: a computer processor; a memory; a camera; at least one illuminator providing infrared illumination; circuitry coupled to the camera and the illuminator, wherein the circuitry is configured to: obtain a plurality of images of a face of a user using the camera, wherein the plurality of images is obtained for user authentication attempts on the device using a facial recognition authentication process; for each obtained image: encode the image to generate at least one feature vector for the image, wherein the at least one feature vector represents one or more facial features of the user in the image; compare the at least one feature vector to one or more reference templates stored in the memory of the device to obtain a matching score for the at least one feature vector; store the at least one feature vector as a temporary template in the memory of the device when the matching score for the at least one feature vector is above a threshold; after a selected number of temporary templates are stored in the memory of the device, select one of the temporary templates based on an assessment of the temporary templates; obtain a plurality of additional images for additional user authentication attempts of the device using the facial recognition authentication process; assess, using the plurality of additional images, a performance of the reference templates in the facial recognition authentication process during the additional user authentication attempts of the device; assess, using the plurality of additional images, a performance of the selected temporary template in an additional facial recognition authentication process during the additional user authentication attempts of the device; compare the performance of the reference templates and the performance of the selected temporary template during the additional user authentication attempts of the device; and add the selected temporary template to the reference templates in response to the performance of the selected temporary template being higher than at least one of the reference templates during the additional user authentication attempts of the device. ### Claim 10 (depends on 9) The device of claim 9, wherein the camera comprises an infrared sensor. ### Claim 11 (depends on 9) The device of claim 9, wherein the at least one illuminator comprises a flood infrared illuminator and a pattern infrared illuminator. ### Claim 12 (depends on 9) The device of claim 9, wherein the circuitry is configured to perform an enrollment process to generate the reference templates. ### Claim 13 (depends on 12) The device of claim 12, wherein the circuitry is, in generating the reference templates, configured to: obtain a plurality of reference images of the face of the user using the camera; select reference images for encoding based on the selected reference images meeting selected criteria for the reference images; encode a plurality of facial features of the user from the selected reference images to generate the plurality of reference feature vectors; and store the plurality of reference feature vectors as the reference templates in the memory of the device. ### Claim 14 (depends on 9) The device of claim 9, wherein the circuitry is configured to select the temporary template by selecting a median temporary template from the temporary templates. ### Claim 15 (depends on 9) The device of claim 9, wherein the circuitry is configured to select the temporary template by assessing one or more clusters of the temporary templates in a feature space and selecting a median temporary template in a cluster of temporary templates. ### Claim 16 (depends on 9) The device of claim 9, wherein the facial recognition authentication process is operated to authenticate the user as an authorized user of the device. ### Claim 17 A non-transient computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method, comprising, comprising: obtaining a plurality of images of a face of a user using a camera located on a device, the device comprising a computer processor and a memory, wherein the plurality of images is obtained for user authentication attempts on the device using a facial recognition authentication process; for each obtained image: encoding the image to generate at least one feature vector for the image, wherein the at least one feature vector represents one or more facial features of the user in the image; comparing the at least one feature vector to one or more reference templates stored in the memory of the device to obtain a matching score for the at least one feature vector; storing the at least one feature vector as a temporary template in the memory of the device when the matching score for the at least one feature vector is above a threshold; after a selected number of temporary templates are stored in the memory of the device, selecting one of the temporary templates based on an assessment of the temporary templates; obtaining a plurality of additional images for additional user authentication attempts of the device using the facial recognition authentication process; assessing, using the plurality of additional images, a performance of the reference templates in the facial recognition authentication process during the additional user authentication attempts of the device; assessing, using the plurality of additional images, a performance of the selected temporary template in an additional facial recognition authentication process during the additional user authentication attempts of the device, comparing the performance of the reference templates and the performance of the selected temporary template during the additional user authentication attempts of the device; and adding the selected temporary template to the reference templates in response to the performance of the selected temporary template being higher than at least one of the reference templates during the additional user authentication attempts of the device. ### Claim 18 (depends on 17) The non-transient computer-readable medium of claim 17, wherein the facial recognition authentication process comprises: obtaining at least one user authentication attempt image of the user using the camera located on the device; encoding the at least one user authentication attempt image to generate at least one user authentication attempt feature vector, wherein the user authentication attempt feature vector represents one or more facial features of the user in the at least one user authentication attempt image; comparing the at least one user authentication attempt feature vector to the reference templates to obtain a first matching score; comparing the at least one user authentication attempt feature vector to the selected temporary template to obtain a second matching score; and authorizing the user to perform at least one operation on the device that requires authentication when either the first matching score or the second matching score is above an unlock threshold. ### Claim 19 (depends on 17) The non-transient computer-readable medium of claim 17, wherein the images are infrared images of the face of the user. ### Claim 20 (depends on 17) The non-transient computer-readable medium of claim 17, wherein assessing the performance of the reference templates and the performance of the selected temporary template comprises assessing acceptance rates and rejection rates of the plurality of additional images obtained during the additional user authentication attempts of the device.