# US 10,482,336 B2 · Face recognition and image search system using sparse feature vectors, compact binary vectors, and sub-linear search > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,336 B2 - Document ID: US10482336B2 - Status: Granted - Kind: B2 - Application number: 15/724,936 - Filed: 2017-10-04 - Published: 2018-04-12 - Granted: 2019-11-19 - Earliest claimed: 2016-10-07 - Assignees: Noblis, Inc. - Applicants: Noblis, Inc. - Inventors: Mark J. Burge; Jordan Cheney - Web page: https://patentgrep.com/patent/US10482336B2 ## CPC - G06V 10/82: using neural networks - G06F 16/367: {Ontology} - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06F 16/5838: {using colour} - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/169: {Holistic features and representations, i.e. based on the facial image taken as a whole} - G06V 40/172: {Classification, e.g. identification} - G06V 40/50: Maintenance of biometric data or enrolment thereof - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods are provided for performing automated face recognition and comparison. An input image of a face may be received and cropped, and the image may be processed through a deep neural network (DNN) to produce a k-dimensional feature vector. The k-dimensional feature vector may be converted to a k-dimensional binary vector by transforming each value in the vector to either 1 or 0. To search for nearest matches of the image in a database of gallery images of faces, the system may compare sub-strings of the binary vector to hash tables created from sub-strings of the gallery images, enabling sub-linear searching that allows locating the closest matches from among the entire gallery without requiring an exhaustive linear search of the entire gallery. ## Claims ### Claim 1 A method, performed by a search system including one or more servers for enrollment and searching, for searching a database of gallery binary vectors representing gallery images of faces, comprising: receiving an input image of a face; processing the received input image of the face, comprising: processing through a plurality of layers of a deep neural network (DNN) to generate a k-dimensional feature vector; and binarizing the feature vector to generate a k-dimensional binary vector by converting respective values in the feature vector to corresponding values of 0 or 1 in the binary vector, wherein the conversion is based on whether respective values in the feature vector satisfy predefined criteria; dividing the binary vector into a plurality of sub-strings; creating a plurality of hash tables based on gallery sub-strings of gallery binary vectors; and comparing one or more of the sub-strings of the binary vector to the one or more of the hash tables to identify a subset of the gallery binary vectors that are closest matches to the binary vector. ### Claim 2 (depends on 1) The method of claim 1, wherein processing the received input image of the face comprises, prior to processing the image through the DNN, cropping the input image of the face to a predefined size. ### Claim 3 (depends on 1) The method of claim 1, wherein identifying the subset of the gallery binary vectors that are the closest matches is based on a number of comparisons that is fewer than a total number of gallery binary vectors from which the closest matches are identified. ### Claim 4 (depends on 1) The method of claim 1, wherein identifying the subset of the gallery binary vectors that are the closest matches is based on a number of comparisons that is equal to or less than log(N), where N is a total number of gallery binary vectors from which the closest matches are identified. ### Claim 5 (depends on 1) The method of claim 1, wherein the DNN comprises a fully connected layer configured to reduce dimensionality in the output k-dimensional feature vector to k. ### Claim 6 (depends on 1) The method of claim 1, wherein k is less than 1/500 the number of data dimensions the cropped input image of the face input into DNN. ### Claim 7 (depends on 1) The method of claim 1, wherein k is 256. ### Claim 8 (depends on 1) The method of claim 1, wherein the binary vector is less than 1/4000 the file size of the cropped input image of the face input into the DNN. ### Claim 9 (depends on 1) The method of claim 1, wherein the binary vector is 32 bytes in file size. ### Claim 10 A system, for searching a database of gallery binary vectors representing gallery images of faces, the system comprising one or more servers for enrollment and searching and memory, the memory storing instructions that, when executed by the one or more servers, cause the one or more servers to: receive an input image of a face; process the received input image of the face, comprising: processing through a plurality of layers of a deep neural network (DNN) to generate a k-dimensional feature vector; and binarizing the feature vector to generate a k-dimensional binary vector by converting respective values in the feature vector to corresponding values of 0 or 1 in the binary vector, wherein the conversion is based on whether respective values in the feature vector satisfy predefined criteria; divide the binary vector into a plurality of sub-strings; create a plurality of hash tables based on gallery sub-strings of gallery binary vectors; and compare one or more of the sub-strings of the binary vector to the one or more of the hash tables to identify a subset of the gallery binary vectors that are closest matches to the binary vector. ### Claim 11 (depends on 10) The system of claim 10, wherein processing the received input image of the face comprises, prior to processing the image through the DNN, cropping the input image of the face to a predefined size. ### Claim 12 (depends on 10) The system of claim 10, wherein identifying the subset of the gallery binary vectors that are the closest matches is based on a number of comparisons that is fewer than a total number of gallery binary vectors from which the closest matches are identified. ### Claim 13 (depends on 10) The system of claim 10, wherein identifying the subset of the gallery binary vectors that are the closest matches is based on a number of comparisons that is equal to or less than log(N), where N is a total number of gallery binary vectors from which the closest matches are identified. ### Claim 14 (depends on 10) The system of claim 10, wherein the DNN comprises a fully connected layer configured to reduce dimensionality in the output k-dimensional feature vector to k. ### Claim 15 (depends on 10) The system of claim 10, wherein k is less than 1/500 the number of data dimensions the cropped input image of the face input into DNN. ### Claim 16 (depends on 10) The system of claim 10, wherein k is 256. ### Claim 17 (depends on 10) The system of claim 10, wherein the binary vector is less than 1/4000 the file size of the cropped input image of the face input into the DNN. ### Claim 18 (depends on 10) The system of claim 10, wherein the binary vector is 32 bytes in file size. ### Claim 19 A non-transitory computer-readable storage medium storing instructions that, when executed by one or more servers for enrollment and searching, cause the one or more servers to: receive an input image of a face; process the received input image of the face, comprising: processing through a plurality of layers of a deep neural network (DNN) to generate a k-dimensional feature vector; and binarizing the feature vector to generate a k-dimensional binary vector by converting respective values in the feature vector to corresponding values of 0 or 1 in the binary vector, wherein the conversion is based on whether respective values in the feature vector satisfy predefined criteria; divide the binary vector into a plurality of sub-strings; create a plurality of hash tables based on gallery sub-strings of gallery binary vectors; and compare one or more of the sub-strings of the binary vector to the one or more of the hash tables to identify a subset of the gallery binary vectors that are closest matches to the binary vector. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein processing the received input image of the face comprises, prior to processing the image through the DNN, cropping the input image of the face to a predefined size. ### Claim 21 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein identifying the subset of the gallery binary vectors that are the closest matches is based on a number of comparisons that is fewer than a total number of gallery binary vectors from which the closest matches are identified. ### Claim 22 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein identifying the subset of the gallery binary vectors that are the closest matches is based on a number of comparisons that is equal to or less than log(N), where Nis a total number of gallery binary vectors from which the closest matches are identified. ### Claim 23 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the DNN comprises a fully connected layer configured to reduce dimensionality in the output k-dimensional feature vector to k. ### Claim 24 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein k is less than 1/500 the number of data dimensions the cropped input image of the face input into DNN. ### Claim 25 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein k is 256. ### Claim 26 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the binary vector is less than 1/4000 the file size of the cropped input image of the face input into the DNN. ### Claim 27 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the binary vector is 32 bytes in file size.