# US 2022/0284688 A1 · Lightweight real-time facial alignment with one-shot neural architecture search > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0284688 A1 - Document ID: US20220284688A1 - Status: Published application - Kind: A1 - Application number: 17/685,691 - Filed: 2022-03-03 - Published: 2022-09-08 - Earliest claimed: 2021-03-03 - Assignees: L'oreal - Applicants: L'oreal - Inventors: Zihao CHEN; Zhi Yu; Parham Aarabi - Web page: https://patentgrep.com/patent/US20220284688A1 ## CPC - G06N 3/045: Combinations of networks - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming - G06N 3/09: Supervised learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06Q 30/0631: {Recommending goods or services} - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 10/87: {using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system} - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract With Convolutional Neural Networks (CNN), facial alignment networks (FAN) have achieved significant accuracy on a wide range of public datasets, which comes along with larger model size and expensive computation costs, making it infeasible to adapt them to real-time applications on edge devices. There is provided a model compression approach on FAN using One-Shot Neural Architecture Search to overcome this problem while preserving performance criteria. Methods and devices provide efficient training and searching (on a single GPU), and resultant models can deploy to run real-time in browser-based applications on edge devices including tablets and smartphones. The compressed models provide comparable cutting-edge accuracy, while having a 30 times smaller model size and can run 40.7 ms per frame in a popular browser on a popular smartphone and OS. ## Claims ### Claim 1 A method comprising: defining a face alignment network (FAN) model for run-time execution by an edge device to process facial images, wherein defining comprises: generating a set of candidate submodels from a network structure through training the network structure using i) a training dataset; and ii) expand and shrink training (EST) operations, which EST operations define and retain candidate submodel instances of various structure parameters with which to define the set of candidate submodels; and performing an evolutionary search of the candidate submodels using speed and accuracy evaluation criteria to select the FAN model from the candidate submodels. ### Claim 2 (depends on 1) The method of claim 1, wherein the EST operations initiate training from a baseline model previously defined in accordance with the network structure. ### Claim 3 (depends on 2) The method of claim 2, wherein the EST operations utilize pre-trained parameters from the baseline model as partially trained parameters in each of the candidate submodels for further training using EST operations. ### Claim 4 (depends on 3) The method of claim 3, wherein the candidate submodels are further trained for any one of: between 2 epochs and 60 epochs; between 2 and 25 epochs; and 2 epochs. ### Claim 5 (depends on 2) The method of claim 2, wherein the EST operations activate random subnets in each training iteration to train partial weights of subnets as integral and to train the sharing weights effectively among submodels. ### Claim 6 (depends on 2) The method of claim 2, wherein the EST operations define a search space for generation of candidate submodels in accordance with a plurality of search dimensions comprising: depth size; kernel size; channel ratio; and expansion ratio. ### Claim 7 (depends on 6) The method of claim 6, wherein the EST operations initiate with a small depth size, a small kernel size, a large channel ratio, and a large expansion ratio and then operations, in phases and in the following order, gradually expand depth size and kernel size to respective maximums and, at ending phases, progressively shrink channel ratio and expansion ratio to respective minimums. ### Claim 8 (depends on 6) The method of claim 6, wherein the EST operations comprise kernel training operations in an early phase that train a first size kernel and use the first size kernel to subsequently train a second size kernel where the first size kernel is a center of the second size kernel. ### Claim 9 (depends on 5) The method of claim 5, wherein the EST operations comprise depth training operations in a middle phase that train fewer layers and then gradually add larger layer options to the training. ### Claim 10 (depends on 2) The method of claim 2, wherein the evolutionary search: i) evaluates the baseline model to determine the speed and accuracy evaluation criteria; and ii) evolves from the baseline model to define a population of candidate submodels for subsequent sampling and evaluation to select the FAN model. ### Claim 11 (depends on 10) The method of claim 10, wherein the evolutionary search: performs a plurality of cycles comprising: sampling from the population to select a quantity of candidate submodels; selecting an optimal submodel from the quantity of candidate submodels according to at least some of the speed and accuracy evaluation criteria; evolving the optimal submodel to determine a related submodel; removing an oldest submodel from the population; and if the related submodel satisfies a threshold test associated with the speed and accuracy evaluation criteria: adding the related submodel to the population for availability to subsequently sample and adding the related submodel to a set of final candidate submodels; and following the performance of the plurality of cycles, selects one submodel from the set of final candidate submodels as the FAN model, in accordance with final selection criteria. ### Claim 12 (depends on 11) The method of claim 11, wherein: the optimal submodel is a fastest submodel of the quantity of candidate submodels in accordance with a speed criteria, the threshold test is a speed threshold test, and the one submodel is the fastest of the set of final candidate submodels. ### Claim 13 (depends on 11) The method of claim 11, wherein: the optimal submodel is a most accurate of the quantity of candidate submodels in accordance with an accuracy criteria, the threshold test is an accuracy threshold test, and the one submodel is the most accurate of the set of final candidate submodels. ### Claim 14 (depends on 1) The method of claim 1 comprising defining an application comprising the FAN model for execution by the edge device. ### Claim 15 (depends on 14) The method of claim 14, wherein the application is configured to process a video by the FAN model, and wherein the video comprises the facial images. ### Claim 16 (depends on 14) The method of claim 14, wherein the edge device is a smartphone or tablet and the FAN model is configured for execution by a graphics processing unit (GPU). ### Claim 17 (depends on 14) The method of claim 14, wherein the application is any one of: a make-up simulation application to simulate the trying on of make-up products on the face; a skin analysis application to analyze and score a region or regions of the face for at least one skin sign; an effects simulation application to simulate a graphical effect associated with a region of the face; and a facial tracking application to track and indicate a location of a face. ### Claim 18 (depends on 14) The method of claim 14, wherein the application comprises a make-up simulation application or a skin analysis application and the application is configured to provide one or both of: an interface to an ecommerce purchase service to purchase a product or service associated to make-up and skin treatment respectively; and an interface to an ecommerce recommendation service that recommends a product or service associated to make-up and skin treatment respectively. ### Claim 19 A computing device comprising a processing unit coupled to a storage device storing instructions, that when executed by the processing unit, cause the computing device to perform a method comprising: defining a face alignment network (FAN) model for execution by an edge device to process an image comprising a face, wherein defining comprises: generating a set of candidate submodels from a network structure through training the network structure using i) a training dataset; and ii) expand and shrink training (EST) operations, which EST operations define and retain candidate submodel instances of various structure parameters with which to define the set of candidate submodels; and performing an evolutionary search of the candidate submodels using speed and accuracy evaluation criteria to select the FAN model from the candidate submodels. ### Claim 20 (depends on 1) A computer device comprising a processing unit coupled to a storage device storing instructions, that when executed by the processing unit, cause the computing device to provide a FAN model defined in accordance with the method of claim 1.