# US 2017/0256084 A1 · Precise application of cosmetic looks from over a network environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0256084 A1 - Document ID: US20170256084A1 - Status: Published application - Kind: A1 - Application number: 15/511,818 - Filed: 2015-09-30 - Published: 2017-09-07 - Earliest claimed: 2014-09-30 - Assignees: TCMS Transparent Beauty LLC - Applicants: TCMS Transparent Beauty, LLC - Inventors: David C. Iglehart; Albert Durr Edgar - Web page: https://patentgrep.com/patent/US20170256084A1 ## CPC - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06T 11/60: Creating or editing images; Combining images with text - G06T 2200/24: involving graphical user interfaces [GUIs] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method for guiding a user to manually apply cosmetics by overlaying graphical indicators on a real-time image of the user, the method including receiving a real-time image of the area of skin. Generating a registration layer containing a mapping between features of the real-time image and a graphical model of the area of skin. Presenting, for display to a user, a control layer overlaid on the real-time image according to the registration layer, the control layer including at least one indicator of a location on the area of skin where a reflectance modifying agent (RMA) should be applied. Determining, based on the real-time image, that the RMA has been applied to the location on the area of skin. In response to determining that the RMA has been applied to the location on the area of skin, modifying the at least one indicator included in the control layer. ## Claims ### Claim 1 A computer implemented method for applying reflectance modifying agents (RMA) to an area of skin, the method comprising: accessing, by one or more processors, a plurality of makeup effects for applying RMA to the area of skin; presenting, for display to a user, the plurality makeup effects; receiving, by the one or more processors, a selection of a makeup effect from the plurality of makeup effects to be applied to the area of skin; obtaining, by the one or more processors, data defining the selected makeup effect, the data identifying one or more of: an RMA to apply to the area of skin, an amount of RMA to apply to the area of skin, and a location on which to apply RMA; and sending, by the one or more processors, the data defining the selected makeup effect to an applicator device, wherein the applicator device is configured to map the data to the area of skin and automatically apply RMA to the area of skin according to the data defining the selected makeup effect. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of makeup effects include one or more of predefined looks, each of the predefined looks comprising data identifying one or more RMAs and one or more locations to apply the RMAs in order to apply the predefined look to the area of skin. ### Claim 3 (depends on 1) The method of claim 1, further comprising receiving an image of the area of skin, and wherein presenting the plurality of makeup effects comprises presenting, for display to a user, a graphical user interface (GUI) including the image of the area of skin and the plurality of makeup effects. ### Claim 4 (depends on 3) The method of claim 3, further comprising modifying the image of the area of skin based on the selected makeup effect to show the user a preview of the selected makeup effect applied to the area of skin. ### Claim 5 (canceled) ### Claim 6 (depends on 1) The method of claim 1, further comprising receiving an image of the area of skin, and wherein presenting the plurality of makeup effects comprises presenting, for display to another user, a graphical user interface (GUI) including the image of the area of skin and the plurality of makeup effects, and wherein receiving a selection of a makeup effect from the plurality of makeup effects to be applied to the area of skin comprises receiving the selection from the other user. ### Claim 7 (depends on 1) The method of claim 1, further comprising: receiving, from the applicator device, information about the area of skin; and selecting, based on the information about the area of skin, a set of makeup effects from the plurality of makeup effects for presentation to the user, wherein presenting the plurality of makeup effects comprises presenting the set of makeup effects. ### Claim 8 (depends on 7) The method of claim 7, wherein the information about the area of skin includes information obtained by the applicator device from a scan of the area of skin. ### Claim 9 A device for applying reflectance modifying agents (RMA) to an area of skin, the device comprising: one or more deposition nozzles; one or more sensors; a display; a container that contains the RMA and that is in fluid communication with at least one of the one or more deposition nozzles; one or more processors that are in communication with the one or more sensors, the one or more deposition nozzles, and the display; and a data store coupled to the one or more processors having instructions stored thereon which, when executed by the at least one processor, causes the one or more processors to perform operations comprising: accessing a plurality of makeup effects for applying reflectance modifying agents (RMA) to the area of skin; presenting, for display to a user, the plurality makeup effects; receiving, from an input device, a selection of a makeup effect from the plurality of makeup effects to be applied to the area of skin; obtaining data defining the selected makeup effect, the data identifying, one or more RMAs to apply to the area of skin, an amount of RMA to apply to the area of skin, and a location on which to apply RMA; mapping the data to the area of skin using a feature map of the area of skin obtained from a scan of the area of skin by the one or more sensors; and causing the one or more deposition nozzles to deposit RMA on the area of skin according the mapped data defining the selected makeup effect. ### Claim 10 (depends on 9) The device of claim 9, wherein the plurality of makeup effects include one or more of predefined looks, each of the predefined looks comprising data identifying one or more RMAs and one or more locations to apply the RMAs in order to apply the predefined look to the area of skin. ### Claim 11 (depends on 9) The device of claim 9, wherein the operations further comprise receiving an image of the area of skin, and wherein presenting the plurality of makeup effects comprises presenting, for display to a user, a graphical user interface (GUI) including the image of the area of skin and the plurality of makeup effects. ### Claim 12 (depends on 11) The device of claim 11, wherein the operations further comprise modifying the image of the area of skin based on the selected makeup effect to show the user a preview of the selected makeup effect applied to the area of skin. ### Claim 13 (canceled) ### Claim 14 (depends on 9) The device of claim 9, wherein the operations further comprise receiving an image of the area of skin, and wherein presenting the plurality of makeup effects comprises presenting, for display to another user, a graphical user interface (GUI) including the image of the area of skin and the plurality of makeup effects, and wherein receiving a selection of a makeup effect from the plurality of makeup effects to be applied to the area of skin comprises receiving the selection from the other user. ### Claim 15 (depends on 9) The device of claim 9, wherein the operations further comprise: receiving, from the applicator device, information about the area of skin; and selecting, based on the information about the area of skin, a set of makeup effects from the plurality of makeup effects for presentation to the user, wherein presenting the plurality of makeup effects comprises presenting the set of makeup effects. ### Claim 16 A computer implemented method for applying cosmetics to an area of skin, the method comprising: receiving, by one or more processors, a real-time image of the area of skin; generating, by the one or more processors, a registration layer containing a mapping between features of the real-time image and a graphical model of the area of skin; presenting, for display to a user, a control layer overlaid on the real-time image according to the registration layer, the control layer including at least one indicator of a location on the area of skin where a reflectance modifying agent (RMA) should be applied; determining, based on the real-time image, that the RMA has been applied to the location on the area of skin; and in response to determining that the RMA has been applied to the location on the area of skin, modifying the at least one indicator included in the control layer. ### Claim at position 17 17-19. (canceled) ### Claim 20 (depends on 16) The method of claim 16, further comprising sending, to a computing device for display to a second user, the control layer and the graphical model of the area of skin, the control layer being mapped to the graphical model according to the registration layer, wherein the at least one indicator included in the control layer is generated from an input received from the second user, the input identifying the location on the area of skin where the cosmetic should be applied. ### Claim 21 (depends on 16) The method of claim 16, further comprising sending, to a second and third computing device for display to a second and a third user, the control layer and the graphical model of the area of skin, the control layer being mapped to the graphical model according to the registration layer, wherein the at least one indicator includes a first and a second indicator, the first indicator being generated from a first input received from the second user, the first input identifying a first location on the area of skin where a first cosmetic should be applied, and the second indicator being generated from a second input received from the third user, the second input identifying a second location on the area of skin where a second cosmetic should be applied. ### Claim 22 (depends on 21) The method of claim 21, further comprising sending, to the second and third computing device for display to the second and the third user, the real-time image of the area of skin, the control layer being mapped to the real-time image according to the registration layer. ### Claim 23 (depends on 16) The method of claim 16, further comprising generating the at least one indicator included in the control layer using an algorithm for detecting blemishes on the area of skin. ### Claim 24 (depends on 23) The method of claim 23, wherein the algorithm for detecting blemishes on the area of skin comprises: obtaining a regional image layer comprising regional median values around each pixel of a frame of the real-time image of the area of skin; obtaining a spot image layer comprising blurred average values across areas centered at each pixel of the frame of the real-time image of the area of skin; determining a correction image layer based on the regional image layer and the spot image layer; and identifying at least one local minima in the correction image layer, wherein the at least one indicator included in the control layer is generated from the at least one local minima in the correction image. ### Claim 25 (depends on 24) The method of claim 24, wherein modifying the at least one indicator included in the control layer comprises: determining whether a sufficient amount of the RMA has been applied to the location by preforming the algorithm for detecting blemishes on the area of skin; and adjusting a color, size, shape, or opacity of the at least one indicator.