# US 12,178,307 B2 · Instrumenting makeup applicators for improving and sharing application techniques > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,178,307 B2 - Document ID: US12178307B2 - Status: Granted - Kind: B2 - Application number: 17/937,207 - Filed: 2022-09-30 - Published: 2024-04-04 - Granted: 2024-12-31 - Earliest claimed: 2022-09-30 - Assignees: L'oreal - Applicants: L'Oreal - Inventors: Juwan Hong; Grégoire Charraud; Ji Heun Lee - Web page: https://patentgrep.com/patent/US12178307B2 ## CPC - A45D 44/005: {for selecting or displaying personal cosmetic colours or hairstyle} - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/73: using feature-based methods - G06T 7/74: {involving reference images or patches} - G06T 11/60: Creating or editing images; Combining images with text - A45D 2200/25: Kits - G06T 2207/10016: Video; Image sequence - G06T 2207/30201: Face - A45D: HAIRDRESSING OR SHAVING EQUIPMENT (wigs, toupees or the like A41G3/00, A41G5/00; hairdressers' chairs A47C1/04; hair cutting appliances, razors B26B); EQUIPMENT FOR COSMETICS OR COSMETIC TREATMENTS, e.g. FOR MANICURING OR PEDICURING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In some embodiments, a system for generating presentations of makeup applications is provided. The system comprises a camera, at least one motion sensor associated with a first applicator; and a computing device that includes at least one processor and a non-transitory computer-readable medium having computer-executable instructions stored thereon. The instructions, in response to execution by the at least one processor, cause the computing device to perform actions comprising: receiving a first video of a makeup application process from the camera, wherein the first video depicts at least a portion of a face and at least a portion of an applicator; receiving signals from the at least one motion sensor indicating motion of the applicator; measuring the motion of the applicator with respect to one or more landmarks of the face to determine a relative applicator path; and storing the relative applicator path. ## Claims ### Claim 1 A system for generating presentations of makeup applications, the system comprising: a camera; at least one motion sensor associated with a first applicator; and a computing device that includes at least one processor and a non-transitory computer-readable medium having computer-executable instructions stored thereon that, in response to execution by the at least one processor, cause the computing device to perform actions comprising: receiving, by the computing device, a first video of a makeup application process from the camera, wherein the first video depicts at least a portion of a face and at least a portion of an applicator; receiving, by the computing device, signals from the at least one motion sensor indicating motion of the applicator; measuring, by the computing device, the motion of the applicator with respect to one or more landmarks of the face to determine a relative applicator path; measuring, by the computing device, an amount of deflection shown in the first video of the portion of the applicator to determine an amount of force with which the applicator is applied; adding, by the computing device, the amount of force to the relative applicator path; and storing, by the computing device, the relative applicator path. ### Claim 2 (depends on 1) The system of claim 1, wherein the at least one motion sensor is removably coupled to the applicator, is permanently attached to the applicator, or is coupled to a glove worn by a hand holding the applicator. ### Claim 3 (depends on 1) The system of claim 1, wherein the computing device is a mobile computing device that includes a display. ### Claim 4 (depends on 3) The system of claim 3, wherein the camera is a front-facing camera of the mobile computing device. ### Claim 5 (depends on 4) The system of claim 4, wherein the actions further comprise: receiving, by the computing device, a second video from the camera, wherein the second video is a live video that depicts at least a portion of the face and at least a portion of the applicator; aligning, by the computing device, the relative applicator path to one or more landmarks of the face; and presenting, by the computing device, the relative applicator path in association with capturing the second video. ### Claim 6 (depends on 5) The system of claim 5, wherein the first video and the second video include depth information. ### Claim 7 (depends on 5) The system of claim 5, wherein presenting the relative applicator path in association with capturing the second video includes providing the presentation of the relative applicator path as superimposed images on the second video on the display. ### Claim 8 (depends on 5) The system of claim 5, wherein presenting the relative applicator path in association with capturing the second video includes: comparing a detected location of the applicator to the relative applicator path; and providing haptic feedback in response to determining that the detected location of the applicator is outside of the relative applicator path. ### Claim 9 A computer-implemented method of generating presentations of makeup applications, the method comprising: receiving, by a computing device, a first video of a makeup application process from a camera, wherein the first video depicts at least a portion of a face and at least a portion of an applicator; receiving, by the computing device, signals from at least one motion sensor indicating motion of the applicator; measuring, by the computing device, the motion of the applicator with respect to one or more landmarks of the face to determine a relative applicator path; measuring, by the computing device, an amount of deflection shown in the first video of the portion of the applicator to determine an amount of force with which the applicator is applied; adding, by the computing device, the amount of force to the relative applicator path; and storing, by the computing device, the relative applicator path. ### Claim 10 (depends on 9) The method of claim 9, wherein the at least one motion sensor is removably coupled to the applicator, is permanently attached to the applicator, or is coupled to a glove worn by a hand holding the applicator. ### Claim 11 (depends on 9) The method of claim 9, wherein the computing device is a mobile computing device that includes a display. ### Claim 12 (depends on 11) The method of claim 11, wherein the camera is a front-facing camera of the mobile computing device. ### Claim 13 (depends on 12) The method of claim 12, further comprising: receiving, by the computing device, a second video from the camera, wherein the second video is a live video that depicts at least a portion of the face and at least a portion of the applicator; aligning, by the computing device, the relative applicator path to one or more landmarks of the face; and presenting, by the computing device, the relative applicator path in association with capturing the second video. ### Claim 14 (depends on 13) The method of claim 13, wherein the first video and the second video include depth information. ### Claim 15 (depends on 13) The method of claim 13, wherein presenting the relative applicator path in association with capturing the second video includes providing the presentation of the relative applicator path as superimposed images on the second video on the display. ### Claim 16 (depends on 13) The method of claim 13, wherein presenting the relative applicator path in association with capturing the second video includes: comparing a detected location of the applicator to the relative applicator path; and providing haptic feedback in response to determining that the detected location of the applicator is outside of the relative applicator path. ### Claim 17 A computing device, comprising: circuitry for receiving a first video of a makeup application process from a camera, wherein the first video depicts at least a portion of a face and at least a portion of an applicator; circuitry for receiving signals from at least one motion sensor indicating motion of the applicator; circuitry for measuring the motion of the applicator with respect to one or more landmarks of the face to determine a relative applicator path; circuitry for measuring an amount of deflection shown in the first video of the portion of the applicator to determine an amount of force with which the applicator is applied; circuitry for adding the amount of force to the relative applicator path; and circuitry for storing the relative applicator path. ### Claim 18 (depends on 17) The computing device of claim 17, wherein the at least one motion sensor is removably coupled to the applicator, is permanently attached to the applicator, or is coupled to a glove worn by a hand holding the applicator. ### Claim 19 (depends on 17) The computing device of claim 17, wherein the computing device is a mobile computing device that includes a display, wherein the camera is a front-facing camera of the mobile computing device, and wherein the computing device further comprises: circuitry for receiving a second video from the camera, wherein the second video is a live video that depicts at least a portion of the face and at least a portion of the applicator; circuitry for aligning the relative applicator path to one or more landmarks of the face; and circuitry for presenting the relative applicator path in association with capturing the second video. ### Claim 20 (depends on 19) The computing device of claim 19, wherein presenting the relative applicator path in association with capturing the second video includes providing the presentation of the relative applicator path as superimposed images on the second video on the display.