# US 10,945,514 B2 · Information processing apparatus, information processing method, and computer-readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,945,514 B2 - Document ID: US10945514B2 - Status: Granted - Kind: B2 - Application number: 14/980,630 - Filed: 2015-12-28 - Published: 2016-05-12 - Granted: 2021-03-16 - Earliest claimed: 2011-03-01 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Mari Saito; Tatsuki Kashitani - Web page: https://patentgrep.com/patent/US10945514B2 ## CPC - A45D 44/005: {for selecting or displaying personal cosmetic colours or hairstyle} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06T 11/00: Two-dimensional [2D] image generation - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - A45D 2044/007: {Devices for determining the condition of hair or skin or for selecting the appropriate cosmetic or hair treatment} - A45D: HAIRDRESSING OR SHAVING EQUIPMENT (wigs, toupees or the like A41G3/00, A41G5/00; hairdressers' chairs A47C1/04; hair cutting appliances, razors B26B); EQUIPMENT FOR COSMETICS OR COSMETIC TREATMENTS, e.g. FOR MANICURING OR PEDICURING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method is provided for generating output image data. The method comprises receiving image data representing an input image, the input image containing at least one facial image. The method further comprises recognizing the facial image in the image data, and recognizing facial features of the facial image. The method further comprises generating data representing a makeup image based on the recognized facial features, the makeup image providing information assisting in the application of makeup. The method also comprises generating output image data representing the makeup image superimposed on the facial image. ## Claims ### Claim 1 An augmented reality makeup support technology apparatus comprising: circuitry configured to: acquire, from sensors attached to cosmetic products, cosmetics information of the cosmetic products used by each makeup artist of a plurality of makeup artists and an order in which the cosmetics products are used by a respective makeup artist of the plurality of makeup artists; store, in a database and for each makeup artist of the plurality of makeup artists, makeup scheme information of a makeup pattern including the cosmetics information of the cosmetic products used by the respective makeup artist and a process number identifying the order in which the cosmetics products are used by the respective makeup artist, the cosmetics information being associated with the process number in the database; control a display to display the plurality of makeup artists for selection by a user; acquire makeup scheme information of a selected makeup artist selected from the plurality of makeup artists; generate a makeup action image based on the makeup scheme information of the selected makeup artist; and control the display to display the makeup action image by superimposing the makeup action image on a face image of a face of the user. ### Claim 2 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the database further stores a makeup procedure which is acquired by a camera and an action sensor attached to the selected makeup artist. ### Claim 3 (depends on 2) The augmented reality makeup support technology apparatus according to claim 2, wherein the action sensor is attached to an arm or hand of the selected makeup artist, and wherein the action sensor acquires pressure information based on contact by makeup tools upon the cosmetic products or skin. ### Claim 4 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the database further stores a makeup procedure which is acquired by one or more sensors attached to a mannequin. ### Claim 5 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the database further stores a makeup procedure which is acquired by one or more sensors attached to makeup tools. ### Claim 6 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the makeup action image is displayed to be overlaid on the face image of the user and refers to a makeup procedure done by the selected makeup artist. ### Claim 7 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the cosmetics information comprises at least one of a maker, a brand name, a product name, a price, and a product explanation. ### Claim 8 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the cosmetics information provides information of appropriate cosmetics to be used when makeup based on the makeup scheme information is applied by the user. ### Claim 9 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, further comprising: a camera configured to capture an image of a new makeup scheme of the user. ### Claim 10 (depends on 9) The augmented reality makeup support technology apparatus according to claim 9, wherein the circuitry is further configured to: initiate transmission of the captured image of the new makeup scheme of the user to a server that stores the database thereon. ### Claim 11 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the circuitry is further configured to: control the display to display at least one makeup scheme provided by each makeup artist of the plurality of makeup artists. ### Claim 12 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the circuitry is further configured to: acquire more than one makeup scheme information from the selected makeup artist. ### Claim 13 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the circuitry is further configured to: obtain makeup scheme information of other users; and determine a makeup pattern as a suitable makeup pattern based on a comparison with the makeup scheme information of other users. ### Claim 14 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the circuitry is further configured to: determine a progress degree of real makeup being applied by the user based on a comparison between an image of the face image of the user when the user is applying makeup and a makeup completion image indicating a makeup completion state and generated based on the makeup scheme information of the selected makeup artist; modify the makeup action image based on the determined progress degree and the order in which the cosmetics products are used by the selected makeup artist; and control the display to display the modified makeup action image. ### Claim 15 (depends on 14) The augmented reality makeup support technology apparatus according to claim 14, wherein the modified makeup action image includes portions of the makeup action image removed or made transparent based on the determined progress degree. ### Claim 16 (depends on 14) The augmented reality makeup support technology apparatus according to claim 14, wherein the makeup scheme information includes a makeup procedure including an order of a plurality of procedures done by the selected makeup artist to generate the makeup completion image, and the determined progress degree indicates procedures of the plurality of procedures which have been completed. ### Claim 17 (depends on 16) The augmented reality makeup support technology apparatus according to claim 16, wherein the makeup action image indicates one procedure of the plurality of procedures and the modified makeup action image indicates another procedure of the plurality of procedures. ### Claim 18 (depends on 17) The augmented reality makeup support technology apparatus according to claim 17, wherein the order in the makeup procedure includes the one procedure and then the another procedure. ### Claim 19 (depends on 1) The augmented reality makeup support technology apparatus according to claim 1, wherein the circuitry is further configured to: control the display to display a buy button in conjunction with the cosmetics information, the buy button provided for making a purchase of corresponding cosmetics. ### Claim 20 An augmented reality makeup support technology method executed using at least one processor, the method comprising: acquiring, from sensors attached to cosmetic products, cosmetics information of the cosmetic products used by each makeup artist of a plurality of makeup artists and an order in which the cosmetics products are used by a respective makeup artist of the plurality of makeup artists; storing, for each makeup artist of the plurality of makeup artists, makeup scheme information of a makeup pattern including the cosmetics information of the cosmetic products used by the respective makeup artist and a process number identifying the order in which the cosmetics products are used by the respective makeup artist, the cosmetics information being associated with the process number in the database; displaying the plurality of makeup artists for selection by a user; acquiring makeup scheme information of a selected makeup artist from the plurality of makeup artists; generating a makeup action image based on the makeup scheme information of the selected makeup artist; and displaying the makeup action image by superimposing the makeup action image on a face image of a face of the user. ### Claim 21 A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising: acquiring, from sensors attached to cosmetic products, cosmetics information of the cosmetic products used by each makeup artist of a plurality of makeup artists and an order in which the cosmetics products are used by a respective makeup artist of the plurality of makeup artists; storing, for each makeup artist of the plurality of makeup artists, makeup scheme information of a makeup pattern including the cosmetics information of the cosmetic products used by the respective makeup artist and a process number identifying the order in which the cosmetics products are used by the respective makeup artist, the cosmetics information being associated with the process number in the database; displaying the plurality of makeup artists for selection by a user; acquiring makeup scheme information of a selected makeup artist from the plurality of makeup artists; generating a makeup action image based on the makeup scheme information of the selected makeup artist; and displaying the makeup action image by superimposing the makeup action image on a face image of a face of the user.