# US 11,908,057 B2 · Image regularization and retargeting system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,057 B2 - Document ID: US11908057B2 - Status: Granted - Kind: B2 - Application number: 17/518,521 - Filed: 2021-11-03 - Published: 2022-02-24 - Granted: 2024-02-20 - Earliest claimed: 2016-09-07 - Assignees: Soul Machines Limited; Sony Interactive Entertainment America LLC - Applicants: Sony Interactive Entertainment America LLC; Soul Machines Limited - Inventors: Mark Andrew Sagar; Tim Szu-Hsien Wu; Frank Filipe Bonniwell; Homoud B. Alkouh; Colin Joseph Hodges - Web page: https://patentgrep.com/patent/US11908057B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 7/251: {involving models} - G06T 7/75: {involving models} - G06T 13/00: Animation - G06T 15/04: Texture mapping - G06T 2207/10016: Video; Image sequence - G06T 2207/30201: Face - G06T 2207/30208: Marker matrix - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods for image retargeting are provided. Image data may be acquired that includes motion capture data indicative of motion of a plurality of markers disposed on a surface of a first subject. Each of the markers may be associated with a respective location on the first subject. A plurality of blendshapes may be calculated for the motion capture data based on a configuration of the markers. An error function may be identified for the plurality of blendshapes, and it may be determined that the plurality of blendshapes can be used to retarget a second subject based on the error function. The plurality of blendshapes may then be applied to a second subject to generate a new animation. ## Claims ### Claim 1 A computer-implemented method for retargeting from an image or sequence of images, the method comprising: identifying a body surface portrayed in a reference image of a physical subject based on respective body surface marker points in the reference image; generating a representation of the body surface by reference to a group of known body surface positions that correspond with one or more facial muscles associated with a predetermined facial expression, the representation generated by applying a weight to at least one of the known body surface positions and further by utilization of L1-norm regularization via a physiological based energy equation; retargeting of the representation of the body surface determined according to selection of a plurality of shape primitives; and based on the retargeting, performing the predetermined facial expression due to a result of evaluating the plurality of shape primitives according to an L1-norm energy minimization function reaching a required minimum activation energy threshold of the predetermined facial expression. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein one or more of the respective facial muscles associated with the known body surface positions comprise an active muscle state. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein determining selection of a plurality of shape primitives comprises: identifying a previous shape weight associated with a respective facial muscle referenced in the representation of the body surface, the previous shape weight indicating a contraction of the respective facial muscle. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising: applying a temporal response function with the identified previous shape weight for the respective facial muscle during the evaluating of the plurality of shape primitives according to an energy minimization function. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein applying the temporal response function comprises: adjusting the temporal response function to account for a contracted respective facial muscle. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the contracted respective facial muscle represents a facial muscle contraction prior to initiation of performing of the predetermined facial expression. ### Claim 7 A non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor, cause the processor to perform a series of operations comprising: identifying a body surface portrayed in a reference image of a physical subject based on respective body surface marker points in the reference image; generating a representation of the body surface by reference to a group of known body surface positions that correspond with one or more facial muscles associated with a predetermined facial expression, the representation generated by applying a weight to at least one of the known body surface positions and further by utilization of L1-norm regularization via a physiological based energy equation; retargeting of the representation of the body surface determined according to selection of a plurality of shape primitives; and based on the retargeting, performing the predetermined facial expression due to a result of evaluating the plurality of shape primitives according to an energy minimization function reaching a required minimum activation energy threshold of the predetermined facial expression. ### Claim 8 (depends on 7) The non-transitory computer readable medium of claim 7, wherein one or more of the respective facial muscles associated with the known body surface positions comprise an active muscle state. ### Claim 9 (depends on 8) The non-transitory computer readable medium of claim 8, wherein determining selection of a plurality of shape primitives comprises: identifying a previous shape weight associated with a respective facial muscle referenced in the representation of the body surface, the previous shape weight indicating a contraction of the respective facial muscle. ### Claim 10 (depends on 9) The non-transitory computer readable medium of claim 9, further comprising: applying a temporal response function with the identified previous shape weight for the respective facial muscle during the evaluating of the plurality of shape primitives according to an energy minimization function. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein applying the temporal response function comprises: adjusting the temporal response function to account for a contracted respective facial muscle. ### Claim 12 (depends on 11) The non-transitory computer readable medium of claim 11, wherein the contracted respective facial muscle represents a facial muscle contraction prior to initiation of performing of the predetermined facial expression. ### Claim 13 A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to: identifying a body surface portrayed in a reference image of a physical subject based on respective body surface marker points in the reference image; generating a representation of the body surface by reference to a group of known body surface positions that correspond with one or more facial muscles associated with a predetermined facial expression, the representation generated by applying a weight to at least one of the known body surface positions and further by utilization of L1-norm regularization via a physiological based energy equation; retargeting of the representation of the body surface determined according to selection of a plurality of shape primitives; and based on the retargeting, performing the predetermined facial expression due to a result of evaluating the plurality of shape primitives according to an energy minimization function reaching a required minimum activation energy threshold of the predetermined facial expression. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein one or more of the respective facial muscles associated with the known body surface positions comprise an active muscle state. ### Claim 15 (depends on 14) The computer program product of claim 14, wherein determining selection of a plurality of shape primitives comprises: identifying a previous shape weight associated with a respective facial muscle referenced in the representation of the body surface, the previous shape weight indicating a contraction of the respective facial muscle. ### Claim 16 (depends on 15) The computer program product of claim 15, further comprising: applying a temporal response function with the identified previous shape weight for the respective facial muscle during the evaluating of the plurality of shape primitives according to an energy minimization function. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein applying the temporal response function comprises: adjusting the temporal response function to account for a contracted respective facial muscle. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein the contracted respective facial muscle represents a facial muscle contraction prior to initiation of performing of the predetermined facial expression. ### Claim 19 (depends on 17) The computer program product of claim 17, wherein implementing Non-Uniform Rational B Splines (NURBS); based on the implementation of NURBS, producing a mathematical representation of a 3-D geometry corresponding to a physical head of a physical subject; and generating the reference image based on the 3-D geometry.