# US 11,908,098 B1 · Aligning user representations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,098 B1 - Document ID: US11908098B1 - Status: Granted - Kind: B1 - Application number: 18/370,699 - Filed: 2023-09-20 - Granted: 2024-02-20 - Earliest claimed: 2022-09-23 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Michael S. Hutchinson - Web page: https://patentgrep.com/patent/US11908098B1 ## CPC - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 20/64: Three-dimensional [3D] objects - G06T 5/70: Denoising; Smoothing - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06T 7/55: from multiple images - G06T 7/73: using feature-based methods - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 15/04: Texture mapping - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06V 10/54: relating to texture - G06V 10/94: Hardware or software architectures specially adapted for image or video understanding - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/174: {Facial expression recognition} - G06V 40/19: Sensors therefor - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10048: Infrared image - G06T 2207/20044: Skeletonization; Medial axis transform - G06T 2207/30201: Face - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Various implementations disclosed herein include devices, systems, and methods that generate a combined 3D representation of a user based on an alignment based on a 3D reference point. For example, a process may include obtaining a predetermined three-dimensional (3D) representation that is associated with a 3D reference point defined relative to a skeletal representation of the user. The process may further include obtaining a sequence of frame-specific 3D representations corresponding to multiple instants in a period of time, each of the frame-specific 3D representations representing a second portion of the user at a respective instant of the multiple instants in the period of time. The process may further include generating combined 3D representations of the user generated by combining the predetermined 3D representation with a respective frame-specific 3D representation based on an alignment which is based on the 3D reference point. ## Claims ### Claim 1 A method comprising: at a processor of a device: obtaining a predetermined three-dimensional (3D) representation of at least a first portion of a user, wherein the predetermined 3D representation is associated with a 3D reference point defined relative to a skeletal representation of the user; obtaining a sequence of frame-specific 3D representations corresponding to multiple instants in a period of time, each of the frame-specific 3D representations representing a second portion of the user at a respective instant of the multiple instants in the period of time and each of the frame-specific 3D representations associated with the 3D reference point; and generating combined 3D representations of the user for the multiple instants in the period of time, wherein each of the combined 3D representations is generated by combining the predetermined 3D representation with a respective frame-specific 3D representation based on an alignment, wherein the alignment is based on the 3D reference point. ### Claim 2 (depends on 1) The method of claim 1, wherein the second portion of the user is a sub-portion of the first portion. ### Claim 3 (depends on 2) The method of claim 2, wherein: the first portion of the user comprises a face portion and an additional portion of the user; and the second portion of the user comprises the face portion without the additional portion of the user. ### Claim 4 (depends on 1) The method of claim 1, wherein the 3D reference point is associated with a 3D position of an atlas joint of a skeletal representation of the user. ### Claim 5 (depends on 4) The method of claim 4, wherein the 3D reference point is associated with a center of eyes of the user defined at a position at an offset from the atlas joint. ### Claim 6 (depends on 1) The method of claim 1, wherein combining the predetermined 3D representation with the respective frame-specific 3D representations comprises adjusting a sub-portion of the predetermined 3D representation. ### Claim 7 (depends on 6) The method of claim 6, wherein adjusting the sub-portion of the predetermined 3D representation comprises adjusting positions of vertices of the predetermined 3D representation and applying texture based on each of the frame-specific 3D representations. ### Claim 8 (depends on 1) The method of claim 1, wherein each of the frame-specific 3D representations is defined using height values specified within a heightfield that is positionally associated with the 3D reference point. ### Claim 9 (depends on 8) The method of claim 8, wherein vertices of the predetermined 3D representation are mapped to positions on the heightfield and associated with height values, wherein the vertices are mapped based on the 3D reference point. ### Claim 10 (depends on 9) The method of claim 9, wherein combining the predetermined 3D representation with each of the frame-specific 3D representations comprises adjusting at least some of the height values associated with the vertices based on the height values specified for the respective frame-specific 3D representation. ### Claim 11 (depends on 10) The method of claim 10, wherein adjusting the vertices comprises blending, using specified alpha values, at least some of the height values associated with the vertices with the height values specified for the respective the frame-specific 3D representation. ### Claim 12 (depends on 11) The method of claim 11, wherein adjusting the vertices comprises blending, using the specified alpha values, at least some texture values associated with the vertices with texture values specified for the respective frame-specific representation. ### Claim 13 (depends on 1) The method of claim 1, wherein the first portion represented by the predetermined 3D representation represents an upper body and a head of the user. ### Claim 14 (depends on 1) The method of claim 1, wherein the predetermined 3D representation is generated using a pixel-aligned implicit function (PI Fu) technique that locally aligns pixels of 2D enrollment images with a global context to form the predetermined 3D representation. ### Claim 15 (depends on 1) The method of claim 1, wherein each of the frame-specific 3D representations is generated based on sensor data from an inward or down-facing camera and enrollment data, wherein the enrollment data comprises images of the second portion of the user exhibiting a plurality of different facial expressions. ### Claim 16 (depends on 1) The method of claim 1, wherein each of the frame-specific 3D representations comprises a configuration of a skeletal representation of the user at a respective instant of the multiple instants in the period of time, wherein the 3D reference point is defined based on a node of the skeletal representation. ### Claim 17 (depends on 1) The method of claim 1, further comprising: rendering a view of the combined 3D representations. ### Claim 18 (depends on 17) The method of claim 17, wherein the rendering occurs during a communication session in which a second device captures sensor data and provides the sequence of frame-specific 3D representations corresponding to the multiple instants in the period time based on the sensor data. ### Claim 19 (depends on 18) The method of claim 18, wherein the second device provides the predetermined 3D representation of the user during the communication session. ### Claim 20 (depends on 18) The method of claim 18, wherein the second device generates the combined 3D representation of the user and provides the combined 3D representation to the device during the communication session. ### Claim 21 (depends on 1) The method of claim 1, wherein views of the combined 3D representations are displayed on the device in real-time relative to the multiple instants in the period of time. ### Claim 22 (depends on 1) The method of claim 1, wherein the device is a head-mounted device (HMD). ### Claim 23 A device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining a predetermined three-dimensional (3D) representation of at least a first portion of a user, wherein the predetermined 3D representation is associated with a 3D reference point defined relative to a skeletal representation of the user; obtaining a sequence of frame-specific 3D representations corresponding to multiple instants in a period of time, each of the frame-specific 3D representations representing a second portion of the user at a respective instant of the multiple instants in the period of time and each of the frame-specific 3D representations associated with the 3D reference point; and generating combined 3D representations of the user for the multiple instants in the period of time, wherein each of the combined 3D representations is generated by combining the predetermined 3D representation with a respective frame-specific 3D representation based on an alignment, wherein the alignment is based on the 3D reference point. ### Claim 24 (depends on 23) The device of claim 23, wherein the second portion of the user is a sub-portion of the first portion. ### Claim 25 A non-transitory computer-readable storage medium, storing program instructions executable on a device including one or more processors to perform operations comprising: obtaining a predetermined three-dimensional (3D) representation of at least a first portion of a user, wherein the predetermined 3D representation is associated with a 3D reference point defined relative to a skeletal representation of the user; obtaining a sequence of frame-specific 3D representations corresponding to multiple instants in a period of time, each of the frame-specific 3D representations representing a second portion of the user at a respective instant of the multiple instants in the period of time and each of the frame-specific 3D representations associated with the 3D reference point; and generating combined 3D representations of the user for the multiple instants in the period of time, wherein each of the combined 3D representations is generated by combining the predetermined 3D representation with a respective frame-specific 3D representation based on an alignment, wherein the alignment is based on the 3D reference point.