# US 12,192,679 B2 · Updating 3D models of persons > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,679 B2 - Document ID: US12192679B2 - Status: Granted - Kind: B2 - Application number: 17/249,469 - Filed: 2021-03-02 - Published: 2021-11-18 - Granted: 2025-01-07 - Earliest claimed: 2020-05-12 - Assignees: Truemeeting, Ltd - Applicants: Truemeeting, Ltd - Inventors: Ran Oz; Yuval Gronau; Michael Rabinovich; Osnat Goren-Peyser; Tal Perl; Erez Posner - Web page: https://patentgrep.com/patent/US12192679B2 ## CPC - H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 7/11: Region-based segmentation - G06T 7/251: {involving models} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 15/04: Texture mapping - G06T 15/20: Perspective computation - G06T 15/205: {Image-based rendering} - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - H04N 7/144: {camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact} - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - H04N 7/152: {Multipoint control units therefor} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G06T 2219/2004: Aligning objects, relative positioning of parts - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method for updating a current three dimensional (3D) model of a person, that method may include calculating current locations, within a two-dimensional (2D) space, of current face landmark points of a face of a person within a first image; the calculating is based on the current 3D model, and one or more current acquisition parameters of a 2D camera; wherein the current 3D model of the person is located within a 3D space; calculating second locations, within the 2D space, of second face landmark points of the face of the person within a second image that follows the first image; calculating correspondences between the current locations and the second locations; calculating, based on the correspondences, locations of the second face landmark points within the 3D space; and modifying the current 3D model based on the locations of the second face landmark points within the 3D space. ## Claims ### Claim 1 A method for updating a current three dimensional (3D) model of a person, comprising: (a) participating, using a computerized system of the person, in a virtual 3D video conference; (b) calculating current locations, within a two-dimensional (2D) space, of current face landmark points of a face of a person within a first image; the calculating is based on the current 3D model, and one or more current acquisition parameters of a 2D camera; wherein the current 3D model of the person is located within a 3D space; (c) calculating second locations, within the 2D space, of second face landmark points of the face of the person within a second image that follows the first image; (d) calculating correspondences between the current locations and the second locations; (e) calculating, based on the correspondences, locations of the second face landmark points within the 3D space; and (f) modifying the current 3D model based on the locations of the second face landmark points within the 3D space; wherein steps (b)-(f) are also executed by the computerized device of the person; wherein step (a) also comprises capturing the first image and the second image by a camera of the computerized device of the person; wherein the method further comprises: determining parameters of the current 3D model of the person; wherein the parameters differ from image pixels and there are less than three hundred parameters of the current 3D model; transmitting the parameters of the current 3D model, without transmitting the current 3D model, from the computerized device of the person to computerized devices of other participants of the virtual 3D video conference; wherein the transmitting comprises reducing latency by directly transmitting the parameters to the computerized devices of other participants without sending the parameters via a central server; and reconstructing an avatar of the person by the computerized devices of the other participants based on the parameters of the current 3D model; wherein the transmitting the parameters of the current 3D model is preceded by selecting the parameters of the current 3D model, out of a larger group of parameters of the current 3D model, based on (i) an available bandwidth allocated for transmission and (ii) a contribution of the parameters of the current 3D model to a visual perception of the face of the person; where the available bandwidth is several hundred bits per second. ### Claim 2 (depends on 1) The method according to claim 1 wherein the current face landmark points are edge points of the current face landmarks. ### Claim 3 (depends on 1) The method according to claim 1 wherein the current face landmark points are edge points of the current face landmarks and non-edge points of the current face landmarks. ### Claim 4 (depends on 1) The method according to claim 1 wherein the current 3D model comprises a reference 3D model and a current 3D deformation model, wherein the modifying of the current 3D model comprises modifying the current 3D deformation model without substantially modifying the reference 3D model. ### Claim 5 (depends on 4) The method according to claim 4, comprising determining an update frequency of the 3D model based on the bandwidth. ### Claim 6 (depends on 1) The method according to claim 1 further comprising allocating bitrate, based on the available bandwidth, for a transmission of audio and for the transmission of the parameters of the current 3D model. ### Claim 7 (depends on 1) The method according to claim 1 wherein step (c) comprises segmentation. ### Claim 8 (depends on 1) The method according to claim 1 wherein the parameters define a shape, an expression and a pose of the person. ### Claim 9 (depends on 1) The method according to claim 1 wherein the user computerized device is a laptop without a graphics processing unit (GPU). ### Claim 10 (depends on 1) The method according to claim 1, wherein the user computerized device is mobile phone. ### Claim 11 (depends on 1) The method according to claim 1, wherein the parameters are ordered according to importance of the parameters. ### Claim 12 (depends on 1) The method according to claim 1, wherein a number of bits per parameter is determined according to the available bandwidth allocated to the transmission. ### Claim 13 A non-transitory computer readable medium for updating a current three dimensional (3D) model of a person, the non-transitory computer readable medium stores instructions for: (a) participating, using a computerized system of the person, in a virtual 3D video conference; (b) calculating current locations, within a two-dimensional (2D) space, of current face landmark points of a face of a person within a first image; the calculating is based on the current 3D model, and one or more current acquisition parameters of a 2D camera; wherein the current 3D model of the person is located within a 3D space; (c) calculating second locations, within the 2D space, of second face landmark points of the face of the person within a second image that follows the first image; (d) calculating correspondences between the current locations and the second locations; (e) calculating, based on the correspondences, locations of the second face landmark points within the 3D space; and (f) modifying the current 3D model based on the locations of the second face landmark points within the 3D space; wherein steps (b)-(f) are also executed by the computerized device of the person; wherein step (a) also comprises capturing the first image and the second image by a camera of the computerized device of the person; wherein the non-transitory computer readable medium further stores instructions for: determining parameters of the current 3D model of the person; wherein the parameters differ from image pixels and there are less than three hundred parameters of the current 3D model; transmitting the parameters of the current 3D model, without transmitting the current 3D model, from the computerized device of the person to computerized devices of other participants of the virtual 3D video conference; wherein the transmitting comprises reducing latency by directly transmitting the parameters to the computerized devices of other participants without sending the parameters via a central server; and reconstructing an avatar of the person by the computerized devices of the other participants based on the parameters of the current 3D model; wherein the transmitting the parameters of the current 3D model is preceded by selecting the parameters of the current 3D model, out of a larger group of parameters of the current 3D model, based on (i) an available bandwidth allocated for transmission and (ii) a contribution of parameters of the current 3D model to a visual perception of the face of the person; and wherein the available bandwidth is several hundred bandwidths per second. ### Claim 14 (depends on 13) The non-transitory computer readable medium according to claim 13 wherein the current face landmark points are edge points of the current face landmarks. ### Claim 15 (depends on 13) The non-transitory computer readable medium according to claim 13 wherein the current face landmark points are edge points of the current face landmarks and non-edge points of the current face landmarks. ### Claim 16 (depends on 13) The non-transitory computer readable medium according to claim 13 wherein the current 3D model comprises a reference 3D model and a current 3D deformation model, wherein the modifying of the current 3D model comprises modifying the current 3D deformation model without substantially modifying the reference 3D model. ### Claim 17 (depends on 16) The non-transitory computer readable medium according to claim 16, comprising determining an update frequency of the 3D model based on the bandwidth. ### Claim 18 (depends on 13) The non-transitory computer readable medium according to claim 13 that stores instructions for allocating bitrate, based on the available bandwidth, for a transmission of audio and for the transmission of the parameters of the current 3D model. ### Claim 19 (depends on 13) The non-transitory computer readable medium according to claim 13 wherein step (b) comprises segmentation. ### Claim 20 (depends on 13) The non-transitory computer readable medium according to claim 13 wherein the parameters define a shape, an expression and a pose of the person. ### Claim 21 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the user computerized device is a laptop without a graphics processing unit (GPU).