# US 12,008,701 B2 · Computing system and method for rendering avatars > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,701 B2 - Document ID: US12008701B2 - Status: Granted - Kind: B2 - Application number: 17/806,701 - Filed: 2022-06-13 - Published: 2023-12-14 - Granted: 2024-06-11 - Earliest claimed: 2022-06-13 - Assignees: Lemon Inc. - Applicants: Lemon Inc. - Inventors: Jie Meng; Tingwen Liao; Maryyann Crichton; Siding Chen; Vishnuvardhan Tanguturi - Web page: https://patentgrep.com/patent/US12008701B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 13/40: of characters, e.g. humans, animals or virtual beings - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/52: involving aspects of the displayed game scene - G06F 3/012: {Head tracking input arrangements} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/73: using feature-based methods - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)} - A63F 2300/204: the platform being a handheld device - A63F 2300/5553: user representation in the game field, e.g. avatar - A63F 2300/572: Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video - G06T 2207/30201: Face - G06T 2219/024: Multi-user, collaborative environment - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A computing system is provided comprising a processor is configured to execute an augmented reality avatar program to receive a second-user avatar-data object and second-user pose information, via a computer network, receive an image from a live image feed captured by a camera of the computing device, identify a virtual surface corresponding to a physical surface that appears in the image; calculate a position and a pose of a second-user avatar relative to the virtual surface based on the second-user pose information, render the second-user avatar-data object to thereby display the second-user avatar of the second user superimposed upon the image. The second-user avatar is anchored to the virtual surface according to the calculated position and pose of the second-user avatar. The virtual surface is world-locked to the physical surface in the image in a real world 3-D environment. ## Claims ### Claim 1 A computing system, comprising: a processor and memory of a computing device of a first user, the processor being configured to execute an augmented reality (AR) avatar program using portions of memory to: receive a second-user avatar-data object and second-user pose information of a second user, via a computer network; receive an image from a live image feed captured by a camera of the computing device; identify a virtual surface corresponding to a physical surface that appears in the image; calculate a position and a pose of a second-user avatar relative to the virtual surface based on the second-user pose information; render the second-user avatar-data object to thereby display the second-user avatar of the second user superimposed upon the image, the second-user avatar being anchored to the virtual surface and posed according to the calculated position and pose of the second-user avatar, the virtual surface being world-locked to the physical surface in the image in a real world three-dimensional environment. ### Claim 2 (depends on 1) The computing system of claim 1, wherein the second-user pose information is initial second-user pose information of a session of the AR avatar program; and the image is an initial image of the session of the AR avatar program. ### Claim 3 (depends on 1) The computing system of claim 1, wherein the AR avatar program is executed to, in a loop: receive updated second-user pose information of the second user via the computer network; receive an updated image from the live image feed captured by the camera of the computing device; calculate a position and a pose of the second-user avatar relative to the virtual surface based on the updated second-user pose information; and render the second-user avatar-data object to thereby display the second-user avatar superimposed upon the updated image and posed according to the updated pose information on each pass through the loop while the second-user avatar remains anchored to the virtual surface, which remains world locked to the physical surface in the image in the real world three-dimensional environment. ### Claim 4 (depends on 3) The computing system of claim 3, wherein the second-user pose information and the updated second-user pose information include second face tracking data; the processor is configured to render the second-user avatar-data object of the second user based on the second face tracking data to thereby display the second-user avatar of the second user posed with a face which corresponds to a facial expression of the second user; and the face of the second user avatar is rendered so as to change appearance as the second face tracking data changes in successive passes through the loop. ### Claim 5 (depends on 1) The computing system of claim 1, wherein the second-user pose information includes second face tracking data; and the processor is configured to render the second-user avatar-data object of the second user based on the second face tracking data to thereby display the second-user avatar of the second user posed with a face which corresponds to a facial expression of the second user. ### Claim 6 (depends on 1) The computing system of claim 1, wherein the second-user pose information includes second skeletal tracking data; and the processor is configured to render a body of the second-user avatar so as to change appearance as the second skeletal tracking data changes. ### Claim 7 (depends on 1) The computing system of claim 1, wherein the camera is a rear-facing camera and the computing device is a first computing device, the computing system further comprising: a front-facing camera configured to capture a first-user image of the first user, wherein the processor is configured to: identify tracking features in the first-user image of the first user, and generate first-user pose information of the first user based on the identified tracking features; and send the first-user pose information to a second computing device of the second user via the computer network. ### Claim 8 (depends on 7) The computing system of claim 7, wherein the first-user pose information includes first face tracking data; and the processor is configured to render a first-user avatar-data object of the first user based on the first face tracking data to thereby display the first-user avatar of the first user posed with a face which corresponds to a facial expression of the first user. ### Claim 9 (depends on 7) The computing system of claim 7, wherein the processor is further configured to render a first-user avatar-data object of the first user to thereby display the first-user avatar of the first user superimposed upon the image, the first-user avatar posed according to the first-user pose information. ### Claim 10 (depends on 9) The computing system of claim 9, wherein the front-facing camera and the rear-facing camera are operated to capture the first-user image and the image, respectively, in real-time with time stamps syncing the identified tracking features in the image with the identified tracking features in the first-user image of the first user. ### Claim 11 (depends on 7) The computing system of claim 7, wherein the processor is configured to: identify tracking features in the image captured by the rear-facing camera and the first-user image of the first user; generate first-user pose information of the first user based on the identified tracking features; and send the first-user pose information to the second computing device of the second user via the computer network. ### Claim 12 (depends on 1) The computing system of claim 1, wherein the computing device is a first computing device; the first user and the second user each have associated accounts on a social media platform; the first computing device and a second computing device each execute an instance of the AR avatar program that is configured to communicate with a server computing system of the social media platform; and a first-user pose information of the first user, a first-user avatar-data object of the first user, the second-user pose information, and the second-user avatar-data object are exchanged between the AR avatar program executed on the first computing device and the AR avatar program executed on the second computing device via the server computing system. ### Claim 13 A method for a client program executed on a computing device operated by a first user, the method comprising steps to: receive a second-user avatar-data object and second-user pose information of a second user, via a computer network; receive an image from a live image feed captured by a camera of the computing device; identify a virtual surface corresponding to a physical surface that appears in the image; calculate a position and a pose of a second-user avatar relative to the virtual surface based on the second-user pose information; and render the second-user avatar-data object to thereby display the second-user avatar of the second user superimposed upon the image, the second-user avatar being anchored to the virtual surface according to the calculated position and pose of the second-user avatar, the virtual surface being world-locked to the physical surface in the image in a real world three-dimensional environment. ### Claim 14 (depends on 13) The method of claim 13, further comprising steps to, in a loop: receive updated second-user pose information of the second user via the computer network; receive an updated image from the live image feed captured by the camera of the computing device; calculate a position and a pose of the second-user avatar relative to the virtual surface based on the updated second-user pose information; and render the second-user avatar-data object to thereby display the second-user avatar of the second user superimposed upon the updated image, the second-user avatar posed according to the updated second-user pose information on each pass through the loop while the second-user avatar remains anchored to the virtual surface, which remains world-locked to the physical surface in the image in the real world three-dimensional environment. ### Claim 15 (depends on 13) The method of claim 13, wherein the second-user pose information includes second face tracking data; and the method further comprises rendering the second-user avatar-data object of the second user based on the second face tracking data to thereby display the second-user avatar of the second user posed with a face which corresponds to a facial expression of the second user. ### Claim 16 (depends on 13) The method of claim 13, wherein the computing device is a first computing device; and the method further comprises steps to: capture a first-user image of the first user; identify tracking features in the first-user image of the first user; generate first-user pose information of the first user based on the identified tracking features; and send the first-user pose information to a second computing device of the second user via the computer network. ### Claim 17 (depends on 16) The method of claim 16, wherein the first-user pose information includes first face tracking data; and the method further comprises rendering a first-user avatar-data object of the first user based on the first face tracking data to thereby display the first-user avatar of the first user posed with a face which corresponds to a facial expression of the first user. ### Claim 18 A server computing device, comprising: a processor; a non-volatile storage device operatively coupled to the processor; and an avatar hosting program stored in the non-volatile storage device and executed by the processor of the server computing device, wherein the avatar hosting program is configured to host an avatar session to: receive and store a second-user avatar-data object and second-user pose information of a second user from a second computing device; responsive to receiving a first request from a first computing device for a second-user avatar, send the second-user avatar-data object and the second-user pose information to the first computing device, thereby causing the first computing device to render the second-user avatar-data object and display the second-user avatar; receive and store a first-user avatar-data object and initial first-user pose information of a first user from the first computing device; responsive to receiving a second request from the second computing device for a first-user avatar, send the first-user avatar-data object and the initial first-user pose information to the second computing device, thereby causing the second computing device to render the first-user avatar-data object and display the first-user avatar; and in a loop: receive updated second-user pose information of the second user from the second computing device; and responsive to receiving the updated second-user pose information, send the updated second-user pose information to the first computing device, thereby causing the first computing device to render the second-user avatar-data object using the updated second-user pose information and display the second-user avatar; receive updated first-user pose information of the first user from the first computing device; and responsive to receiving the updated first-user pose information, send the updated first-user pose information to the second computing device, thereby causing the second computing device to render the first-user avatar-data object using the updated first-user pose information and display the first-user avatar. ### Claim 19 (depends on 18) The server computing device of claim 18, wherein the initial first-user pose information and the updated first-user pose information include first face tracking data; the second computing device is caused to render the first-user avatar-data object of the first user based on the first face tracking data to thereby display the first-user avatar of the first user posed with a face which corresponds to a facial expression of the first user; the second-user pose information and the updated second-user pose information include second face tracking data; and the first computing device is caused to render the second-user avatar-data object of the second user based on the second face tracking data to thereby display the second-user avatar of the second user posed with a face which corresponds to a facial expression of the second user. ### Claim 20 (depends on 18) The server computing device of claim 18, wherein the initial first-user pose information and the updated first-user pose information include first skeletal tracking data; the second computing device is caused to render the first-user avatar-data object of the first user based on the first skeletal tracking data to thereby display the first-user avatar of the first user posed with a body which changes appearance as the first skeletal tracking data changes in successive passes through the loop; the second-user pose information and the updated second-user pose information include second skeletal tracking data; and the first computing device is caused to render the second-user avatar-data object based on the second skeletal tracking data of the second user to thereby display the second-user avatar of the second user posed with a body which changes appearance as the second skeletal tracking data changes in successive passes through the loop.