# US 2022/0044478 A1 · Photo-video based spatial-temporal volumetric capture system for dynamic 4d human face and body digitization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0044478 A1 - Document ID: US20220044478A1 - Status: Published application - Kind: A1 - Application number: 17/414,540 - Filed: 2019-12-20 - Published: 2022-02-10 - Earliest claimed: 2018-12-20 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Kenji Tashiro; Chuen-Chien Lee; Qing Zhang - Web page: https://patentgrep.com/patent/US20220044478A1 ## CPC - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/596: {from three or more stereo images} - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 15/04: Texture mapping - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 40/165: {using facial parts and geometric relationships} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)} - H04N 5/2625: {for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect (sequence generated by event triggered capturing H04N7/188)} - H04N 23/60: Control of cameras or camera modules - H04N 23/611: where the recognised objects include parts of the human body - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - G06T 2207/10004: Still image; Photographic image - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The photo-video based spatial-temporal volumetric capture system more efficiently, produces high frame rate and high resolution 4D dynamic human videos, without a need for 2 separate 3D and 4D scanner systems, by combining a set of high frame rate machine vision video cameras with a set of high resolution photography cameras. It reduces a need for manual CG works, by temporally up-sampling shape and texture resolution of 4D scanned video data from a temporally sparse set of higher resolution 3D scanned keyframes that are reconstructed both by using machine vision cameras and photography cameras. Unlike typical performance capture system that uses single static template model at initialization (e.g. A or pose), the photo-video based spatial-temporal volumetric capture system stores multiple keyframes of high resolution 3D template models for robust and dynamic shape and texture refinement of 4D scanned video sequence. For shape up-sampling, the system can apply mesh-tracking based temporal shape super resolution. For texture up-sampling, the system can apply machine learning based temporal texture super resolution. ## Claims ### Claim 1 A method comprising: capturing content using one or more photography cameras and one or more video cameras; triggering, with a device, the one or more photography cameras and the one or more video cameras to acquire one or more keyframes; and generating, with the device, one or more models based on the captured content and the one or more keyframes. ### Claim 2 (depends on 1) The method of claim 1 wherein capturing the content includes capturing dynamic facial expressions and/or dynamic body actions. ### Claim 3 (depends on 1) The method of claim 1 wherein triggering includes utilizing trigger timing to simultaneously generating triggers to the one or more photography cameras and the one or more video cameras. ### Claim 4 (depends on 3) The method of claim 3 wherein trigger tinning of the one or more photography. cameras includes periodic triggering. ### Claim 5 (depends on 3) The method of claim 3 wherein trigger timing of the one or more photography cameras includes utilizing a human observer to detects a specific facial and/or body expression. ### Claim 6 (depends on 3) The method of claim 3 wherein trigger timing of the one or more photography cameras includes automatically detecting a specific facial or body expression by the device, the one or more photography cameras and/or the one or more video cameras. ### Claim 7 (depends on 1) The method of claim 1 wherein the one or more models are used to implement: mesh-tracking based temporal shape super-resolution on low-resolution but high frame-rate video-based 4D scanned volumetric sequence refined by using 3D scanned high-resolution templates at multiple keyframes, captured by both the one or more photography cameras and the one or more video cameras, for recovering high-resolution surface dynamics in a long action sequence. ### Claim 8 (depends on 1) The method of claim 1 wherein the one or more models are used to implement: machine learning based temporal texture super-resolution on low-resolution but high frame-rate video-based 4D scanned volumetric sequence refined by training high-resolution and low-resolution UV texture pairs at multiple keyframes, at each keyframe, a high-resolution UV texture map is generated using a photography camera's image data, while a low-resolution UV texture map is generated using a video camera's image data, and a same mesh topology is used such that textures are collocated between high and low resolution UV texture pairs for higher training efficiency. ### Claim 9 (depends on 1) The method of claim 1 further comprising generating computer graphics using the content and the one or more models including the facial expressions and body actions, wherein generating the computer graphics includes utilizing keyframes of the content and non-keyframes of the content to transition between the keyframes. ### Claim 10 An apparatus comprising: a non-transitory memory for storing an application, the application for: implementing triggering to acquire one or more keyframes, wherein trigger signals are sent to one or more photography cameras and one or more video cameras; and generating one or more models based on the one or more keyframes and content captured from the one or more photography cameras and the one or more video cameras; and a processor coupled to the memory, the processor configured for processing the application. ### Claim 11 (depends on 10) The apparatus of claim 10 wherein triggering includes utilizing trigger timing to simultaneously generating triggers to the one or more photography cameras and the one or more video cameras. ### Claim 12 (depends on 11) The apparatus of claim 11 wherein trigger timing of the one or more photography cameras includes periodic triggering. ### Claim 13 (depends on 11) The apparatus of claim 11 wherein trigger timing of the one or more photography cameras includes utilizing a human observer to detects a specific facial and/or body expression. ### Claim 14 (depends on 11) The apparatus of claim 11 wherein trigger timing of the one or more photography cameras includes automatically detecting a specific facial or body expression by the device, the one or more photography cameras and/or the one or more video cameras. ### Claim 15 (depends on 10) The apparatus of claim 10 wherein the one or more models are used to implement: mesh-tracking based temporal shape super-resolution on low-resolution but high frame-rate video-based 4D scanned volumetric sequence refined by using high-resolution 3D scanned templates at multiple keyframes, captured by both the one or more photography cameras and the one or more video cameras, for recovering high-resolution surface dynamics in a long action sequence. ### Claim 16 (depends on 10) The apparatus of claim 10 wherein the one or more models are used to implement: machine learning based temporal texture super-resolution on low-resolution but high frame-rate video-based 4D scanned volumetric sequence refined by training high-resolution and low-resolution UV texture pairs at multiple keyframes, at each keyframe, a high-resolution UV texture map is generated using a photography camera's image data, while a low-resolution UV texture map is generated using a video camera's image data, and a same mesh topology is used such that textures are collocated between high and low resolution UV texture pairs for higher training efficiency. ### Claim 17 (depends on 10) The apparatus of claim 10 wherein the application is further configured for generating computer graphics using the content and the one or more models including the facial expressions and body actions, wherein generating the computer graphics includes utilizing keyframes of the content and non-keyframes of the content to transition between the keyframes. ### Claim 18 A system comprising: a first set of devices configured for capturing image content of a subject; a second set of devices configured for capturing video content of the subject; and a computing device configured for: implementing triggering to acquire one or more keyframes based on the image content and the video content; and generating one or more models based on the image content and the video content and the one or more keyframes. ### Claim 19 (depends on 18) The system of claim 18 wherein capturing the content includes capturing facial expressions and/or body actions. ### Claim 20 (depends on 18) The system of claim 18 wherein triggering includes utilizing trigger timing to simultaneously generating triggers to the one or more photography cameras and the one or more video cameras. ### Claim 21 (depends on 20) The system of claim 20 wherein trigger timing of the one or more photography cameras includes periodic triggering. cameras includes utilizing a human observer to detects a specific facial and/or body expression. ### Claim 23 (depends on 20) The system of claim 20 wherein trigger timing of the one or more photography cameras includes automatically detecting a specific facial or body expression by the computing device, the first set of devices and/or the second set of devices. ### Claim 24 (depends on 18) The system of claim 18 wherein the one or more models are used to implement: mesh-tracking based temporal shape super-resolution on low-resolution but high frame-rate video-based 4D scanned volumetric sequence refined by using high-resolution 3D scanned templates at multiple keyframes, captured by both the one or more photography cameras and the one or more video cameras, for recovering high-resolution surface dynamics in a long action sequence. ### Claim 25 (depends on 18) The system of claim 18 wherein the one or more models are used to implement: machine learning based temporal texture super-resolution on low-resolution but high frame-rate video-based 4D scanned volumetric sequence refined by training high-resolution and low-resolution UV texture pairs at multiple keyframes, at each keyframe, a high-resolution UV texture map is generated using a photography camera's image data, while a low-resolution UV texture map is generated using video camera's image data, and a same mesh topology is used such that textures are collocated between high and low resolution UV texture pairs for higher training efficiency. ### Claim 26 (depends on 18) The system of claim 18 wherein the computing device is further configured for generating computer graphics using the image content and the video content and the one or more models including facial expressions and body actions, wherein generating the computer graphics includes utilizing keyframes of the content and non-keyframes of the content to transition between the keyframes.