# US 11,998,819 B2 · Virtual environment construction apparatus, video presentation apparatus, model learning apparatus, optimal depth decision apparatus, methods for the same, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,998,819 B2 - Document ID: US11998819B2 - Status: Granted - Kind: B2 - Application number: 17/845,010 - Filed: 2022-06-21 - Published: 2022-10-06 - Granted: 2024-06-04 - Earliest claimed: 2016-01-28 - Assignees: Nippon Telegraph and Telephone Corporation - Applicants: Nippon Telegraph and Telephone Corporation - Inventors: Kosuke Takahashi; Dan Mikami; Mariko Isogawa; Akira Kojima; Hideaki Kimata; Ayumi Matsumoto - Web page: https://patentgrep.com/patent/US11998819B2 ## CPC - A63B 69/0002: {for baseball} - G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - A63B 2069/0006: {for pitching} - A63B 2069/0008: {for batting} - A63B: APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT (apparatus for passive exercising, massage A61H) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A virtual reality system is provided that includes a video presentation apparatus, the virtual reality system comprising: the video presentation apparatus which includes circuitry configured to: obtain as input a video sequence composed of a plurality of frames and mask information specifying a complementation target region in the video sequence; separate a frame into a foreground region and a background region based on binary images representing differences between the plurality of frames included in the video sequence; determine either one of patch-search-based completion and paste synthesis as a complementation method for the complementation target region based on a number of pixels belonging to the foreground region and located within a given distance from a periphery of the complementation target region; and complement the complementation target region in accordance with the complementation method; and a virtual reality head mounted display which presents the complemented video sequence to the user. ## Claims ### Claim 1 A virtual reality system that includes a virtual environment construction apparatus that constructs a virtual environment to be used for generating a video based on a real environment in which a plurality of objects are present, the virtual reality system comprising: the virtual environment construction apparatus which includes circuitry configured to for at least one of the plurality of objects as a target object, decide a model construction method indicating whether a model of the object is constructed with computer graphics (CG) or with a live-action video based on at least one of a sensing difficulty level and a CG drawing difficulty level of the target object, measure a state of the target object in the real environment, construct an environment model for drawing in the virtual environment in accordance with the model construction method for a static object which does not involve temporal change, and construct a model for drawing in the virtual environment in accordance with the model construction method for a dynamic object which involves temporal change, and synthesizes the model with the environment model; and a virtual reality head mounted display which presents a video to the user based on the synthesized model constructed by the virtual environment construction apparatus. ### Claim 2 (depends on 1) A non-transitory computer readable recording medium on which a program recorded thereon for causing a computer to function as the virtual environment construction apparatus according to claim 1. ### Claim 3 (depends on 1) The virtual environment construction apparatus according to claim 1, wherein when SE is the sensing difficulty level, CM is the CG drawing difficulty level, θmax is a discrepancy between an angle of observation and an angle of viewing, th is a predefined threshold, and α, β, and γ are predefined parameters, the model construction method decision unit decides the model construction method based on a formula γθmax/(αSE*βCM)>th. ### Claim 4 (depends on 3) The virtual environment construction apparatus according to claim 3, wherein if a dynamic object whose model is constructed with CG has been pictured on an object whose model is constructed with live-action video, the dynamic object construction unit uses the live-action video modified so as not to contain the dynamic object instead of the live-action video, for a region of the live-action video where the dynamic object has been pictured. ### Claim 5 (depends on 4) A non-transitory computer readable recording medium on which a program recorded thereon for causing a computer to function as the virtual environment construction apparatus according to claim 4. ### Claim 6 (depends on 3) A non-transitory computer readable recording medium on which a program recorded thereon for causing a computer to function as the virtual environment construction apparatus according to claim 3. ### Claim 7 (depends on 1) The virtual environment construction apparatus according to claim 1, wherein if a dynamic object whose model is constructed with CG has been pictured on an object whose model is constructed with live-action video, the dynamic object construction unit uses the live-action video modified so as not to contain the dynamic object instead of the live-action video, for a region of the live-action video where the dynamic object has been pictured. ### Claim 8 (depends on 7) A non-transitory computer readable recording medium on which a program recorded thereon for causing a computer to function as the virtual environment construction apparatus according to claim 7. ### Claim 9 A virtual environment construction method for constructing a virtual environment for viewing by a user based on a real environment in which a plurality of objects are present, the virtual environment construction method comprising: for at least one of the plurality of objects as a target object, deciding a model construction method indicating whether a model of the object is constructed with computer graphics (CG) or with a live-action video based on at least one of a sensing difficulty level and a CG drawing difficulty level of the target object by processing circuitry of a virtual environment construction apparatus; measuring a state of the target object in the real environment by the processing circuitry of the virtual environment construction apparatus; constructing an environment model for drawing in the virtual environment in accordance with the model construction method for a static object which does not involve temporal change by the processing circuitry of the virtual environment construction apparatus; and constructing a model for drawing in the virtual environment in accordance with the model construction method for a dynamic object which involves temporal change, and synthesizing the model with the environment model, by the processing circuitry of the virtual environment construction apparatus presenting a video to the user based on the synthesized model, by a virtual reality head mounted display.