# US 2019/0172497 A1 · Video enhancement > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0172497 A1 - Document ID: US20190172497A1 - Status: Published application - Kind: A1 - Application number: 16/272,313 - Filed: 2019-02-11 - Published: 2019-06-06 - Earliest claimed: 2015-11-18 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Anthony L. COLE; Thomas J. DAVISON; Daniel DEL PICCOLO; Daniel LANE; James S. LUKE; Martine M. PULVENIS - Web page: https://patentgrep.com/patent/US20190172497A1 ## CPC - G11B 27/036: Insert-editing - G06T 7/13: Edge detection - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 11/60: Creating or editing images; Combining images with text - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - H04N 7/142: {Constructional details of the terminal equipment, e.g. arrangements of the camera and the display} - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Methods, systems, and computer program products for generating revised videos. The method includes, for example, receiving, by one or more processor, first video data of an environment having at least one projection on a screen from a first point of view, and generating, by the one or more processor, revised video data of the environment having the at least one projection on the screen from the first point of view based on the first video data, the revised video data comprising a plurality of frames with the at least one projection on the screen disposed in the first area relative to the environment in a plurality of frames being revised based on data for projecting the at least one projection on the screen in the first video data. ## Claims ### Claim 1 A computer-implemented method comprising: receiving first video data of an environment having an object observed from a first point of view, the first video data comprising a plurality of frames having the object disposed in an area of the plurality of frames; and generating revised video data of the environment having the object from the first point of view based on the first video data, the revised video data comprising a plurality of revised frames with the object disposed in the area of the plurality of frames being revised based on data corresponding to the object separate from the first video data, wherein the first video data of the environment comprises the object comprising a virtual reality headset, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset during the receiving the first video data. ### Claim 2 (depends on 1) The computer-implemented method of claim 1 wherein the environment is a virtual reality conference, wherein the generating comprises synchronizing the second video data obtained of the environment behind the virtual reality headset during the receiving the first video data with the first video data. ### Claim 3 (depends on 1) The computer-implemented method of claim 1 wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by a respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind a respective virtual reality headset of each of the plurality of participants during the receiving the first video data. ### Claim 4 (depends on 1) The computer-implemented method of claim 1 wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data, the second video being obtained using cameras located with the respective virtual reality headsets of the plurality of participants. ### Claim 5 (depends on 1) The computer-implemented method of claim 1 wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data, the second video being obtained using cameras located with the respective virtual reality headsets of the plurality of participants, wherein the method is performed so that when a certain participant of the plurality of participants turns his or her head, a boundary of a particular virtual reality headset of the plurality of virtual reality headsets changes in the first video data, wherein the method is performed so that the second video data obtained using cameras located within respective virtual reality headsets of the plurality of participants is used to overlay live facial expressions of the respective participants on virtual reality headset portions of the first video data. ### Claim 6 (depends on 1) The computer-implemented method of claim 1 wherein the receiving and generating are performed in real time, wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the first video data represents a changing viewing point and is obtained using a front facing camera facing forward of a certain participant of the plurality of participants, wherein the method is performed so that participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data, the second video being obtained using cameras located within the respective virtual reality headsets of the plurality of participants, wherein the generating comprises synchronizing the second video data obtained of the environment behind the respective virtual reality headsets of the plurality of participants during the receiving the first video data with the first video data, the first video data being a live feed of video frames, wherein the method is performed so that when the certain participant turns his or her head, a boundary of a particular virtual reality headset of the plurality of virtual reality headsets changes in the first video data, wherein the method is performed so that the second video data obtained using cameras located within respective virtual reality headsets of the plurality of participants is used to overlay live facial expressions of the respective participants on virtual reality headset portions of the first video data. ### Claim 7 (depends on 1) The computer-implemented method of claim 1 wherein the generating comprises identifying the object in the frame and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, and the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprise identifying the area of the object in one of the frames of the plurality of frames, and using the identified area in the one of the frames for identifying the area of the object in a subsequent frame. ### Claim 8 (depends on 1) The computer-implemented method of claim 1 wherein the generating comprises identifying the object in the frame and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, and the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprise identifying the area of the object in one of the frames of the plurality of frames, and using the identified area in the one of the frames for identifying the area of the object in a subsequent frame, wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data. ### Claim 9 (depends on 1) The computer-implemented method of claim 1 wherein the generating comprises identifying the object in the plurality of frames and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the generating comprises synchronizing the second video data obtained of the environment behind the virtual reality headset during the receiving the first video data with the first video data. ### Claim 10 (depends on 1) The computer-implemented method of claim 1 wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, wherein the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprise identifying the area of the object in one of the frames of the plurality of frames, and using the identified area in the one of the frames for identifying the area of the object in a subsequent frame. ### Claim 11 A computer program product comprising: a computer readable storage medium readable by one or more processing circuit and storing instructions for execution by one or more processor for performing a method comprising: receiving first video data of an environment having an object observed from a first point of view, the first video data comprising a plurality of frames having the object disposed in an area of the plurality of frames; and generating revised video data of the environment having the object from the first point of view based on the first video data, the revised video data comprising a plurality of revised frames with the object disposed in the area of the plurality of frames being revised based on data corresponding to the object separate from the first video data, wherein the first video data of the environment comprises the object comprising a virtual reality headset, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset during the receiving the first video data. ### Claim 12 (depends on 11) The computer program product of claim 11 wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data. ### Claim 13 (depends on 11) The computer program product of claim 11 wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data, the second video being obtained using cameras located with the respective virtual reality headsets of the plurality of participants, wherein the method is performed so that when a certain participant of the plurality of participants turns his or her head, a boundary of a particular virtual reality headset of the plurality of virtual reality headsets changes in the first video data, wherein the method is performed so that the second video data obtained using cameras located within respective virtual reality headsets of the plurality of participants is used to overlay live facial expressions of the respective participants on virtual reality headset portions of the first video data. ### Claim 14 (depends on 11) The computer program product of claim 11 wherein the receiving and generating are performed in real time, wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the first video data represents a changing viewing point and is obtained using a front facing camera facing forward of a certain participant of the plurality of participants, wherein the method is performed so that participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data, the second video being obtained using cameras located within the respective virtual reality headsets of the plurality of participants, wherein the generating comprises synchronizing the second video data obtained of the environment behind the respective virtual reality headsets of the plurality of participants during the receiving the first video data with the first video data, the first video data being a live feed of video frames, wherein the method is performed so that when the certain participant turns his or her head, a boundary of a particular virtual reality headset of the plurality of virtual reality headsets changes in the first video data, wherein the method is performed so that the second video data obtained using cameras located within the respective virtual reality headsets of the plurality of participants is used to overlay live facial expressions of the respective participants on virtual reality headset portions of the first video data. ### Claim 15 (depends on 11) The computer program product of claim 11 wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets wherein the generating comprises identifying the object in the frame and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, and the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprise identifying the area of the object in one of the frames of the plurality of frames, and using the identified area in the one of the frames for identifying the area of the object in a subsequent frame. ### Claim 16 (depends on 11) The computer program product of claim 11 wherein the generating comprises identifying the object in the plurality of frames and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, and the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprises identifying the area of the object in one of the frames of the plurality of frames, and using the identified area in the one of the frames for identifying the area of the object in a subsequent frame, wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by the respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data. ### Claim 17 A system comprising: a memory; one or more processor in communication with the memory; and program instructions executable by the one or more processor via the memory to perform a method for controlling display of an image on a display screen, the method comprising: receiving first video data of an environment having an object observed from a first point of view, the first video data comprising a plurality of frames having the object disposed in an area of the plurality of frames; and generating revised video data of the environment having the object from the first point of view based on the first video data, the revised video data comprising a plurality of revised frames with the object disposed in the area of the plurality of frames being revised based on data corresponding to the object separate from the first video data. ### Claim 18 (depends on 17) The system of claim 17, wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the generating comprises identifying the object in the plurality of frames and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, and the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprise identifying the area of the object in one of the frames of the plurality of frames, and using the identified area in the one of the frames for identifying the area of the object in a subsequent frame. ### Claim 19 (depends on 17) The system of claim 17, wherein the generating comprises identifying the object in the plurality of frames and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, and the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprises identifying the area of the object in one frame of the plurality of frames, and using the identified area in the one frame for identifying the area of the object in a subsequent frame, wherein the environment is a virtual reality conference, wherein a plurality of participants of the virtual reality conference are wearing virtual reality headsets, wherein the method is performed so that participants of the plurality of participants see each other in virtual reality space, wherein facial features of the plurality of the participants are hidden by respective virtual reality headsets of the plurality of participants, wherein the first video data of the environment comprises the object comprising a virtual reality headset of each of the plurality of participants, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset of each of the plurality of participants during the receiving the first video data. ### Claim 20 (depends on 17) The system of claim 17, wherein the generating comprises identifying the object in the frame and retrieving the data corresponding to the object from a database based on the identified object, wherein the data corresponding to the object separate from the first video data comprises video data obtained of the environment behind the object during the receiving the first video data, wherein the receiving and the generating are performed in real-time, wherein the generating comprises determining a peripheral edge of the area of the object in the plurality of frames, and the revised video data comprises a revised object based on the determined peripheral edge, wherein the data corresponding to the object separate from the first video data comprises the object from a second point of view, and the generating comprises reorienting the object from the second point of view to correspond to the first point of view, wherein the generating comprises replacing the object disposed in a first area in the plurality of frames of the revised video data with the data corresponding to the object separate from the first video data, and wherein the generating comprises identifying the area of the object in one of the frames of the plurality of frames, and using the identified area in the one of the frames for identifying the area of the object in a subsequent frame, wherein the first video data of the environment comprises the object comprising a virtual reality headset, and the revised video data comprises the plurality of frames with the virtual reality headset disposed in the area of the plurality of frames being revised based on second video data obtained of the environment behind the virtual reality headset during the receiving the first video data, wherein the generating comprises synchronizing the second video data obtained of the environment behind the virtual reality headset during the receiving the first video data with the first video data.