# US 2021/0281802 A1 · IMPROVED METHOD AND SYSTEM FOR VIDEO CONFERENCES WITH HMDs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0281802 A1 - Document ID: US20210281802A1 - Status: Published application - Kind: A1 - Application number: 16/334,432 - Filed: 2017-02-03 - Published: 2021-09-09 - Earliest claimed: 2017-02-03 - Applicants: Vestel Elektronik Sanayi Ve Ticaret A.S. - Inventors: Barbaros KIRISKEN - Web page: https://patentgrep.com/patent/US20210281802A1 ## CPC - H04N 7/15: Conference systems - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/017: {Head mounted} - H04N 7/142: {Constructional details of the terminal equipment, e.g. arrangements of the camera and the display} - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract The present invention refers to a method for modifying video data during a video conference session. This method comprises at least the steps: Providing a first terminal (100A), comprising a first camera unit (103X) for capturing of at least visual input and a first head-mounted display (102); Providing a second terminal (100B) at least for outputting visual input, Providing or capturing first basic image data or first basic video data of a head of a first person (101) with the first camera unit (103X), Capturing first process image data or first process video data of the head of said first person (101) with the first camera unit while said first person (101) wears the head-mounted display (102), Determining first process data sections of the first process image data or first process video data representing the visual appearance of the first head-mounted display (102), Generating a first set of modified image data or modified video data by replacing the first process data sections of the first process image data or first process video data by first basic data sections, wherein the first basic data sections are part of the first basic image data or first basic video data and are representing parts of the face of said person, in particularly the eyes of said first person (101). ## Claims ### Claim 1 Method for modifying video data during a video conference session, at least comprising the steps: Providing a first terminal, comprising a first camera unit for capturing of at least visual input and a first head-mounted display; Providing a second terminal at least for outputting visual input, Providing a server means, wherein said first terminal and said second terminal are connected via the server means for data exchange, Providing or capturing first basic image data or first basic video data of a head of a first person with the first camera unit, Capturing first process image data or first process video data of the head of said first person with the first camera unit while said first person wears the head-mounted display, Determining first process data sections of the first process image data or first process video data representing the visual appearance of the first head-mounted display, Generating a first set of modified image data or modified video data by replacing the first process data sections of the first process image data or first process video data by first basic data sections, wherein the first basic data sections are part of the first basic image data or first basic video data and are representing parts of the face of said person, in particular the eyes of said first person. ### Claim 2 (depends on 1) Method according to claim 1, characterized in that the second terminal comprises a second camera unit and a second head-mounted display, and by steps: Providing or capturing second basic image data or second basic video data of a head of a second person with the second camera unit, Capturing second process image data or second process video data of the head of said second person with the second camera unit while said second person wears the second head-mounted display, Determining second process data sections of the second process image data or second process video data representing the visual appearance of the second head-mounted display, Forming a second set of modified image data or modified video data by replacing the second process data sections of the second process image data or second process video data by second basic data sections, wherein the second basic data sections are part of the second basic image data or second basic video data and are representing parts of the face of said second person, in particularly the eyes of said second person, Outputting the second modified image data or second modified video data via the first terminal. ### Claim 3 (depends on 1) Method according to claim 1, characterized in that the first modified image data or first modified video data and/or the second modified image data or second modified video data is/are outputted via at least one further terminal connected to the server means. ### Claim 4 (depends on 1) Method according to claim 1, characterized in that a further terminal comprises a further camera unit and a further head-mounted display, and by steps: Providing or capturing further basic image data or further basic video data of a head of a further person with the further camera unit, Capturing further process image data or further process video data of the head of said further person with the further camera unit while said further person wears the further head-mounted display, Determining further process data sections of the further process image data or further process video data representing the visual appearance of the further head-mounted display, Forming a further set of modified image data or modified video data by replacing the further process data sections of the further process image data or further process video data by further basic data sections, wherein the further basic data sections are part of the further basic image data or further basic video data and are representing parts of the face of said further person, in particular the eyes of said further person, Outputting the further modified image data or further modified video data via the first terminal and/or via the second terminal, in particular at the same time. ### Claim 5 (depends on 1) Method according to claim 1, characterized in that first, second and/or further basic video data or first, second and/or further basic image data are stored in a memory of the respective terminal and/or on the server means, wherein first, second and/or further basic video data or first, second and/or further basic image data are captured once and processed in case first, second and/or further modified video data or first, second and/or further modified image data is required or wherein first, second and/or further basic video data or first, second and/or further basic image data are captured each time said first, second and/or third person joins a video conference and the first, second and/or further basic video data or first, second and/or further basic image data is updated or replaced and processed in case first, second and/or further modified video data or first, second and/or further modified image data is required. ### Claim 6 (depends on 1) Method according to claim 1, characterized in that at least one terminal and preferably the majority of terminals or all terminals are comprising means for capturing and/or outputting audio data, wherein said captured audio data captured by one terminal is at least routed to one or multiple further terminals. ### Claim 7 (depends on 1) Method according to claim 1, characterized in that the position of the first head-mounted display with respect to the face of the first person is determined by means of object recognition, the shape of the first head-mounted display is determined by object recognition and/or identification data is visually or electronically provided. ### Claim 8 (depends on 1) Method according to claim 1, characterized in that face movement data representing movements of skin portions of the face of the first person is generated, wherein the movements of the skin portions are captured by said first camera unit. ### Claim 9 (depends on 1) Method according to claim 1, characterized in that eye movement data representing movements of at least one eye of the first person is generated, wherein the movements of the eye are captured by an eye tracking means. ### Claim 10 (depends on 8) Method according to claim 8, characterized in that first basic data sections are modified in dependency of said captured face movement data of the face of the first person, and/or of said captured eye movement data of at least one eye of the first person. ### Claim 11 (depends on 10) Method according to claim 10, characterized in that eye data representing shapes of the eyes of the first person is identified as part of the first basic data sections, wherein the eye data is modified in dependency of said captured eye movement data, and/or skin data representing the skin portions of the face of the first person is modified above and/or below the eyes in the first basic data section is identified, wherein the skin data in dependency of said captured face movement data. ### Claim 12 (depends on 10) Method according to claim 10, characterized in that eye tracking means is preferably a near eye PCCR tracker, wherein said eye tracking means is arranged on or inside the first head-mounted display. ### Claim 13 (depends on 1) Method according to claim 1, characterized by receiving information relating to the pose of the head of the first person; orienting a virtual model of the head and facial gaze of the head according to the pose of the object; projecting visible pixels from a portion of the videoconference communication onto the virtual model; creating synthesized eyes of the head that produces a facial gaze at a desired point in space; orienting the virtual model according to the produced facial gaze; and projecting the virtual model onto a corresponding portion of the videoconference communication, wherein at least one part of the first set of modified image data or modified video data is replaced by said virtual model. ### Claim 14 (depends on 1) Computer program product for executing a method according to claim 1. ### Claim 15 System for video conference sessions, at least comprising: A first terminal, comprising a first camera unit for capturing of at least visual input and a first head-mounted display, a second terminal at least for outputting visual input, a server means, wherein said first terminal and said second terminal are connected via the server means for data exchange, wherein first basic image data or first basic video data of a head of a first person is provided or captured with the first camera unit, wherein first process image data or first process video data of the head of said first person is captured with the first camera unit while said first person wears the head-mounted display, wherein first process data sections of the first process image data or first process video data representing the visual appearance of the first head-mounted display is determined, wherein a first set of modified image data or modified video data is formed by replacing the first process data sections of the first process image data or first process video data by first basic data sections, wherein the first basic data sections are part of the first basic image data or first basic video data and are representing parts of the face of said person, in particularly the eyes of said person, wherein the first modified image data or first modified video data, in particular representing a complete face of said person, are outputted via the second terminal.