# US 11,122,258 B2 · Method and apparatus for generating and displaying 360-degree video based on eye tracking and physiological measurements > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,122,258 B2 - Document ID: US11122258B2 - Status: Granted - Kind: B2 - Application number: 16/626,214 - Filed: 2018-06-22 - Published: 2020-04-23 - Granted: 2021-09-14 - Earliest claimed: 2017-06-30 - Assignees: PCMS Holdings, Inc. - Applicants: PCMS Holdings, Inc. - Inventors: Sungjae Hwang; Jongho Kim - Web page: https://patentgrep.com/patent/US11122258B2 ## CPC - H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes - H04N 21/42201: {biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user (input arrangements for interaction with the human body based on nervous system activity detection G06F3/015)} - A61B 5/0022: {Monitoring a patient using a global network, e.g. telephone networks, internet} - A61B 5/163: {by tracking eye movement, gaze, or pupil change} - A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety} - A61B 5/33: specially adapted for cooperation with other devices - A61B 5/372: Analysis of electroencephalograms - A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles} - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06T 7/74: {involving reference images or patches} - G16H 40/67: for remote operation - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - H04N 13/344: with head-mounted left-right displays - H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras} - H04N 21/274: Storing end-user {multimedia} data in response to end-user request {, e.g. network recorder} - H04N 21/4223: Cameras (H04N23/00 takes precedence) - H04N 21/440245: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment} - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - H04N 21/816: {involving special video data, e.g 3D video} - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - H04N 23/63: by using electronic viewfinders - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - A61B 3/112: {for measuring diameter of pupils} - A61B 3/113: for determining or recording eye movement - A61B 5/02438: {with portable devices, e.g. worn by the patient} - A61B 5/11: Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb {(A61B5/1038 takes precedence; motion detection to correct for motion artifacts in physiological signals A61B5/721)} - A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0141: {characterised by the informative content of the display} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0181: {Adaptation to the pilot/driver} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract Some embodiments of a method may include: identifying an object of interest in a multi-view video based on a point of view and physiological data of a camera operator; displaying the multi-view video; and displaying a visual effect associated with the object of interest. Some embodiments of a method may include: capturing a multi-view video with a camera; tracking a point of view of a camera operator controlling the camera; measuring physiological data of the camera operator; identifying an object of interest in the multi-view video based on the point of view and the physiological data; and storing the multi-view video and information indicating the object of interest identified. ## Claims ### Claim 1 A method comprising: displaying multi-view video on a head mounted display (HMD) device of a user, the multi-view video associated with a point of view of a camera operator; responsive to determining that a point of view of the user overlaps with the point of view of the camera operator, setting an effect intensity as a function of an amount of overlap between the point of view of the camera operator and the point of view of the user, wherein the point of view of the user is defined by a viewable display of the HMD device; and using an effect, in accordance with the effect intensity, to highlight an object of interest in the displayed multi-view video. ### Claim 2 (depends on 1) The method of claim 1, wherein using the effect in accordance with the effect intensity to highlight the object of interest adjusts a brightness of the object of interest, and wherein the effect intensity is a brightness level to which the brightness of the object of interest is adjusted. ### Claim 3 (depends on 1) The method of claim 1, wherein using the effect in accordance with the effect intensity to highlight the object of interest adjusts a zoom level of the object of interest, and wherein the effect intensity is the zoom level to which the object of interest is adjusted. ### Claim 4 (depends on 1) The method of claim 1, wherein using the effect in accordance with the effect intensity to highlight the object of interest plays a sound, and wherein the effect intensity is a volume level at which the sound is played. ### Claim 5 (depends on 1) The method of claim 1, wherein using the effect in accordance with the effect intensity to highlight the object of interest causes a haptic module to vibrate, and wherein the effect intensity is an intensity at which the haptic module vibrates. ### Claim 6 (depends on 1) The method of claim 1, further comprising determining the object of interest based on eye tracking of the camera operator at a time of capture of the multi-view video. ### Claim 7 (depends on 1) The method of claim 1, further comprising determining the object of interest based on physiological sensor information recorded from the camera operator at a time of capture of the multi-view video. ### Claim 8 (depends on 1) The method of claim 1, further comprising: capturing the multi-view video with a camera; tracking the point of view of the camera operator; measuring physiological data of the camera operator; and identifying the object of interest in the multi-view video based on the point of view of the camera operator and the physiological data of the camera operator. ### Claim 9 (depends on 1) The method of claim 1, further comprising: comparing the point of view of the camera operator with the point of view of the user; and determining the amount of overlap that the point of view of the user overlaps with the point of view of the camera operator. ### Claim 10 (depends on 1) The method of claim 1, further comprising: responsive to detecting a proximity between the point of view of the user and the point of view of the camera operator, displaying a visual effect indicating the proximity between the point of view of the user and the point of view of the camera operator, wherein displaying the multi-view video comprises displaying the multi-view video to be synchronous with a head movement of the user. ### Claim 11 (depends on 1) The method of claim 1, wherein the effect has an intensity equal to the effect intensity. ### Claim 12 (depends on 1) The method of claim 1, further comprising determining the point of view of the user. ### Claim 13 (depends on 1) The method of claim 1, further comprising determining whether the point of view of the user overlaps with the point of view of the camera operator. ### Claim 14 (depends on 1) The method of claim 1, wherein the object of interest is identified as being within the point of view of the camera operator and is indicated as being of interest to the camera operator. ### Claim 15 (depends on 1) The method of claim 1, wherein setting the effect intensity as the function of the amount of overlap between the point of view of the camera operator and the point of view of the user comprises setting the effect intensity based proportionally on the amount of overlap between the point of view of the camera operator and the point of view of the user. ### Claim 16 An apparatus comprising: a processor; and a non-transitory computer-readable medium storing instructions that are operative, when executed by the processor, to cause the apparatus to: display multi-view video on a head mounted display (HMD) device of a user, the multi-view video associated with a point of view of a camera operator; responsive to determining that a point of view of the user overlaps with the point of view of the camera operator, set an effect intensity as a function of an amount of overlap between the point of view of the camera operator and the point of view of the user, wherein the point of view of the user is defined by a viewable display of the HMD device; and use an effect, in accordance with the effect intensity, to highlight an object of interest in the displayed multi-view video. ### Claim 17 (depends on 16) The apparatus of claim 16, further comprising: a gaze-tracking sensor; a camera; a physiological sensor; and a display. ### Claim 18 A method comprising: displaying multi-view video on a head mounted display (HMD) device of a user, the multi-view video associated with a point of view of a camera operator; rendering, within a point of view of the user and while the point of view of the user overlaps with the point of view of camera operator, as part of the displayed multi-view video, an object of interest identified as being in the point of view of the camera operator, the point of view of the user being defined by a viewable display of the HMD device; and responsive to determining that the point of view of the user no longer overlaps with the point of view of the camera operator, implementing a display effect to continue rendering the identified object of interest within the point of view of the user. ### Claim 19 (depends on 18) The method of claim 18, further comprising: receiving information regarding the point of view of the camera operator and the object of interest in the multi-view video, the point of view of the camera operator being with respect to the multi-view video, and the object of interest is indicated as being of interest to the camera operator. ### Claim 20 (depends on 19) The method of claim 19, further comprising: capturing the multi-view video with a multi-view camera; tracking a view direction and the point of view of the camera operator controlling the camera using an eye tracker mounted on the camera; measuring physiological data of the camera operator; and identifying the object of interest in the multi-view video based on the view direction and the physiological data.