# US 10,122,985 B2 · Camera based safety mechanisms for users of head mounted displays > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,122,985 B2 - Document ID: US10122985B2 - Status: Granted - Kind: B2 - Application number: 15/671,126 - Filed: 2017-08-07 - Published: 2017-12-14 - Granted: 2018-11-06 - Earliest claimed: 2013-09-30 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Jeffrey Roger Stafford - Web page: https://patentgrep.com/patent/US10122985B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD] - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/5255: according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character - G02B 27/017: {Head mounted} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F 2203/012: Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract The disclosure provides methods and systems for warning a user of a head mounted display that the user approaches an edge of field of view of a camera or one or more tangible obstacles. The warning includes presenting audio and/or displayable messages to the user, or moving the display(s) of the head mounted displays away of the user's eyes. The determination that the user approaches the edge of scene or a tangible obstacle is made by dynamically tracking motions of the users through analysis of images and/or depth data obtained from image sensor(s) and/or depth sensor(s) secured to either the head mounted display, arranged outside of the scene and not secured to the head mounted display, or a combination of both. ## Claims ### Claim 1 A method for using a head mounted display, the method comprising: providing interactive content for rendering on a display portion of the head mounted display (HMD) worn by a user, the HMD when worn by the user is configured to block a view to a real-world environment; receiving images of a physical environment in a vicinity of the user wearing the HMD while the user is viewing the interactive content; tracking location information of the user wearing the HMD as the user is viewing the interactive content, wherein the location information being tracked in relation to an area within the physical environment in which the user is able to move around while interacting with the interactive content, the area defined by calibration based on an amount of space available for the user to move within the physical environment; dynamically determining the user approaching an edge of the area while viewing the interactive content rendered on the display of the HMD; and providing a warning to the user at the HMD that is indicative of a proximity of the user to the edge of the area, wherein the warning is provided as an image highlighting the edge of the area for rendering on the display, the image including one or more real-world objects that are inside of the area. ### Claim 2 (depends on 1) The method of claim 1, wherein tracking location information includes, processing the images received from one or more depth sensors to generate a depth map of a three-dimensional space for the area within the physical environment, the depth map automatically calibrated to capture depth of real-world objects disposed in the physical environment; and tracking movement of the user in the physical environment in relation to the real-world objects, using the depth map. ### Claim 3 (depends on 2) The method of claim 2, wherein the image highlighting the edge of the area is blended with the interactive content, and wherein the area is defined within a field of view of the one or more depth sensors. ### Claim 4 (depends on 2) The method of claim 2, wherein tracking location information further includes, receiving secondary images of the area within the physical environment in which the user is present, from one or more secondary sensors coupled to a control device; and processing the secondary images to determine relative location of the user in relation to the real-world objects, the processing of the secondary images includes fusing the secondary images with the images received from the one or more depth sensors to determine distance of the user from the real-world objects. ### Claim 5 (depends on 2) The method of claim 2, wherein tracking the movement of the user includes, determining coordinates of a virtual skeleton of the user and of the real-world objects within the three-dimensional space of the physical environment; and computing distance of different body parts of the virtual skeleton of the user from each one of the real-world objects. ### Claim 6 (depends on 1) The method of claim 1, wherein tracking location information of the user includes tracking location of various body parts of the user. ### Claim 7 (depends on 6) The method of claim 6, wherein the warning is provided upon detecting any one of the body parts of the user approaches within a pre-defined distance from the edge of the area. ### Claim 8 (depends on 1) The method of claim 1, wherein the image of the edge is provided in a grid format, wherein the image of the edge is presented with the interactive content and wherein an intensity of the image is adjusted to reflect current position of the user in relation to the edge of the area. ### Claim 9 (depends on 8) The method of claim 8, wherein the intensity of the image is set to increase as the user continues to approach the edge of the area and decrease as the user moves away from the edge of the area. ### Claim 10 (depends on 1) The method of claim 1, wherein the image of the edge is provided as an outline, an intensity of the outline adjusted to reflect current position of the user in relation to the edge of the area. ### Claim 11 (depends on 1) The method of claim 1, wherein the warning is further provided as an audio message, an intensity of the audio message representative of a current position of the user in relation to the edge of the area. ### Claim 12 (depends on 1) The method of claim 1, wherein the warning provides guiding instructions to direct the user back towards a center of the area or away from the edge of the area, and wherein an intensity of the image highlighting the edge of the area automatically adjusted to reflect relative distance of the user from the edge of the area as the user is guided away from the edge of the area. ### Claim 13 A method for using a head mounted display, the method comprising: providing interactive content for rendering on a display portion of the head mounted display (HMD) worn by a user, the HMD when worn by the user is configured to block a view to a real-world environment; receiving images of a physical environment in a vicinity of the user wearing the head mounted display while the user is viewing the interactive content; tracking location information of the user wearing the HMD as the user is interacting with the interactive content, wherein the location information is tracked in relation to an area within the physical environment in which the user is able to move around while interacting with the interactive content, the area defined by calibration based on an amount of space available for the user to move within the physical environment; determining the user approaching an edge of a safe interaction zone defined in the area of the physical environment while interacting with the interactive content that is being rendered on the display portion of the HMD; and adding a highlighted indicator to the interactive content currently being rendered on the display portion of the HMD, the highlighted indicator indicative of the edge of the safe interaction zone defined in the area of the physical environment in which the user is interacting with the interactive content. ### Claim 14 (depends on 13) The method of claim 13, wherein the safe interaction zone is an area where the user is able to move freely and is devoid of any real-world objects. ### Claim 15 (depends on 13) The method of claim 13, wherein tracking location information includes, generating a depth map of a three-dimensional space for the area represented within the physical environment, using the images provided by one or more depth sensors; and computing the location of the user in relation to the edge of the safe interaction zone in the physical environment, using the depth map. ### Claim 16 (depends on 13) The method of claim 13, wherein tracking location of the user includes, generating a virtual skeleton of the user within the safe interaction zone defined in the physical environment while the user is interacting with the interactive content; tracking coordinates of different body parts represented in the virtual skeleton of the user; and computing distance of different body parts of the virtual skeleton of the user from the edge of the safe interaction zone in the physical environment. ### Claim 17 (depends on 16) The method of claim 16, wherein the highlighted indicator is added upon detecting any one of the different body parts in the virtual skeleton of the user approaches within a pre-defined distance from the edge of the safe interaction zone. ### Claim 18 (depends on 13) The method of claim 13, wherein an intensity of the highlighted indicator is adjusted based on a distance of the user to the edge of the area.