Mixed reality gaming system
OwnerDean Lindsay DELAMONTinventor
Life of this patent
Estimating the term…
Abstract
An interactive mixed reality system for one or more users, in which both real-world entities and virtual world entities are capable of being interacted with by one or more users, or by objects (47, 91) for use by users, and the system arranged to computationally maintain game state and the evolution of events in the real-world and the virtual world, and the system arranged to generate a response (such as visual or tactile/haptic, or by way of cause and effect) which is experienced or perceived by the one or more users.
Claims39
An interactive mixed reality system for one or more users, in which both real-world entities and virtual world entities are capable of being interacted with by one or more users, or by objects for use by users, and the system arranged to computationally maintain game state and the evolution of events in the real-world and the virtual world, and the system arranged to generate a response (such as visual or tactile/haptic, or by way of cause and effect) which is experienced or perceived by the one or more users.
A system as claimed in any preceding claim in which the display on the surface of the physical real-world object may be by way of light being projected onto the surface and/or by way of the surface being arranged to emit light.
An enhanced IR mixed reality and laser tag game system comprising one or more of the following:
- real world space in which this comprises physical real-world game objects, users, surfaces, AI characters;
- virtual world space in which this comprises virtual game objects, scenery, AI and remote users;
- real-world game objects and users in which the game scenes and 3D renderings of real-world game objects and user avatars are visible to the naked eye and touchable as physical objects that can be interacted with;
- display of rendered virtual images of virtual game objects, game scenes, texture-maps, decal, lighting effects and remote users augmented over the real-world space of the game and objects or users in the real-world, which may be made visible to naked eye displayed via the use of externally projected light or a real-world game objects surface display panel/screen faces or visible as augmented reality virtual images displayed via a user's head-mounted augmented-reality (“AR”) display apparatus as a form of two 2D stereoscopic images or a holographic or hologram image;
- Where:
- a physical action by a user or real-world AI character in the real world such as the pulling of a real-world game objects trigger mechanism can affect the state of the game in the real-world and/or virtual world;
- the users, game objects, game state and geometry in the real-world is maintained computationally in which changes in real world game objects, users, surfaces are tracked via sensor(s) or generated mesh data through observations using external camera(s) and/or other means;
- real world game objects are physical objects in the user's real-world space for which their 3D renderings can be seen by the naked eye and these objects can be used, in which a real-world user or AI physical actions using these objects serves as an input that can affect the state of game, other real or virtual game objects, users, AI characters and game scenes in the real-world and/or virtual world;
- real world game objects users and real-world AI characters have paired virtual game objects, 3D models and mesh data which is maintained computationally used in the game rendering and mathematical operations including collision detections, lighting effects and transformations;
- real-world game objects comprise of a visible mesh, 3D model and texture-maps that are visible to the naked eye, where these objects maybe wearable or handled or part of the game scenery, where in the case of a wearable a 3D rendered avatar maybe displayed over a user's body visible to the naked eye;
- virtual images, texture-maps, lighting effects and game renderings may be displayed directly via a real-world game object surface display panel/screen faces seen by the naked eye or via the use of external projectors using 3D projection mapping which may also be seen by the naked eye or alternatively maybe displayed via a user's augmented-reality (“AR”) display apparatus;
- real-world fired IR beam or IR Laser beams have generated and rendered virtual game objects and/or line renderings or rays together with lighting effects in which the beam normally invisible to the human eye is made visible via a real-world game object surface display panel/screen faces seen by the naked eye or via the use of external projectors using 3D projection mapping which may be seen by the naked eye or alternatively maybe displayed via a user's augmented-reality (“AR”) display apparatus, in which a user can see a beam travelling in the users real-world space as a virtual image;
- rendering operations of real-world game objects, users, surfaces and real-world objects is supported by the generating of virtual game objects, 3D models, mesh data and the use of a rendering module and games engine together with CPUs and/or GPUs by clients and/or the game server or host;
- the state of real-world game objects, AI characters and users in the game real-world space is tracked via sensors and/or the use of spatial mapping or structural light scanning techniques;
- in parallel the state of the virtual world and virtual game objects including paired virtual game objects is maintained computationally by clients and/or a game server or host;
- transformations are applied to paired virtual game objects of real-world users, and real-world game objects based on detected changes in state based on retrieved sensor data or mesh data;
- collisions between users or objects including the collision of an IR beam or IR Laser beam with a user or real world game object may be pre-determinable based on retrieved sensor data and state information in which hits may occur on another real-world game object, user, surface or AI character from the physical use of a real-world game object resulting the emitting of an IR beam or IR Laser beam via an IR LED or IR Laser Diode, where a hit maybe computed based on the relative projectile and positions of the beam with another object derived from sensor data of the real-world game object that fired the beam, or based on the detection of the light or radiation emissions via an IR Sensor which may invoke a physical response in the real-world and/or the display of decal and diffuse lighting effects which can be seen via the naked eye or other means where;
- hits from an IR beam or IR Laser beam or virtual weapon can affect also the state of real-world objects and users;
- hits from an IR beam or IR Laser beam or a virtual weapon maybe seen on a real-world game object surface display panel/screen faces or via the use of external projectors and 3D projection mapping techniques on a user, real-world game object, object or surfaces that maybe seen by the naked eye;
- or alternatively hits from an IR beam or IR Laser beam or a virtual weapon maybe seen via a user's augmented-reality (“AR”) display apparatus as a stereoscopic, hologram or holographic image;
- decal/diffuse lighting effects may be displayed at the determined point of collision;
- the game maybe played in the home, outside and/or in a laser tag arena or in a combination of above and in a variety of configurations of the system detailed in the embodiments of the invention;
A method rendering and displaying a real-world game object as a fully rendered 3D image that has physical presences in the real-world environment and is visible in the users by naked eye without any visual aid, comprising one or more of:
- Recreating complex 3D shapes and 3D meshes that are visible in the real-world in the form of a real-world game object through arranging a series of physical surface display screen faces in the form of triangles/polygons or quads that cover the surfaces and faces of a real-world game objects to provide a semi or full 3D mesh and 3D model or wireframe visible to the naked eye, in which the surface display screen faces may be a form of organic display such as OLED or a form of LCOS or LCD made of a glass or polymer material or a form of Graphene/Graphite flexible display, where the surface display screen faces may be arranged in a tiled configuration to form a mesh and 3D model made up of many triangles/polygons or quads that is visible in the real world by the naked eye;
- Generating a virtual game object, virtual mesh and 3D model via a games engine and/or rendering module, in which the generated virtual game object, mesh data and 3D model is geometrically exactly the same in terms of its representative corresponding mesh data, polygons, triangles or quads, 3D model coordinates, surfaces, faces, edges, and vertices to that of the real-world game objects own real-world model polygons/triangles or quads in the form of physical surface display screen faces, together with its model coordinates, and addressable surfaces, faces, edges, and vertices in which:
- Each surface display screen face has a comprises of one or more of triangles/polygons or quads that each has a set of vertices, each of which has a coordinate of (x,y,z) referred to as a point or vertex;
- The vertices which are referenced by the faces, define the overall shape of the 3D model of the objects real-world shape and 3D model;
- Pairing the generated virtual game object with a real-world game object that is geometrically identical;
- Mapping a real-world game objects mesh, vertices, points, edges, faces, triangles/polygons or quads and surfaces to the generated paired virtual-game objects mesh, vertices, points, edges, faces, triangles polygons or quads and surfaces;
- Arranging physical surface display screen faces triangles/polygons and quads in the order in which the faces references each of the vertices is provided as a list of faces;
- Invoking the rendering texture-maps, lighting and shading effects over the surfaces and faces of the virtual game object via a games engine and or rendering module;
- The usage of Materials, Texture Maps, 2D Coordinates, Vector Arrays in the rendering process;
- Applying rasterisation, fragmentation and shading;
- Generating of rendered texture maps or virtual images as an output of the rendering process of the paired virtual-game object in the form of rasterised 2D virtual images;
- Applying or overlaying texture maps or 2D virtual images output from the rendering process over the paired real-world game objects physical surface display panel/screen faces, through a variety of mapping techniques including but not limited to;
- The use of pixel coordinate based mappings, in which every surface display panel/screen face has a unique set of pixel coordinates;
- The use texture mapping techniques and the use 2D coordinates to map texture maps or 2D virtual images to the surface display panel faces of a real-world game object;
- Use of a centroid of a triangle where the position each surface display panel/screen face represents a vector of the centroid of a triangle and each surface display panel/screen face represents a triangle/polygon or quad;
- Loading pixel coordinates and colour values provided in the form of a generated RGB array or in-memory bitmap into each surface display panel/screen faces frame buffers or a single frame buffer as an output of the rendering process;
- Where in the case a single frame buffer the surface display panel/screen faces maybe configured in a tiled in which the pixel coordinates span all of the different surface display panel/screen faces, where every surface display panel/screen faces has a unique addressable pixel set of coordinates or RGB positions in which individual texture maps and/or 2D images are mapped based on their respective pixel coordinates or RGB positions to each surface display panel/screen faces;
- Reading/processing of the pixel colours and pixel coordinates from a single frame buffer or the frame buffer for each of the surface display panel/screen faces by one or more light display modules;
- Illuminating, activating or exciting each of the surface display panel/screen faces using a light display module based on the assigned pixel coordinates and colour values and displaying multiple 2D images or rendered texture-maps over the respective surface display panel/screen faces of the real-world game object to form a fully rendered 3D image and recreate complex 3D shapes that have physical in presence in the real world and which are visible to the human naked eye without any visual aid.
A method of applying transformations to the paired virtual game object of a real-world game objects based on changes in the state of a real-world game object and displaying updates to the displayed texture-maps, lighting and shading effects seen by the naked eye comprising one or more of;
- Detecting and capturing changes in movements and state of a real-world game object, including changes to the position, orientation, rotation, directional heading, elevation, velocity and acceleration of a real-world game object in real-world space using the real-world three dimensional game object's sensor(s);
- Processing captured sensor measurements that maybe converted or translated computationally to relational x,y,z three dimensional cartesian coordinates representing the real-world game objects positional coordinates/world coordinates/vector position and directional heading/directional vector among other states, together with the pitch, roll and yaw (p,r,y) values that represent the orientation and rotation of the real-world game object in the real world space:
- Transmitting and storing of state and sensor data of captured changes including the position, orientation, rotation, directional heading, elevation, velocity and acceleration of a real-world game object in real-world space where state is stored both stored locally by the real-world game object and transmitted via the network to the game server and/or host together with other real-world games objects, used in the computational and rendering operations of the game;
- Retrieving state and sensor data stored locally and/or via the network from the game server or host
- Processing of retrieved state and sensor data including a real-world game objects positional coordinates/world coordinates/vector position and directional heading/directional in the form of x,y,z three dimensional cartesian coordinates and set of vector values together with its pitch, roll and yaw (p,r,y) values, together with other captured state information by a games engine and rendering module.
- Applying transformations to the paired virtual game object of the real world game object, based on detected changes in movement and/or state, via the games engine and rendering module of the real-world game object using the apparatus CPU and/or GPU, where:;
- a 4×4 matrix may be used to pass values;
- Or optionally transformations may be applied using Quaternions;
- changes in movements and the state of a real-world game object including changes to its position, rotation and other values are applied to its paired virtual game object through transformation operation's by the games engine and rendering module relative to that of captured changes to;
- in movements and state of the real-world game object based on the processed state and sensor data captured via the objects sensor(s) which includes new x,y,z and p,r,y values that represent changes to the real-world game objects vector positional coordinates, world coordinates, directional vector, orientation and rotation in real-world space;
- Generating updated rendered texture-maps, lighting and shading effects by the games engine and rendering module of the real-world game object;
- Applying rasterisation, fragmentation and shading techniques;
- Where in the lighting modelling process:
- reflection of light from the lighting components of other real-world game objects paired virtual game objects or other virtual game objects in a game scene and/or ambient lighting sources maybe applied;
- reflection of light, diffuse and specular lighting effects of an animated moving IR beam or IR Laser beam virtual game object and/or ray or line rendering maybe applied;
- Light sources from either of these real-world objects or virtual objects may be applied;
- light sources maybe real Mixed Reality Scenery or virtual augmented game scenery in which these may have a defined as coordinate (x,y,z) and a form of ambient light, colour and an projection orientation into the three-dimensional space alike to a directional vector in which the light is cast in a determinable direction which may be applied;
- lighting effects may be applied dynamically movements or changes in position (x,y,z) or orientation (p,r,y) of real-world game objects, users, AI character's and virtual game objects, AI character's or remote users relational each others light sources and that of other fixed scenery light sources or ambient light sources that may change with time where these may be virtual and part of a game scene;
- Generating texture maps or virtual images with applied updates for lighting and shading operations as an output of the rendering process in the form of 2D rasterised images and overlaying the texture-maps or virtual images over each physical surface display panel/screen faces, based on their respective vertices and model coordinates or texture map coordinates among other mapping means;
- Loading into each surface display panel/screen faces frame buffer pixel coordinates and colour values provided in the form of a generated RGB array or in-memory bitmap and as an output of the rendering process;
- Illuminating, exciting or activating each of the surface display panel/screenfaces based on the pixel coordinates and colour values to form a fully rendered 3D image over the surfaces of the real-world game object in which the updates displayed changes to the texture maps and lighting effects resulting from transformations are visible in the three-dimensional space of user's real-world environment by the human naked eye.
A method of generating and displaying augmented scenes of game scenes together with spawned or generated virtual-game objects or line renderings or rays of real-world moving IR beams or Laser beams overlaid over a user's real-world space and field of view comprising one or more of:
- Capturing two sets of video and images of a user's real world environment using the two external front facing cameras of a user's head mounted wearable augmented reality (“AR”) display apparatus representing the users left and right eye field of view (FOV);
- Using the captured video and/or images together with spatial mapping techniques to generate geometric, surface, depth and volumetric data by an observer module on the user's real-world space environment, objects and surfaces, in the form of surface data, mesh data, mesh filters, wireframes and 3D models for the purpose of overlaying augmented images of game scenes and moving IR beams or IR Laser beams;
- Optional use of techniques and algorithms such as Kernel-based tracking and/or Contour tracking and/or Condensation algorithm, and/or Kalman filter and/or Particle filter in the processing of captured video and/or images;
- Generating of depth information using Computer Vision, ToF or Computer Stereo Vision techniques and capture video or images;
- Using the generated mesh data, mesh filters and wireframes containing the surface, volumetric, geometric structural, and depth data of a user's real world space and environment by a games engine and/or rendering module to create a virtual game objects and 3D models of the real-world objects and surfaces observed from the users front facing cameras and respective field of view (FOV);
- Applying surface rendering materials and/or texture maps to generated virtual game objects and 3D models of the observed real-world objects and surfaces in the user's real-world space;
- Capturing and generating state information and sensor data on one or more real-world game object(s) world coordinates, vector position, directional vector, orientation and rotation in the 3D space of the grid in the form of x,y,z cartesian coordinates and pitch, roll and yaw (p,r,y) values based on sensor outputs captured from the real-world game objects sensor(s) at the time of the trigger being pulled or the apparatus being activated for each beam fired, which may be provided in the form of a matrix array together with a boolean value indicating the trigger was pulled;
- Receiving state information and sensor data via the network from the game server or host or another real-world game object, on any moving IR beams or IR Laser beams indicated by a boolean flag together with the relating real-world game objects state at the time that the beam was fired including sensor data on its vector position or three dimensional cartesian coordinate or world coordinates together with its orientation, rotation, directional vector and elevation for each fired beam;
- Determining the starting position (origin), projectile and trajectory of each fired IR beam or IR Laser beam in real world space based on the retrieved relating state and sensor data which may be in the form of a matrix array containing a vector position or three-dimensional cartesian coordinate and/or world coordinates in the form of x,y,z values together with the rotation and orientation in the form of pitch, roll, yaw (p,r,y) values, a directional vector and values for the height elevation of real-world game object at the time the IR beam or IR Laser beam was emitted, resulting from pulling the trigger mechanism or activating of the device;
- Formulating the distance travelled from a determined point of origin based on an assumed velocity and range of the real-world game object or real-world game objects, where the beams originate from differing real-world game objects, for which the distance shall be formulated for each fired IR beam or IR Laser beam;
- Generating one or more virtual game objects or line renderings with virtual game objects or a ray using a games engine and rendering module and rendering in 3D an virtual image depicting each moving beam based on a computation of its relative origin, directional vector, angle, and projectile/trajectory;
- Applying transformations to move the virtual game object or virtual game objects or line rendering or ray to progress it rendering along a number of vector points according to the assumed velocity and distance of the beam where the virtual game object or virtual game object(s)or line rendering or ray is moved at equal positions and rate of velocity along points in three-dimensional space to that of the respective real IR beam or IR Laser beam light moving in the user's real world space relative to each fired beam;
- Rending virtual images of game scenes, virtual game objects including one or more paired virtual game objects or line renderings or rays of one or more moving IR beam or IR Laser Beam together with lighting effects over a user's observed real world space and FOV based on the generated mesh data, mesh filters, wireframes and 3D models using a games engine and rendering module together with the augmented reality (“AR”) display apparatus processing units that includes one or more CPUs and GPUs;
- Applying rasterisation, fragmentation and shading to the rendered virtual images;
- Invoking the display of rendered game scenes and virtual game objects including one or more paired virtual game objects or line renderings or rays of moving IR beams or Laser Beams together with lighting effects via the user's micro-display and light display module, as an augmented virtual image over the user's left and right FOV by illuminating, activating or exciting the display in accordance with supplied pixel coordinate and colour values that maybe supplied in the form of an RGB Array or in-memory bitmap to the frame buffer of the users augmented reality (“AR”) display apparatus, where the users brain perceives the displayed augmented virtual image to be moving at specific locations in the three dimensional space of their real world in which:
- The displayed augmented virtual image maybe a form of two stereoscopic images or a form of a hologram or holographic image;
- Through a process of spatial mapping, mesh creation and rendering techniques such as ray casting the rendered game scenes and any virtual game objects or line rendering or ray of the moving IR beams or IR Laser beams appear at specific locations in the real-world environment at a defined points in three dimensional space that intersect on the cartesian x,y,z planes;
- For each real world fired IR beam or IR Laser beam the moving augmented image of the IR beam or IR Laser beam together with lighting effects are displayed based on the determined trajectory and projectile, and respective formulated vector positions and coordinates in equal positions to that of the three-dimensional cartesian coordinates/positions velocity and distance to that of the real-world IR beam or IR Laser beam as this travels through the three dimensional space of the users real-world space;
A method of claim, of applying perspective projection and transformations based on users head and eye movements, to rendered game scenes, virtual game objects and paired virtual game objects and/or, line renderings or rays of moving IR beam or IR Laser beams comprising one or more of:
- The use of a virtual camera to provide the users perspective and FOV into the virtual world and augmented reality view of the game where;
- The user's augmented reality (“AR”) display apparatus, forward facing camera(s) may be used representationally in the game computation of virtual camera;
- The game virtual camera(s) may be based on the user's height, the position of the user's field of view or line of sight from either left and right eye determined through the head and eye tracking data, to provide a true first person camera perspective;
- The position of the virtual camera(s) may be adjusted automatically through transformations based on head tracking data on the user's head position, height from the ground, and orientation detectable via the users augmented reality (“AR”) display apparatus, sensor(s) together with their eye position and directional gaze captured from via the inward facing camera(s);
- Capturing and determining a user's head position, orientation, together with their eyes focal point and gaze from a head and motion tracking module and eye tracking module of a user's augmented reality (“AR”) display apparatus in which tracking data is captured through the use of the users;
- augmented reality (“AR”) display apparatus sensor(s);
- And;
- the apparatus inward facing camera(s) in which images and/or video are captured of the users left and right eye position, orientation and focal gaze where through the use of spatial mapping a virtual game object is reconstructed of the users left and right eye at the same position, orientation and focal gaze/directional vector, used to determine the users eye positions, orientation, focal points and directional gaze;
- Providing of tracking data by a head and motion tracking module and eye tracking module to the rendering module and/or games engine in the form of six dimensional or greater matrixes representing the users vector or head position and orientation expressed as three dimension Cartesian coordinates (x,y,z) values representing the users head position (x,y,z) and pitch, roll and yaw (p,r,y) values representing the users head orientation, to a rendering module together with their eye positions, orientation, directional gaze/focal point and directional vector;
- Applying transformations to virtual camera(s), game scenes and spawned or generated virtual game objects or line renderings based on detected changes in a user's head position and orientation together with changes to a user's eye positions, focal points and directional gaze in which:
- Users real-world view maybe represented by point A which has a value for x,y,z representing the 3D position that is to be projected;
- Point C has an x, y, z value representing the 3D position of the virtual camera used;
- Point O represents the orientation of the camera (also the users head and eye orientation) that is represented by Tait-Bryan angles;
- Point E is the viewers position (head position and eye position) relative to the surface of micro-display which goes through point C representing the camera view;
- Camera transformation may be applied using matrices or other forms of formulation without using matrices;
- Alternatively, the system may use homogeneous coordinates, in conjunction with other argument variables that factor in the distance of the user's eyes to the display surface;
- A view matrix that may be used to represent the virtual camera field of view;
- A projection matrix may be used to represent the projection of the perspective where projection maybe orthogonal to one of the main axis along the three-dimensional cartesian planes which maybe along the z axis;
- Transformations operations shall be applied to adjust the virtual camera's position, orientation and field of view in line with detected changes in a user's head position and orientation together with changes to a user's eye positions, focal points and directional gaze detectable via the users augmented reality (“AR”) display apparatus sensor(s) and inward facing camera(s) relative to that of changing game scenes and three-dimensional position, orientation of a displayed augmented virtual image of moving IR beam or IR Laser beam in accordance with its positions, directional vector, velocity and distance;
- Transformations operations shall be applied to game scenes, virtual game objects and/or line renderings as required to change the virtual game objects scale and rotation relative to the user's real-world location, rotation and field of view;
- Clipping of images based on the user's perspective projection and size of the micro display;
- Applying rasterisation, shading and fragmentation where:
- fragments are then processed by the games engine fragment shader to generate pixels which are assigned final colour values;
- the pixel values that includes their coordinates and colour are then written to a frame buffer from which the images are then drawn on the micro display;
- Displaying of rendered game scenes and virtual game objects where each display portion depending on the user's augmented reality (“AR”) display apparatus micro display type being LCD, LCOS or TOLED, is then illuminated/activated/excited or the LEDs are activated by the light display modules according to the pixel coordinates and colour in which a virtual augmented image may be displayed as two differing 2D stereoscopic images, or hologram or holographic image over the users real world FOV via the transparent micro-display.
A method of rendering a line rendering and displaying an animated line rendering of a real-world moving low light IR beam or IR Laser beam as augmented virtual image over a user's FOV via their augmented-reality (“AR”) display apparatus, micro-display comprising one or more of:
- Receiving and capturing sensor data from the sensor(s) of a real-world game object containing its vector position, three dimensional cartesian coordinates (x,y,z) values, orientation (p,r,y), directional vector/heading among other state and sensor data;
- Receiving and capturing I/O inputs by a real-world game object upon the physical pulling of a trigger or the activating of a real-world game object by other means including but not limited to voice commands, touch screen inputs etc. resulting in the projecting or emitting of an IR beam or IR Laser beam via the real world game object IR LED or IR Laser diode;
- Storing state information and sensor data on a real-world game objects vector position, cartesian coordinates, orientation, directional vector/heading, and rotation together with a Boolean flag indicating the firing of a beam upon the physical pulling of a trigger mechanism of a real-world game object such as IR Laser Gun Apparatus or activating by other means the firing of an IR beam or IR Laser beam;
- Transmitting state and sensor data by a real-world game object to a game server, or host, and/or another real-world game object and/or Augmented Reality (“AR”) display apparatus;
- Retrieving state and sensor data via the network from the game server or host or another real-world game object, on a moving IR beam or IR Laser beam indicated by a boolean flag and the real-world game objects state at the time that the beam was fired;
- Determining if an IR beam or IR Laser beam has been fired based on the retrieved state information retrieved via the network from the game server or host, or directly via another real-world game object that may be paired to the user receiving the state information or used by another user where state information may be transmitted via the Wifi or Bluetooth depending on the source;
- Determining the origin of an IR beam or IR Laser beam in the three-dimensional space of the real world environment from the retrieved state information and sensor data of the real world game object that fired the beam, determinable from its vector position or three dimensional cartesian coordinate in the form of x, y, and z values that was generated at the time of the invoking of the emitting of the IR beam or IR Laser beam;
- Formulating the projectile and angle of the beam in the three-dimensional space of the real world environment based on the retrieved state information and sensor data on the real-world game object that fired the beams own vector position or three dimensional cartesian coordinate, orientation and directional vector provided by the real-world game objects sensor(s) and tracking and used to formulate the directional vector and angle of the projectile of the IR beam or IR Laser beam used in the line rendering;
- Determining the Line Drawn Speed, Line Distance and Width based on an assumed velocity and distance of the real-world IR beam or IR Laser beam based on the type or class of real-world game object and its optical lens configuration;
- Generating a virtual game object or a series of virtual game objects together with a line rendering component using a games engine and rendering module on a user's head-mounted Augmented Reality (“AR”) Display Apparatus and/or a Game Server or Host.
- Invoking a draw line call in which the origin shall be used as the starting point, and the line shall be drawn per frame according to the formulated projectile and determined line draw speed, distance and width where the destination position/end point of the line rendering shall be determined based on the formulated projectile, and assumed or determined velocity and distance in which distance is derived from the type or class of real-world game object that fired the beam provided in the state information;
- Applying fragmentation, shading and rasterisation;
- Invoking the displaying a moving animated IR beam or IR Laser Beam by a user's head-mounted augmented reality (“AR”) Display Apparatus in which the line rendering is displayed as a stereoscopic, hologram or holographic augmented virtual image overlaid over the users real-world view in which the user perceives the moving animated augmented virtual image of the IR beam or IR Laser Beam as having 3D form and being located in the real-world environment, in which the moving animated image of the line rendering is displayed as moving in the exact same coordinates and at the same rate of velocity as that of the real-world moving IR beam or IR Laser beam.
A further method of claim 195 of displaying a moving augmented virtual image of an IR beam or IR Laser Beam at the same point of origin and moving along equal points in the three dimensional space of the users real-world environment, direction, velocity and distance as that of a real projected IR beam or IR Laser beam of light in real-world using a line rendering or ray comprising one or more of:
- Generating state information on the firing of an IR beam or IR Laser beam in the form of a boolean flag and capturing sensor data from a real-world game objects sensor(s), on the apparatus vector position, world coordinates, orientation and directional heading/directional vector at time of a user pulling the trigger or activating the real-world game object apparatus;
- Retrieving state information and sensor data via the network on a real-world game objects vector position/world coordinates, orientation and directional heading/directional vector at time of a user pulling the trigger or activating the apparatus;
- Formulating a starting point (represented by x,y,z values) in three dimensional space in which the IR Beam/IR Laser beam originated referred to as the origin, where the origin is derived from retrieved sensor data containing the real world game objects position or world coordinates (in the form of x,y,z values) at the time of the trigger being pulled or the device being activated, in which the resulting IR beam or IR Laser beam originated;
- Formulating the projectile/trajectory of an IR beam or IR Laser beam from a determined starting point/origin in three dimensional space in which the IR beam or IR Laser beam originated based on the retrieved state information and sensor data of the real-world game object, containing its position expressed in the form of x,y,z values, together with its orientation in the form a pitch, roll and yaw values (p,r,y) and directional heading that maybe expressed as a direction vector, where the angle of the line or ray is based on the IR beam or IR Laser Beam angle determined by the apparatus pitch, roll and yaw (p,r,y) and its directional heading/vector at the time the device trigger was pulled;
- Use of a line or ray method to represent the directional vector of the IRbeam or Laser Beam, where a line or ray is generated using a rendering module and or games engine based on the formulated starting point/positional vector (origin) and projectile/trajectory using the retrieved sensor data containing the coordinates (x,y,z), orientation (p,r,y) and direction vector of a real-world game object at the time of when the trigger mechanism was pulled;
- Determining the line or rays positions and heading along the x,y,z axis in three dimensional space using vector and/or parametric equations;
- Determining the length/distance of the line or ray based on real-world distance range of an IR beam or IR Laser Beam of an real-world game object apparatus such as an IR Laser Gun Apparatus, where the length of the line or ray is based on the projectile distance of a real-world game object apparatus, which is converted into a number of points along the line or ray;
- Defining multiple points along the line or ray between two points, points A the origin and point B the destination in which each point represents a different vector position and set of (x,y,z) values along the line or ray;
- Applying the determined projectile/trajectory of an IR beam or IR Laser beam to a line rendering or ray;
- Incrementing the length of the line or ray at incremental equal values applied across each point according to an assumed value “X” the distance of the IR beam or IR Laser beam and an assumed velocity;
- Applying assumed variables such as gravity, velocity, wind where velocity is assumed at a constant rate, and an IR beam or IR Laser Beam since it is a form of light is unaffected by physical elements like wind and gravity;
- Varying the origin, directional vector and projectile/trajectory of each fired IR beam or IR Laser Beam according to the position and orientation of each Optical lens piece(s) on a real-world game objects model/mesh in which where an IR Shield Apparatus and IR Proximity Mine Apparatus may have multiple optical pieces and IR LEDs or IR Laser Diodes offsets maybe applied based on the known position, orientation and directional vector of each IR LED or IR Laser diode relational to the devices model coordinates;
- Varying the formulated projectile/trajectory and directional vector of each IR beam or IR Laser beam according a real-world game objects orientation, tilt angle used to formulate the angle of the line;
- Rendering an animation of the fired IR beam or IR Laser Beam in exact precise velocity, position and directional heading/direction vector in 3D space to that of each projected real world IR beam or IR Laser Beam generated from the pulling of the trigger mechanism of a real-world game object or the activation of the apparatus by other means detailed within the embodiments of the invention;
- Displaying a moving rendered augmented virtual image and ray of an IR beam or IR Laser Beam from the same point of origin and moving along equal points in the three dimensional space of the users real-world environment, direction, velocity and distance as that of a real projected IR beam or IR Laser beam light in real-world based on the determined and formulated line or ray position/origin, distance, projectile/trajectory and assumed variables where:
- the users augmented reality (“AR”) display apparatus micro-display is then illuminated/activated/excited or the LEDs are activated by the light display modules according to the pixel coordinates and colour in which a virtual augmented of the of the line rendering or ray of an IR beam or IR Laser beam maybe displayed as two differing 2D stereoscopic images, or a hologram or holographic image;
- Applying transformations as an update function to a line rendering or ray to move forward the line or ray to a new point and vector position in accordance with the determined directional vector/heading, projectile/trajectory, positions or points along the line or ray, velocity, length and range/distance of the real IR beam or IR Laser beam, where:
- the rate at which the transformations are applied to the virtual game object and line rendering or ray of the IR Beam/IR Laser beam and changes to the positions or points along the formulated line or ray are based on the assumed velocity of the real-world moving IR beam or IR Laser beam;
- Displaying the animation of each IR beam or IR Laser Beam using as animated sequence of images or video in which images may be rendered using ray tracing or other high resolution rending techniques such as CGI where the updated virtual images of the IR beam or IR Laser beam are displayed 2D stereoscopic images, or a hologram or holographic image as a sequence of images or video over the users FOV via the users augmented reality (“AR”) display apparatus micro-display in which this appears as a moving image;
- Applying transformation based on changes to a user's head position and orientation together with their eye tracking movements, based on sensor and eye tracking data retrieved from their wearable augmented reality (“AR”) head mounted display apparatus where this is the display apparatus used;
A method of computationally determining a collision of an IR beam or IR Laser beam with a with a real-world user, real-world game object, real-world AI character, surface and/or a virtual-game object or remote users by a client or the game server or host and determining and invoking a collision response comprising one or more of:
- Retrieving of state information and sensor data on a fired IR beam or IR Laser beam and a real-world game object used to fire the beam via the network from the game server or host or directly from the real-world that was used to fire the beam, in which the state and sensor data contains the position (x,y,z), orientation (p,r,y), rotation and directional vector/projectile of real-world game object at the point of the apparatus trigger mechanism being pulled or the device being activated by other means;
- Retrieving of state information and sensor data containing the position (x,y,z), orientation (p,r,y), rotation and directional vector/projectile on another client and/or real-world game object which may comprise of a user's augmented reality (“AR”) display apparatus or a wearable or handheld real-world game object or a real-world AI character, where the state and sensor data may be retrieved locally in the case of an object's client module checking for collisions with itself, or via the network from the game server or host or directly from another real-world game object;
- Use of a collision detection module and collision detection algorithms that support priori or posteriori methods of collision detection;
- Use of a linear algebra, computational geometry, model coordinates, physics data, volumetric and surface data;
- Determining of a hit/collision by a collision manager module using either:
- Priori based collision detection methods in which a collision shall be predicted/formulated computationally based on the determined;
- origin, projectile/directional vector and positions (x,y,z) of an IR beam or IR Laser beam derived from the state and sensor data captured from an real-world game object sensor(s) at the time of its trigger mechanism being pulled or being activated by other means, relative to the determined projectile/directional vector, velocity, world coordinates/vector positions(x,y,z) and model coordinates of another user or object that maybe a real-world user, real-world game object, real-world AI character, surface and/or a virtual-game object or remote user;
- Where:
- the collision maybe predicted and formulated based on the retrieved state and sensor data together with a number of physics variables including velocity, distance and the time variable;
- parametric equation maybe used to find the point of collision,
- the physics variables of a moving IR beam or IR Laser beam such as the velocity of a maybe assumed as a constant at a pre-assumed rate of velocity and distance maybe assumed based on an assumed range of a real-world game objects IR beam or IR Laser beam;
- the velocity of another object maybe captured via its sensor(s);
- Or posteriori collision detection methods where the collision manager module shall check based on the current state if two objects are intersecting based on their relative projectile/directional vector, velocity, world coordinates/positions (x,y,z) and optionally model coordinates in which the IR and Laser beams origin, projectile/directional vector and positions (x,y,z) are derived from the sensor data captured from an real-world game object at the time of its trigger mechanism being pulled or being activated by other means, where together with the captured relative current determined projectile/directional vector, velocity, world coordinates/vector positions(x,y,z) and together with optionally the model coordinates of another user or object a collision maybe determined;
- Use of collision detection techniques including but not limited to;
- pruning, pairwise checks, spatial partitioning methods, bounding boxes, hit boxes, n-body pruning, pairwise checks using bounding rectangles or circles hit boxes, exact pairwise checks, Triangle Centroid Segments, hitscan or raycasting using vector and parametric equations of a line, frame by frame analysis, time of impact (TOI) or polygon based triangulation analysis etc.;
- Determining the point of collision (intersecting points) based on the relative positions/projectile of two objects and optionally variables such as velocity and time where priori methods are used;
- Estimating the time of impact (TOI) by linear interpolation/Newton's method where priori methods are used;
- Simulating the resulting changes by applying physics models and algorithms;
- Determining a collision response in which this may use a physics engine, physics simulators, physics models and algorithms together with physics data in the determining, computation and simulation of a resulting collision response where:
- the time of impact (TOI) and a set of intersection points may be used in the simulation of the outcome resulting from the collision;
- physics data, linear algebra, computational geometry together with volumetric and surface data maybe used in the modelling and/or computing of an outcome of two objects intersecting together with use of algorithms where these variables may consist of the mass of an object, momentum, velocity, force and time, in which the opposing values of each of these variables for each object shall be used in the determination of the outcome;
- the laws of physics maybe applied through physics simulations in which the use of Newtonian physics maybe used in simulation, modelling and algorithms used in determining of an outcome resulting from a collision or intersection;
- rigid body or soft body simulations may be used;
- Invoking of a collision response by a client or server which may include:
- The display of decal and diffuse lighting effects at the point of collision (x,y,z) over a user or an object or surface which may be displayed directly on the surface display panel/screen faces of a real-world game object or via the external projectors where the rendered decal and diffuse lighting effects are visible to the naked eye, or maybe displayed as an virtual image via the users augmented-reality (“AR”) display apparatus, micro-display as a form of two 2D stereoscopic images or a 3D holographic image or hologram that is augmented over a user's real-world view;
- And/or the invoking of a ragdoll simulation where a hit is detected on a real-world AI character;
- And/or the physical feedback via a user's haptic modules on their handheld or wearable real-world game objects including but not limited to their IR Mixed Reality Suit or IR Mixed Reality vest where feedback maybe addressable to one or more haptic modules and directional based on the determined point of collision and projectile/angle of where the beams virtual game object and/or line rendering or ray intersects with a user body and/or object's model coordinates;
- And/or the generating of 3D perceptual based audio sound effects in which the user may hear a sound effect of a hit at the point of collision via a real-world game objects speakers, or the use of external speakers or via the speakers of their augmented-reality (“AR”) display in which techniques such as ITD, ILD, IPD and HRTF functions may be used to correcting place the 3D audio effect at the point of collision;
- Applying transformations to generated virtual game objects used in the rendering of decal and diffuse lighting together with its respective audio and lighting components where:
- In the case of users Augmented Reality (“AR”) head mounted display apparatus apply transforms shall be applied to the virtual camera and/or created virtual game objects based on detected changes to a user's head position and orientation together with their eye position and directional gaze detectable via the head tracking module and eye tracking module using the apparatus sensor(s) and inward facing camera(s) together with spatial mapping according to the users eye positions and orientations and directional gaze, where transforms and new vector values shall be applied according to change the vector position, rotation, scale of the displayed decal/diffuse lighting effects based on detected changes;
- In the case of external projectors transforms shall be applied based on the detection of changes in an object and/or users position through the use of two or more methods including:
- Based on sensor data retrieved via the network from a client that may be a user's Augmented Reality (“AR”) head mounted display or a real-world game object;
- Or based on detected changes via the use of external camera(s) using spatial mapping or the use of structural light scanning techniques using specialist laser apparatus/laser planers or projection using incoherent light and specialist spatial light modulators with the digital external projectors and one or more external camera(s) where the game server or host rendering module and games engine shall construct wireframes or mesh and 3D models of the users real-world space in which changes in an object or users position is detectable;
- In the case of real-world game object transforms shall be based on detection of changes via a real-world game objects sensor(s) in the vector position (x,y,z), rotation or orientation of a real-world game object;
- the same transformations are applied to any generated audio components and 3D positional audio effects of a created virtual game object and rendered decal effects, using the same vector values as applied to created virtual game object which are converted as image inputs into a HRIR database in this process;
- Displaying updated decal and diffuse lighting effects via the applicable display apparatus/method;
- Outputting the updated or new 3D positional audio effects resulting from the transform via the speakers of the applicable apparatus;
A method of detecting a hit or collision from an IR beam or IR Laser beam non-computationally on a real-world game object, real-world space, users or a real-world AI Character and displaying virtual images and augmented game scenes comprising one or more of:
- Detecting a hit or collision of IR beam or IR Laser beam on a real-world object, surface, user or real-world AI character via IR Receiver/IR Sensor through the detecting of IR emissions and IR radiation from an IR beam or IR Laser Beam projected by an IR LED or infrared laser diode of another real-world game object resulting from the pulling the trigger mechanism or the activating of the apparatus by other means, where the IR Receiver/Sensor maybe;
- Part of a real-world game object apparatus where this maybe a form of:
- and held real-world game object such as an IR Laser Gun Apparatus, IR Proximity Mine Apparatus or IR Shield Apparatus;
- or a real world AI Character;
- or a wearable real-world game object such an IR Mixed Reality Vest or IR Mixed Reality Suit Apparatus worn on a user's body;
- or IR Mixed Reality Scenery that maybe placed in or on the surrounding walls, ceilings, floors and structure of the user's real world space and/or laser tag arena;
- Or part of the surrounding walls, ceilings, floors, objects and structure of the user's real world space and/or laser tag arena;
- Or a wearable form of an IR Receiver/IR Sensor such as those on a user's wearable IR Mixed Reality Vest or IR Mixed Reality Suit, or worn separately by a user's body;
- Or part of a user's augmented reality (“AR”) display apparatus;
- A form of fibre optic sensor, photo optic sensor, IR detector, photo-detector or a form of photodiodes or phototransistors, or may be a form of active or passive infrared sensor;
- Part of the same component or separate components on the real-world game objects circuit board where the IR Sensor is configurable to detect light emissions and the IR Receiver is configurable to detect differences in radiation and frequencies where the other real-world game object has an IR Transmitter used in the determination of hit;
- Invoking of physical feedback via one or more haptic modules which:
- may be paired directly with an IR Sensor to give direct feedback based on the detection of light and/or radiation;
- may differ in the physical feedback provided with differing classifications of the weapon detected in the IR signal in the form of raw binary data transmitted in the IR Beam or IR Laser beam where the hit is detected by an IR Sensor of a real-world game object features an IR demultiplexer capable of demultiplexing data via an IR Beam or IR Laser Beam;
- may be positional and directional based the direction of the beam and the relative determinable position of the IR Sensor/point of collision;
- Invoking the rendering and display of augmented game scenes and virtual images:
- Directly via one or more of a real-world game objects surface display screen/panel faces in which rendered virtual images, animations or video including decal and diffuse lighting may be invoked as a collision response:
- directly by the real-world game object apparatus collision module or client module in response to the detection of IR emissions and/or radiation by its IR Sensor(s);
- Or maybe invoked based on the receiving of state information or the registering of a hit via the network with the game server or host;
- Or based on the transmitting and receiving of state and registered hits between clients directly;
- where the real-world game object maybe a form of a wearable IR Mixed reality Vest, IR Mixed Reality Suit Apparatus, or a handheld IR Laser Gun Apparatus, IR Shield Apparatus or real-world AI Character or a IR Mixed Reality Scenery where these real-world game object apparatus feature a multiple surface display screen/panel faces in the form of a LCOS, TOLED, LCD or Graphene/Graphite display and triangles or quads which forms a 3D mesh over the surfaces of a real-world game object configurable of displaying rendered texture maps, decal and lighting effects generated by a rendering module and/or games engine in response to a collision detection;
- where in the case of directly displaying these rendered effects over a real-world game objects surface display panel/screen faces, this shall be mapped to the relative position of the display on the real-world game objects model in which the decal and diffuse lighting will appear on adjacent surface display panel/screen faces to that of the position of the IR sensors and IR Receivers that detected IR emissions/IR radiation;
- Or alternatively different types of rendered texture maps, visual elements such as decal and lighting effects may be invoked directly based on raw binary data in the IR signal;
- Or via a user's head mounted augmented reality display apparatus, micro-display in the form of a holographic image, hologram or two stereoscopic 2D images among other forms of images or video that are augmented over the user's real-world view of the space of the laser tag game or that of arena surrounding walls, ceilings, floors, objects and structure at the point of intersection (x,y.z) that the IR beam or IR Laser beam intersected with or was detected by an IR Sensor/Receiver, where the image is augmented using the spatial mapping data, volumetric, geometric, mesh data, 3D models of user's real-world space;
- Or the use of external projectors by the game server or host as augmented virtual images or video over the real-world objects, users and space of the laser tag game or that of laser tag arena surrounding walls, ceilings, floors, and structure for where the IR beam or IR Laser beam intersected and was detected by an IR Receiver/Sensor which may be seen by the naked eye, in which this is supported through the use of projection mapping/video mapping/spatial augmented reality and/or 3D projection mapping techniques;
- Where decal and diffuse lighting shall be displayed relative to the position/model coordinates of the IR sensors and IR Receivers on a user's body and/or a real-world game object or surrounding surfaces;
A method of generating and positioning 3D audio effects, together with the display of a moving augmented virtual image of an IR beam or IR Laser beam at precise locations in the three dimensional space of the real-world space, equal to that of the light emitted from a real moving IR beam or IR Laser beam, comprising one or more of:
- Adding an 3D positional or spatial audio component to the generated virtual-game object of an IR beam or IR Laser Beam;
- Converting the 3D coordinates and positions of the moving IR beam or IR Laser Beam virtual-game object, line rendering or ray that may include vector and/or three-dimensional cartesian coordinates with x,y,z values, into image inputs which are input into a HRIR database to generate 3D positional audio sound effects of the moving IR beam or IR Laser beam;
- Emulating the biological mechanisms of binaural hearing through the use of digital signal processing (“DSP”) algorithms and sound localisation technologies to emulate the direction and sound location, to create perceptual based 3D sound localisation, in which the user perceives the location of the output positional audio sound effects to be in a specific location in the three dimensional space of their real world equal to that of the positions of the light emitted from a real moving IR beam or IR Laser Beam;
- The use of 3D spatial sound based techniques, 3D spatial audio, Ambisonics and/or 3D perceptual sound localisation techniques using DSP algorithms, ITD, IID, IPD, ILD and HRTF functions where based on the input image coordinates into a HRIR database, sound waves may be processed and manipulated to achieve perceptual 3D audio localisation of objects in which a user perceives the audio coming from a specifiable location in three-dimensional space;
- Applying transformations to the generated audio component/3D positional audio sound effects and the virtual game object, line rendering, ray of the IR beam or IR Laser beam to move its position to a new vector position in which new image inputs are passed into the HRIR data to generate new 3D positional audio sound effects;
- Using a listener's ear to simulate the effect of the audio source being behind, above, below, in front or either side of the user as the IR or Laser beam passes a user, in which the positioning of the 3D positional audio sound effects is based on the computed origin, cartesian coordinates (x,y,z), projectile/trajectory or directional vector of that of the real IR beam or IR laser beam and that of the generated virtual game object and/or line rendering or ray relative to that of to the user's head position/three dimension Cartesian coordinates and orientation (P,R,Y) retrieved from the head tracking module and sensor(s) of the users wearable augmented reality (“AR”) head-mounted display apparatus 1 or via the game server or host via the use of external projectors and spatial mapping;
- Displaying rendered augmented images and visual lighting effects of a moving IR beam or IR Laser Beam via:
- A user's augmented reality (“AR”) display apparatus 1, micro display, in the form of two differing 2D stereoscopic images, a hologram or holographic image;
- Or via external projectors
- Or via a real-world game object surface display panel/screen faces;
- Outputting in parallel the generated 3D sound effects, and sound waves of the audio component of the IR beam or IR Laser beam virtual game object at determined three dimensional positions via respectively:
- a user's augmented reality (“AR”) display apparatus 1, speakers;
- Or via external speakers;
- Or via a real-world game objects speakers;
- Where the user perceives the 3D sound effect to be in precise locations in the three dimensional space of their real world equal to that of the positions of the displayed augmented visual effects of the moving IR beam or IR Laser Beam and that of the light emitted from a real moving IR beam or IR Laser beam fired into the users real-world space;
- Applying Transformations where dependent on the apparatus used to output the 3D sound affect;
- Transformations shall be applied to the generated audio component/3D positional audio sound effect and the virtual game object, line rendering, ray of the IR beam or IR Laser beam based on changes to the user's head position and orientation detectable via the head tracking module and sensor(s) of the users augmented reality (“AR”) display apparatus or via external camera(s) using spatial mapping;
- Or
- Transformations shall be applied in real-time to the 3D audio source files according to changes in the position (x,y,z), orientation (p,r,y) and directional vector of the users real-world game object that is outputting the 3D audio effect via its speakers, where the direction of the sound in three-dimensional space shall remain relational to the original origin, projectile and directional vector heading of the moving IR beam or IR Laser beams in which:
- The sound outputs and audio source files shall be manipulated using 3D perceptual sound localisation techniques including but not limited to the use ITD, IID, ILD and HRTF;
- The real-world game objects own positional vector and orientation shall be provided as six dimensional matrixes provided by its tracking module and sensor(s) to a games engine and/or rendering together with an audio manager in handling of transformations;
- 3D spatial sound and perceptual based 3D locational sound effects may be positionally placed on a three dimensional coordinate of the planes of the Cartesian coordinates x, y, z axes, relational to the real-world game objects own position and orientation based on its pitch, roll and yaw together with its vector position (x,y,z).
A method of externally projecting augmented-reality images of game scenes and virtual-game objects including rendered virtual game objects, line renderings and/or rays of moving IR beam or IR Laser beams and lighting effects together with decal and diffuse lighting from a hit over a real-world user, surface or object which are visible to the naked eye comprising one or more of:
- Using generated wireframes, mesh data or 3D model of the objects or spaces structure and volume, together with any generated virtual game objects by the game server or hosts rendering module and/or games engine to apply texture mapping and other surface materials, shading and lighting effects to real-world objects, users, surfaces and spaces in which:
- the generated wireframes, mesh data, 3D models and virtual game objects may be generated via one or more of:
- forms of structural light scanners using structural light scanning techniques;
- or
- the use of external camera(s) where:
- these camera(s) may be a form of depth aware camera(s), structured light, time-of-flight camera(s) or stereo camera(s)/stereo computer vision camera(s) which may be used with a positioning reference by means of a beam of light which may be infrared where infrared emitters or a lexian stripe may be used in conjunction;
- spatial mapping or other forms of techniques such as structure light scanning may be used;
- The optional use of ray casting or other forms of rendering techniques such radiosity, ray tracing, volume rendering and/or optimisation of the rendering processing using baking techniques;
- The optional use of the game server or host GPUs in this process in the hardware acceleration of the performing of the game and rendering operations.
- The generating of rendered virtual images as an output of the rendering process that maybe rasterised and fragmented prior to the projecting of images;
- Projection and augmenting or superimposing rendered virtual images and/or video from the constructed 3D models and virtual game objects of a real-world objects, users or surfaces via the external projectors in the form of digital projectors capable of supporting 2200 or more lumens and through the use of projection mapping techniques, also referred to as video mapping, spatial augmented reality and/or 3D projection mapping techniques by the game server or host where:
- a display projection module software maps the corners of the image or video to the designated surfaces of the real-world object, user or surface;
- the rendered virtual image is projected back onto the original real-world object based on the objects original Cartesian coordinates in three dimensional spaces (x,y,z) and its orientation (p,r,y) through the illuminating or activating of the external projector by the game server or host;
- the determination of the position and display of the rendered image or video maybe also determinable by the use of three dimensional coordinates and other volumetric and vector variables provided by an observer module as part of the spatial mapping data that may be provided in the form of a mesh or a 3D model or through the translation of cloud points into vertices/model coordinates;
- the image light is directed based on the model coordinates and world coordinates of the real-world object, surface or user using external 3D mapping projection techniques;
- during this process the projected rendered virtual image or video of the new objects surfaces and texture-maps are projected onto the real-world object where the light of the projection of the images pixels is placed relationally to that of the world coordinates (x,y,z), model coordinates and orientation (p, r, y) of the real world object in which the light corresponding to the pixels intersects with object precisely in which this process is referred to as 3D projection mapping or video mapping in which a display projection module software supports this process;
- the augmented virtual 3D images or video maybe projected over the real-world users, objects and space of the game or that of laser tag arena surrounding walls, ceilings, floors, objects and structure which may be seen by the users naked eyes with no visual aids;
- Applying transformations based on detected changes in a real-world object, users or AI characters position derived from retrieved state information and sensor data or based on updated mesh data, in which:
- Adjustments maybe invoked in real-time to the position, angle/directional vector, orientation and/or rotation of an external projects where:
- The projector is mounted on a motorised plate and adjustable arm which is supported by two or more motorised servos on 2-Axis allowing the system to make adjustments in the projection direction of the external projector by changing the rotation and tilt or angle of the external projector;
- Actuators may also be used on a wheel and rod system to move the projector along a rail system or the projector may be fixed on a motorised wheel based frame in which it can be moved forward and backward above the users head;
- The external projector may be mounted on a motorised rail system mounted to the ceilings and walls of the laser tag arena in which it sits on a plate that has motorised wheels allowing movements along the rails as basic forward or backward movements and limited adjustments may be made to the coordinates (x,y,z) of the external projector in the tracking and projecting of 3D projected augmented images or video onto moving objects or users;
- Adjustments maybe based on the position (x,y,z) of a person, object or AI Character based on:
- their known coordinates to the game server or host as part of the maintained state information or upon updated state information in which changes in the position (x,y,z) and orientation (p,r,y) are provided by the client modules of all real-world game objects to the game server or host via the network which is used in this process;
- Or based on observed movement or changes via the external camera(s) and observed updates by the game server or Host's Observer Module using spatial mapping techniques among other techniques such as structural light scanning in which changes to the position (x,y,z) and orientation (p,r,y) of the real-world objects, users, game objects, AI character's 96 are captured in the form of updated mesh data/surface data in this process;
- In this process of adjustments, transformations may be applied to the virtual game objects/3D models of these objects and/or the external projector itself in order to maintain the correct projection of the augmented virtual images or video on existing objects;
- The external projectors maybe rotated from side to side and tilted up or down by the use of two or more motorised servos controllable via a micro-controller or driver which provides a 2-Axis FPV cradle head with a dual servo and steering gear in which based on the target objects orientation, position that 3D images are being projected onto, the external projectors, rotation, orientation, pan and tilt may be adjusted;
- The external projectors maybe adjusted on a tilting mechanical motorised mechanism in which an actuator on a multiple rod mechanism may be used to adjust the position of the external projector in terms of its orientation p, r, y and it's directional facing of its lens into the three dimensional space of the game in terms of its directional heading on the x, y, z axis of the three dimensional cartesian planes;
- The external projector(s) may be mounted on individual plates which support the pan, tilt and rotation of the external projectors.
A method of generating mesh data, 3D models and virtual game objects of a real-world space, surfaces and objects using structural light scanning, used in the display of game scenes and virtual-game objects augmented over a user's real-world space, surfaces and objects comprising:
- The use of specialist mounted Laser Planers that feature two wide planar lasers and the use of Laser interference scanning techniques to scan the three-dimensional real-world surrounding surfaces, objects and users, to generate 3D models, mesh data and virtual game objects which contain geometric, depth and structural data together model coordinates, vertices, edges, faces and surfaces of surrounding surfaces such as walls, ceilings, floors together with objects and users in the real-world which may be used in the rendering and game computation operations of the system;
- Or the use of Spatial Light modulators in conjunction with the use of one or more digital external projectors and one or more external camera(s) using incoherent light techniques where:
- the system is configurable to generate 3D models, mesh data, cloud of points and virtual game objects containing three-dimensional model coordinates (x,y,z) values, vertices, edges, faces and surfaces of a scanned object or user through the projecting of a narrow band of light on the three-dimensional shape of an object, surface or space in which this shall generate a line of illumination which is distorted in appearance from the other prospectives to that of the projector in which this shall be used to determine the geometric structure of the shape of an object or surface based on the light section;
- the 3D coordinates of a real-world object or space is extracted based on the displacement of the stripes, in which for every point, face or surface of a real-world object or space a set of three dimensional coordinates may be captured which can be converted into vertices and model coordinates (x,y,z);
- virtual game objects representing spaces, surfaces or objects in the real-world are generated through the translation of cloud points to vertices, faces, polygons and surfaces etc. generated through the use of structural light scanning techniques;
- the system is configurable to generate cloud or points (CoP), mesh data, 3D models and virtual game objects containing geometric, depth and structural data of a scanned real-world space, surface, object or a user together with the three-dimensional model coordinates (x,y,z) values, vertices, edges, faces and surfaces of a scanned real-world space, surface, object or a user which may be used in the rendering and game computational operations of the system;
A method of generating and displaying an Avatar over a user's body visible to naked eye together with generating hit boxes or hit box hierarchies that maybe used in collision detecting comprising one or more of:
- Scanning of a user's human body geometric structure using:
- Structural Light Scanners located in the user real world space and/or laser tag arena where the game server or host's observer module through the use of:
- Laser Planers that feature two wide planar lasers and laser interference scanning techniques is configurable to generate cloud or points (CoP), mesh data and 3D model data of a scanned object or a user body containing three-dimensional model coordinates (x,y,z) values, vertices, edges, faces and surfaces of a scanned object or user;
- Or;
- Spatial Light modulators in conjunction with the use of one or more digital external projectors and one or more external camera(s) using incoherent light techniques is configurable to generate 3D models, mesh data and virtual game objects containing three-dimensional model coordinates (x,y,z) values, vertices, edges, faces and surfaces of a scanned object or user;
- Or alternatively recording images or video of a user's body via one or more external cameras located in the users real-world space and/or laser tag arena and using spatial mapping techniques by the game server or host's observer module to generate mesh and 3D model data of a user's body containing a user's three-dimensional model coordinates (x,y,z) values, vertices, edges, faces and surfaces;
- Generating mesh data and a precise 3D model of user's body comprising vertices, polygons, triangles, faces, surfaces, edges and model coordinates together with a virtual game object by the game server or host representing a user's body, and generating of user's avatar using a rendering module;
- Generating optional hit boxes and/or hit box hierarchies that are attached to users virtual-game object and body in a computational sense that maybe used in collision detections;
- Presenting a user's rendered Avatar via the users IR Mixed Reality Suit surface display panel/screen faces, or via the external projectors or via a user's augmented reality (“AR”) display apparatus, micro-display;
- Providing the means for users to selectively change their Avatar via:
- voice command(s);
- or hand gesture inputs;
- Generating different rendered Avatar using a user's 3D model by a rendering module based on a user's detected inputs and selection which maybe customisable;
- Presenting a displaying a user's selected Avatar via:
- A users IR Mixed Reality Suit the surface display panel/screen faces to form a 3D rendered avatar that partially or fully covers the users body which is visible to the naked eye;
- Or via the external projectors using 3D projection mapping, video mapping and/or spatial mapping techniques where the user's avatar is projected over the users body;
- Or via a user's augmented reality (“AR”) display apparatus micro display as an augmented-reality virtual image, in which the user's avatar is displayed augmented over the users body via their micro-display as a form of two 2D stereoscopic images or a holographic image or hologram;
- In which in either display mechanism:
- the rendered texture-maps of a selected an Avatar are rendered and displayed over user's body using the generated mesh data, 3D model and virtual-game object of a user's body and avatar rendered through the use of a rendering module.
A method of supporting enhanced Laser tag and IR Mixed reality massive multi-player network gaming in which the game state, game scenes, objects and three-dimensional spaces coexist in multiple geographic locations, real-world spaces and time, comprising one or more of:
- Establishing of connections via a network between remote clients, local clients and the game server or host which maybe over the internet or a private network where a remote client maybe a real-world game object and/or a user's augmented-reality (“AR”) display apparatus;
- Capturing, generating and maintaining of events, inputs, state and sensor data from local or remote user(s) clients and real-world game objects in which data may be retrieved from the client modules and sensor(s) of:
- A user's augmented reality (“AR”) display apparatus;
- And/or a real-world game object including but not limited to an IR Laser Gun Apparatus or a real-world AI character:
- Where state information and sensor data as detailed in the embodiments of the invention may contain:
- Sensor data on position (x,y,z), orientation, directional vector of a user or real-world game object in the three-dimensional space of the real-world among other sensor data;
- State information on the firing of an IR beam or IR Laser beam in the form of a boolean flag together with sensor data on a real-world game objects position, orientation, directional vector at the time of its trigger being pulled or being activated by other means, maybe used in the determining the origin, positions and projectile of an IR beam or IR Laser beam and its virtual game object and line rendering or ray;
- State information on the health of a user or a real-world game object based on determined collisions;
- A user's head position, orientation, together with their eye position, orientation and focal point captured via a user's augmented-reality (“AR”) display apparatus sensor(s) and inward facing camera(s) using spatial mapping to respectively, which may be used to apply transformations in the placement of game scenes and virtual game objects accordingly;
- A user input or event which maybe an I/O input, touch screen or gesture inputs etc.;
- Receiving, transmitting and handling of synchronisation of events, inputs, data, video, images, audio and state between clients directly and between clients in differing geographic locations and the game server or host;
- Maintaining of a converged game state where the game exists in two or more geographic locations, real-world spaces and time both computationally and in reality, in which the three-dimensional space of the game is an exact replica geometrically to that of the laser tag arena or the game space in either geographic location either computationally in the virtual sense through the use of mesh data and/or physically where:
- The game state is maintained by both the local clients and the game server or host which represents both the individual user's local representation in the game state on the local users augmented reality (“AR”) display apparatus, and that of all local and remote real-world users, clients and real-world AI characters as well as spawned virtual-game game objects respectively by the game server or host;
- Remote users, objects and real-world AI characters together with any spawned virtual-game objects from one geographic location are displayed augmented over a user's real-space in another geographic location at their relative vector position (x,y,z) via a user's augmented reality (“AR”) display apparatus or vice versa;
- Handling by game server or host together with the clients of:
- Events/inputs that maybe generated by a physical action such as the pulling of a trigger mechanism or by other means;
- Creation and management of dynamically spawned virtual game objects and line renderings or rays of an IR beam or IR Laser beam or virtual fire by clients resulting from an event or input which may result from a physical pulling of a real-world game objects trigger mechanism or the activating of the apparatus by other means such as voice commands;
- Mesh Data where generated global mesh data and 3D model data of the real-world space maybe retrieved via the game server or host by clients;
- Scene management and objects in the display of augmented game scenes of overlaid texture-maps, lighting, materials together with virtual game objects over real-world spaces and objects including the display of dynamically spawned virtual game objects and line renderings of moving IR beam or IR Laser beams;
- Determining of Collisions:
- Using of posteriori or priori collision detection techniques in the determination of collisions of two objects intersecting or a dynamically spawned virtual game object including the virtual game object and/or line rendering or ray of an IR beam or IR Laser beam intersecting with another local or remote user, real-world object, surface, real-world game object, real-world AI character and/or a virtual game object;
- Where:
- collisions maybe determined based the relative coordinates/vector position (x,y,z), orientation (p,r,y) and directional vector/projectile of another object or users coordinates captured via sensor(s);
- transformations maybe applied to the paired virtual game objects of real-world game objects and/or based on captured sensor data which may be used in this process;
- techniques such as dead reckoning and network prediction may be used;
- Collision responses including the collision or hit of an IR beam or IR Laser beam with another object or user;
- Where:
- The action or a remote user or a real-world AI character in the form of an input such as the pulling of a trigger mechanism of a real-world game object and firing of an IR beam or IR Laser beam in one geographic location and real-world three dimensional space and time can effect the game state in the real-world space, time and virtual world of the laser tag game in another geographic location and vice versa:
- Where an IR beam or IR Laser beam fired in one geographic location may hit a remote object, surface or user in another geographic location based on its relative positions and projectile where a game real-world space exists in parallel computationally in which:
- in the case of a user being hit this may invoke the haptics module on a user's IR Mixed Reality Suit at the point or nearest point for where the beam from the other geographic location intersected with the user;
- decal and diffuse lighting effects displayed over the body of a user or another object or surface in another geographic resulting from the determining of a hit which may be displayed over the surface display panel/screen faces of real-world game objects directly such as users IR Mixed Reality Suit or via users augmented reality (“AR”) display apparatus or via external projectors as per the embodiments of the invention and respective relating claims;
- the physical actions of one user in one geographic location, real-world three-dimensional space and time can effect the visual and physical state of the game and other users or objects in another locations real-world three-dimensional space and time where the game state, world space and time coexists computationally in both geographic locations;
A method of making a physical real-world game object present in user's real-world space invisible to the human eye comprising one or more of:
- Capturing of a user's head position and orientation together with their eye position, orientation, directional gaze and focal point through the recording and processing of images or video by the real-world game using its own spatial mapping, observer module, facial recognition and eye tracking capabilities, where images and video maybe retrieved via the network through:
- the use of external camera(s);
- And/or:
- inward facing camera(s) of a user's augmented reality (“AR”) display apparatus;
- Capturing of a user's head position and orientation and eye tracking data via:
- Sensor Data captured via Sensor(s) of a user's augmented reality (“AR”) display apparatus;
- And/or:
- Processing of captured video or images by a user's augmented reality (“AR”) display apparatus inward facing camera(s), head and motion tracking module and eye tracking module, or via the game server or host using spatial mapping and facial recognition techniques in which a 3D model maybe reconstructed of user's eyes;
- Or alternatively, this data maybe captured directly by the game server or host using the external camera(s) and its facial recognition and spatial mapping capabilities using its own observer module and tracking modules;
- Transmitting via the network of captured values of a user's head position and orientation and eye tracking data in the form of state information and/or sensor data;
- Retrieving via the network a user's head position and orientation and eye tracking data
- Processing of a user's head position and orientation and eye tracking data and values;
- Applying adjustments to multiple adjustable rear camera(s) where the movable rear facing camera(s) may be panned, rotated and tilted via the usage of their motorised dual servo, steering gear and micro-controller invoked by the client module using the other users head and eye position/coordinates (x,y,z), gaze direction/directional vector and orientation (p,r,y) may be converted into different pan/tilt and rotational camera positions of differing orientation (p,r,y) in order to reconstruct the users view of behind scenes on its surface display panel/screenfaces at the same directional gaze, position (x,y,z) and orientation (p,r,y) of the user's eyes and head as well as maintain the correct perspective view of the behind scene and semi-invisibility appearance;
- Displaying and projecting of video and images captured via the rear mounted camera(s) on the surface display panel/screen faces of a real-world game object in which:
- Adjustments are made to the tilt, orientation, rotation and directional heading of each of the rear cameras to match the users directional gaze or their point of gaze together with their head position and orientation where the users sees recorded video and/or images of the real-world space behind the real-world game object on its surface display panel/screen faces in which the object appears invisible or semi-invisible to the user;
- Where there are multiple users are in front of the real-world game object, the client module may attempt to segregate different surface display panel/screen faces where each provides a different view of the real-world space behind according to the different user's head and eye positions and orientation and directional gaze;
- This method applies to real-world game objects including a user's IR Shield Apparatus and a real-world AI Character that feature rear facing adjustable camera(s).
A method of a real-world AI character targeting, firing an IR beam or IR Laser Beam at another target and physically moving around the real-world space of the game and laser tag arena comprising one or more of:
- Capturing of video and images of the real-world space via forward facing camera(s);
- Processing of captured video and images by an Observer Module;
- Generating of mesh data via the use of the real-world AI characters forward facing camera(s) recorded images or video and spatial mapping techniques, to identify real-world surfaces, objects and users. supported by an observer module;
- Identifying of surfaces, objects and users based on generated 3D mesh data, 3D models and virtual game objects of observed objects, users or surfaces in the real-world space;
- Providing of depth of view and depth cues through forward facing camera(s) and spatial mapping to the real-world AI Character to judge the relative distance, size and directional heading of an object or user which may be used in the targeting of objects or users during the laser tag game;
- Providing of detailed generated meshes and wireframes containing depth information that:
- contains detailed points or vertices of a real-world space, object or user allows the AI software to determine the exact 3D model shape of an object or user together with its mass, volume and the 3D positional coordinates (x,y,z) of its parts that may be used for more accurate targeting of an IR beam or IR Laser beam in the use of a real world game object by the real world AI character during a game and/or laser tag game;
- And that may be highlighted to the AI part of the AI Character through colouring of the wireframes by the Observer Module, in which distance is distinguishable by colour i.e. blue=two meters, yellow 3 meters etc.
- The targeting of other users or objects based on their position (x,y,z) and orientation (p,r,y) that maybe retrieved locally based on generated mesh data or via the Game Server or Host which maintains the state of all clients, via the network;
- Converting of Cartesian coordinates or vector positions (x,y,z) of targets into a set of joint angles that invoke the movement of the real-world AI Character's arm, hand, fingers and fingertips through a process of inverse kinematics;
- The use of manipulation to control the AI Character's operating of a real-world game object such as an IR Laser Gun Apparatus through inverse kinematics;
- inverse dynamics;
- trajectory planning (based on computation of the other target users position (x,y,z) and orientation (p,r,y) retrieved from the Game Server or Host or determinable locally based on local generated mesh data, where;
- The use of manipulators allow the AI Character's to grasp an object such as an IR Laser Gun Apparatus in which this may move for example the AI Character's hand, fingers and fingertips to a desired (x,y,z) position and orientation to invoke the pulling of an IR Laser Gun Apparatus trigger mechanism;
- The identification of surrounding surfaces, its depth, volumetric and geometric properties via generated mesh data, in which the real-world AI character can navigate;
- The use of a algorithms and manipulators to support the movements and motion of the rigid body or soft body of the AI Character in which its body may be moved to specific vector position (x,y,z) and orientation pitch, roll and yaw (p,r,y) through the use of one or more manipulators and multiple actuators, that support the movement of the real-world AI character around the game real-world three dimensional space and laser tag arena;
- The use of spatial mapping and the prescribed method herein to enable real world AI Character navigate around the real-world game space and the laser tag arena spaces judging distances so as to avoid collisions and also to facilitate the real world AI Character's ability to physically interact with objects and users.
A method of displaying, tracking and detecting hits from a real-world game object in the form of an IR Sword Apparatus that exists both in the real-world and virtual world of the game simultaneously in which top part is virtual and the bottom part is a physical handheld object comprising one or more of:
- a physical handheld bottom portion of the apparatus that comprises a number of processing units, memory, storage an IR LED that maybe a form of PIR detector, a series of sensor(s) including a multi-axis accelerometer, gyroscope, tilt sensor, motion sensors;
- a bluetooth/wifi module for connecting via the network to the users augmented-reality (“AR”) display apparatus;
- capturing and processing of user gesture inputs detectable via the devices sensor(s) and processed using the apparatus processing units;
- transmitting of captured state and sensor data via the network containing the objects position (x,y,z), orientation, rotation, directional vector and motions;
- receiving transmitted state and sensor data by a user's augmented reality (“AR”) display apparatus via the network from a user's IR Laser Sword Apparatus directly or via the game server or host;
- invoking the rendering and display of the IR Laser Sword Apparatus upper virtual structure and decal or diffuse lighting effects and generating 3D spatial audio effect or perceptual based 3D locational audio effect by a user's augmented reality (“AR”) display apparatus rendering module, audio manager and games engine in which;
- the top portion of the IR Laser Sword Apparatus is rendered and displayed via the users augmented reality (“AR”) display apparatus, micro-display in real-time as an augmented reality virtual image which is identically placed in precise synchronisation with the three dimensional cartesian coordinate/vector position (x,y,z), orientation (p,r,y), and directional vector of the bottom portion of the apparatus to form a full device where the top portion of the apparatus may be displayed in the form of holographic, hologram or two differing two 2D stereoscopic images over the users real-world view;
- A virtual-game object and 3D model is maintained of the real-world game object used in the rendering of the IR Laser Sword Apparatus upper virtual structure in which transformations are applied using matrix operations based on changes:
- to IR Sword Apparatus 3D cartesian coordinates/position (x,y,z), orientation, rotation, directional vector and motions resulting from a user's hand movements in the usage of the bottom portion, detected via the IR Laser Sword apparatus sensor(s) and retrieved via the network, where mathematical operations, matrix operations and transformations are applied to the virtual game object of the IR Laser Sword Apparatus in which the top virtual portion of the apparatus is displayed and maintained in exact synchronisation to the position, orientation and directional vector of the bottom portion of a real-world game object during movements;
- And/or;
- to a user's head and eyes position and orientation together with their focal point/directional gaze, detected via a user's augmented reality (“AR”) display apparatus sensor(s) to maintain correct visual perspective;
- the detection of complex motions and gestures by the IR Laser Sword Apparatus sensor(s), gesture recognition and device tracking module resulting from rapid and varying changing positions (x,y,z), orientation, rotation, directional vector and motions from a user's hand movements, resulting in the display of different virtual images of the IR Laser Sword upper virtual structure and decal or diffuse lighting effects together with 3D spatial audio effect or perceptual based 3D locational audio effect which may be displayed and played out via the users augmented reality (“AR”) display apparatus, micro-display and speakers respectively;
- detecting hits either:
- Computationally based on the relative position, orientation and projectile/directional vector of the IR Laser Sword Apparatus to another object or user using posteriori or priori collision techniques together with state and sensor data retrieved directly or via the internet;
- OR based on the detection of IR light emissions by another user or objects IR Sensor/Receiver in which a hit maybe registered via the network;
- Or based on the reflecting or the IR beam back on onto the devices own infrared IR Sensor/Detector where an object comes into obstruction with the IR Beam or IR Laser beam of the IR Laser Sword Apparatus, where the devices IR LED and IR Sensors/IR Receivers maybe set up as a form of IR Detector where hits are determinable by any portion of the IR Beam being broken essentially by another object that then causes the IR light source to reflect back and where based on the time the IR Light beam takes to come back the exact point of intersection and collision position (x,y,z) maybe determined used in the display of decal and/or diffuse lighting effects;
- Where the IR beam and IR LED is always on in which, hits are determinable by any portion of the IR Beam being broken by another object and the IR light source being reflected back;
- determining and invoking a collision response;
- Where:
- Rendered decal, diffuse lighting and specular lighting effects may be displayed via a user's augmented-reality (“AR”) display apparatus as an augmented virtual image at the determined point of collision upon a collision or hit;
- Haptic feedback maybe invoked via the IR Sword Apparatus and/or on the other user or object for which it intersected with where the other user is wearing a wearable real world game object such as a IR Mixed Reality suit or has handheld real-world game object that features a haptic module;
- The great velocity and acceleration of the user's motion of the IR Laser Sword Apparatus in their hands detectable via the apparatus sensor(s), the greater the rendered lighting effects that maybe shown via the users augmented reality (“AR”) display apparatus as well the greater physical feedback from the devices haptic module upon a collision being determined;
A method of pre-rendering game scenes over hours or days and providing full CGI like experience in real-time via a user's augmented reality(“AR”) display apparatus at higher level of detailed 3D models/triangles/polygons than possible in real-time locally by a user's augmented reality (“AR”) display apparatus comprising one or more of:
- Generating of volumetric, geometric mesh data, 3D models, wireframes, surface Data and CoPs of the games real-world space and/or that of the laser tag arena over time via a plurality of device including:
- the use of a user(s) augmented reality (“AR”) display apparatus observer module and camera(s) using spatial mapping;
- And/or the use by the game server or host of its external cameras with its own spatial mapping capabilities of its observer module;
- And/or the use of Laser interference or projection using incoherent light by the game server or host to generate CoP 3D models and mesh data through the use of one or more laser planner(s) and/or spatial light modulators that are used in conjunction with the digital external projectors, and one or more of the external camera(s) respectively;
- The interfacing of clients in the form of user(s) augmented reality (“AR”) display apparatus via the network with the game server or host and providing of locally generated mesh data and 3D models of the game real-world space and/or that of the laser tag arena;
- Maintaining of a global mesh data by the game server or host of the real-world game space and/or laser tag arena;
- Pre-rendering of game scenes and virtual game objects over hours or days to produce full CGI quality renderings in which:
- More advanced and highly detailed rendering techniques such as raytracing maybe used, which may be augmented on the users view of the real-world through the use of pre-generated volumetric and geometric mesh data, 3D models and wireframes of the real-world;
- more detailed meshes and 3D models with a far greater number of triangles and vertices may be used, for which the local client may simply retrieve the generated images or video based on the user's six dimensional matrixes representing their head position and head orientation in which this allows for a true augmented CGI experience to be provided in real-time during the game;
- Retrieving of pre-rendered game scenes and virtual game objects via the network from the game server or host by users clients in the form of an augmented reality (“AR”) display apparatus, in which selected pre-rendered game scenes and virtual game objects maybe based on an individual user's head and eye position and orientation in the real world space;
- Display of pre-rendered game scenes and virtual game objects by a user's augmented reality (“AR”) display apparatus in which may be provided in the form of 3D holographic or 2D stereoscopic images or video or other forms by the game server or host;
A method of remotely processing and/or pre-rendering and caching pre-rendered data for use by clients in the form of user (“AR”) display apparatus comprising one or more of:
- Capturing and transmitting recorded video or images via the network, from a client in the form of a user's augmented reality (“AR”) display apparatus to a game server or host;
- Remotely processing captured video and images of a user's real-world space by a game server or host;
- Generating mesh data, mesh filters, wireframes 3D models and virtual-game objects of a user's real-world space using spatial mapping and/or other techniques;
- Remotely rendering game scenes and/or virtual game objects, texture maps, surfaces and 3D models by the game server or host;
- Locally rendering game scenes and/or virtual game objects, texture maps, surfaces and 3D models by clients;
- Caching of pre-rendered game scenes and/or virtual game objects, texture maps, surfaces and 3D models by the game server or host or other clients in the form of user(s) augmented reality (“AR”) display apparatus;
- Transmitting via the network of:
- Pre-rendered data via the network, which maybe in the form of:
- Virtual game objects, game scenes, texture maps, surfaces and 3D models;
- virtual images which maybe a pair of two 2D stereoscopic images or a holographic or hologram form or maybe supplied as video;
- Where this data maybe generated and cached by the game server or host or other clients;
- Or produced compute instructions or mathematical outputs from the pre-rendering of game objects, surface or scenes in the form of data which may include but not limited to a list of game objects, vertices, faces, floats and game object IDs;
- Receiving of via the network by a user's augmented reality (“AR”) display apparatus;
- Remotely pre-rendered data from the game server or host or other clients in the form of user(s) augmented reality (“AR”) display apparatus;
- Or pre-generated compute instructions or mathematical outputs that may include a list of game objects, vertices, faces, floats and game object IDs among other data from the game server or host;
- Where the local rendering module of the users augmented reality (“AR”) display apparatus shall then render and apply transformations to the game scene and/or virtual game objects to change the scale and rotation accordingly, in which multi-dimensional matrixes may be used in this process;
- Display of game scenes and/or virtual game objects that maybe:
- rendered entirely remotely by the game server or host or another client where the pre-rendered and cached game scenes, surfaces game objects, texture maps and/or 3D models or virtual images or video may be displayed via the receiving user's augmented reality (“AR”) display apparatus;
- or rendered locally by the users augmented reality (“AR”) display apparatus rendering module using pre-generated compute instructions or metamathematical outputs provided the game server or host.
A method of firing a virtual weapon and invoking the display of decal/diffuse lighting on a real-world game object comprising one or more of:
- The generating of virtual weapon fire and virtual game objects representing the virtual weapon fire resulting from but not limited to:
- The use of a real world game objects trigger mechanisms or activation via other means including voice commands etc.
- Hand Gestures detected via sensor(s) or camera(s) using gesture recognition algorithms;
- The determining of a collision or hit based on the formulated projectile of the virtual weapon and its positions in three dimensional space relative to another object using Priori or Posteriori collision detection methods;
- The determining and invoking of a Collision response which:
- may result in a physical and visual change in the state of real-world game objects which maybe wearable, handheld, IR Mixed Reality scenery and/or real-world AI character's where;
- this may invoke in the case of handheld/wearable haptic feedback to a user:
- And/or:
- this may invoke the display of decal and diffuse lighting over a real-world game object, user, surface or AI character that may be displayed via:
- A user's augmented reality (AR) Head mounted display apparatus as a virtual augmented image over the user's real world view and FOV;
- And/or:
- External Projectors using 3D projection mapping, video mapping and/or spatial mapping techniques visible to the naked eye;
- And/or:
- a real-world game objects surface display panel faces visible to the naked eye.
A method of managing the spawning, rendering and display of virtual game objects, line renderings and rays used in the display of moving IR beam or IR Laser beams together with the management of game scenes, lighting, decal and other displayed effects between clients in the form of real-world game objects, users augmented reality (“AR”) display apparatus and that of the game server or host comprising one or more of;
- Recording video and images of a user's real-world space by:
- Clients in the form of augmented reality (“AR”) display apparatus and/or real-world game objects using their external facing camera(s);
- And/Or
- the game server or host using external camera(s) within the laser tag arena or users real-world space which may depth-aware cameras which are a form of structured light or time-of-flight cameras or maybe a form of stereo cameras which may be used in together with positioning reference such as infrared emitters or a lexian-stripe;
- Processing of recorded video and images of a user's real-world space and generating of mesh data comprising surface data, mesh filters, wireframes and/or 3D models of a user's real-world space, surrounding surfaces and any objects within a user's real-world space by clients in the form of a user's augmented reality (“AR”) display apparatus and/or real-world game objects or by a game server or host using spatial mapping techniques;
- Transmitting and receiving via network of generated mesh data, 3D models, state information and sensor data between clients directly and between clients and between clients and the game server or host;
- Capturing, transmitting and receiving of changes in state of a real-world game object via network, where upon a user activating a real-world game object and firing an IR beam or IR Laser beam detectable from the user of a trigger mechanism in the form of an I/O input or other forms the real-world game objects client shall generate and transmit a boolean flag via the network to other clients in the form of a user's augmented-reality display apparatus or other real world game objects and the game server or host used in the generating of virtual game objects, line renderings and/or rays together with lighting and shading effects by clients and the game server or host;
- Capturing, transmitting and receiving of sensor data from a real-world game object sensor(s) via the network, captured at the time of an IR beam or IR Laser beam being fired or projected, received by the game server or host or a user's augmented-reality display apparatus in which sensor data comprises:
- The real-world game objects vector position/world coordinates in the form of x,y,z cartesian values, rotation, orientation in the form or pitch, roll and yawl (p,r,y) values and directional heading/directional vector used in the generating of virtual game objects, line renderings and/or rays by clients or game server or host in the determining of the origin, projectile/trajectory, direction vector and positions of an IR beam or IR Laser beams virtual game objects, line renderings and/or rays as well in the determining of collisions;
- Delegating authority to clients in the form of augmented reality (“AR”) display apparatus to spawn, render and display locally virtual game objects, line renderings and/or rays via of a moving IR beam or IR Laser beam together with game scenes via their local games engines and rendering modules using their CPUs and/or GPU's where:
- Through the use of state information and sensor data of the real-world game object captured at the time of an IR beam or IR Laser beam being fired by a user pulling the trigger mechanism or activating the device by other means, together with generated surface data, mesh data, mesh filters, wireframes and/or 3D models of a user's real-world space, the clients rendering module, games engine and light display module is configurable to render and display game scenes and virtual game objects, line renderings or rays representing a moving IR beam or IR Laser Beam in the form augmented virtual images over the users real-world FOV via their micro display;
- Delegating authority to clients in the form of real-world game objects to generate virtual game objects and render and display texture-maps, shading, lighting and decal effects directly over their surface display panel/screen faces, visible to the naked eye;
- Generating and display of virtual-game objects and line rendering or rays over a real-world moving IR beam or IR Laser beam by local clients in the form of a user's augmented reality (“AR”) display apparatus over a user's real-world space and FOV using received state and sensor data of the real-world game object that fired the IR beam or IR Laser beam together with mesh data on a user's real-world space,
- Generating and display of virtual-game objects, line rendering or rays together with lighting and shading effects of moving IR beam or IR Laser beam over a user's real-world space as an augmented game scene by Game Server or host over a user's real-world space using external projectors and 3D mapping techniques together with mesh data, and retrieved state and sensor data;
- Transmitting of state information via the network on generated game objects between clients and game server or host.
Dependencies are inferred from the claim text.