Using a three-dimensional environment model in gameplay

A method of operating an interactive computing system comprising:

• receiving a series of depth images from a mobile depth camera in motion around an environment;
• generating a dense 3D model of the environment from the series of depth images;
• integrating at least a part of the dense 3D model into an application, integrating at least the part of the dense 3D model into the application including:
• analyzing the dense 3D model to identify objects in the model;
• when an object is identified, inputting object type and object location information into the application;
• and
• adjusting operation of the application based on the input object type or location information;
• receiving a second series of depth images from the mobile depth camera when not in motion;
• and
• tracking real-time movement in 3D in a region of the environment using the second series of depth images, and
• wherein:
• the object comprises an active object and adjusting performance of the application includes adjusting an output of the active object;
• the object comprises a display and adjusting performance of the application comprises automatically calibrating a position of an object relative to the display based on a detected relative position of the display and the mobile depth camera;
• adjusting performance of the application comprises adjusting the output of another detected object;
• or
• the object comprises another depth camera and adjusting performance of the application comprises calibrating the system based on a relative position of each depth camera.

A gaming system having a first and second phase of operation, the gaming system comprising:

• an input for receiving a first series of depth images from a mobile depth camera in the first phase of operation and a second series of depth images from the mobile depth camera in the second phase of operation, wherein the mobile depth camera is in motion in the first phase of operation and is static in the second phase of operation;
• a real-time camera tracking engine arranged to track a position of the mobile depth camera in the first phase of operation;
• a dense 3D modeling system arranged to generate a 3D model of a room using at least the first series of depth images and the position of the mobile depth camera when each depth image was captured;
• and
• a gaming engine arranged to integrate at least a part of the dense 3D model into a game and to track real-time movement in a part of the room using the second series of depth images as a user input to the game, the gaming engine being further arranged to:
• analyze the dense 3D model to identify objects in the model;
• when an object is identified, input object type information into the game;
• and
• adjust operation of the game based on the input object type, and
• wherein:
• the object comprises an active object and the gaming engine is further arranged to adjust an output of the active object;
• the object comprises a display and the integration engine is further arranged to automatically calibrate a position of an object relative to the display based on a detected relative position of the display and the mobile depth camera;
• the integration engine is further arranged to adjust the output of another detected object;
• or
• the object comprises another depth camera and the integration engine is further arranged to calibrate the system based on a relative position of each depth camera.