# US 8,542,907 B2 · Dynamic three-dimensional object mapping for user-defined control device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,542,907 B2 - Document ID: US8542907B2 - Status: Granted - Kind: B2 - Application number: 12/335,505 - Filed: 2008-12-15 - Published: 2009-06-18 - Granted: 2013-09-24 - Earliest claimed: 2007-12-17 - Assignees: Sony Computer Entertainment America LLC; Sony Computer Entertainment Inc. - Applicants: Gary Zalewski; Tomas Gillo - Inventors: Gary Zalewski; Tomas Gillo - Web page: https://patentgrep.com/patent/US8542907B2 ## CPC - A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/45: Controlling the progress of the video game - A63F 13/812: Ball games, e.g. soccer or baseball - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - A63F 2300/1093: using visible light - A63F 2300/6045: for mapping control signals received from the input arrangement into game commands - A63F 2300/8011: Ball - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/60: Analysis of geometric attributes - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A computer-implemented method is provided to interactively capture and utilize a three-dimensional object as a controlling device for a computer system. One operation of the method is capturing depth data of the three-dimensional object. In another operation, the depth data of the three-dimensional object undergoes processing to create geometric defining parameters for the three-dimensional object. The method can also include defining correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system. The method also includes an operation to save the geometric defining parameters of the three-dimensional object to a recognized object database. In another operation, the correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system in response to recognizing the particular actions are also saved to the recognized object database. ## Claims ### Claim 1 A computer implemented method to interactively capture and utilize a three-dimensional object to be a controlling device for a computer system, comprising: (a) capturing depth data of the three-dimensional object; (b) processing captured depth data of the three-dimensional object to create geometric defining parameters for the three-dimensional object; (c) defining correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system; and (d) saving to a recognized object database the geometric defining parameters of the three-dimensional object and correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system in response to recognizing the particular actions; (e) activating an interactive session processed by the computer system, the interactive session including; (i) capturing depth data and analyzing the depth data, the depth data being compared to the geometric defining parameters stored in the recognized object database; and (ii) rendering on a display, a predefined virtual object, that is selected for the geometric defining parameters found in the analyzed depth data, the predefined virtual object being rendered on the display in place of the three-dimensional object, the predefined virtual object being different than the three dimensional object. ### Claim 2 (depends on 1) The computer implemented method as recited in claim 1, further comprising: triggering activation of actions during the interactive session in response to actions performed with the three-dimensional object, as defined by the correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the predefined virtual object on the display. ### Claim 3 (depends on 2) The computer implemented method as in claim 2, wherein the actions performed with the three-dimensional object include moving the object, turning the object, waving the object, tapping the object, shaking the object, pointing with the objects or combinations thereof. ### Claim 4 (depends on 1) The computer implemented method as recited in claim 1, wherein depth data for the three-dimensional object is captured from a plurality of angles, the plurality of angles allowing the computer system to process the depth data and create the geometric parameters for the three-dimensional object. ### Claim 5 (depends on 1) The computer implemented method as recited in claim 1, wherein the correlations are defined by assigning a particular action performed with the three-dimensional object to a pre-defined particular action to be performed by the computer system. ### Claim 6 (depends on 1) The computer implemented method as recited in claim 1, wherein defining the correlations by having the user interactively move the three-dimensional object in response to a prompt for a pre-defined particular action. ### Claim 7 (depends on 1) The computer implemented method as recited in claim 1, creating macro correlations for a remote control so once a new three-dimensional object is defined, basic movements have the same effect. ### Claim 8 (depends on 1) The computer implemented method as recited in claim 1, saving multiple three-dimensional objects in the recognized object database and switching between them in real-time, each object having customized controls and a corresponding virtual object that is to be displayed on the display in place of the captured depth data of the three-dimensional object. ### Claim 9 A computer implemented method to interactively capture and utilize a three-dimensional object to be a controlling device for a computer system, comprising: (a) identifying the three-dimensional object, the identifying including, (i) capturing depth data of the three-dimensional object; (ii) processing captured depth data of the three-dimensional object to create geometric defining parameters for the three-dimensional object; (iii) defining correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system; and (iv) saving to a recognized object database the geometric defining parameters of the three-dimensional object and correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system; and (b) presenting the three-dimensional object to a camera; (c) displaying a virtual object on a display in place of the three-dimensional object, the virtual object being different than the three dimensional object; and (d) moving the presented three-dimensional object in front of the camera so as to trigger one or more of the particular actions to be performed by the computer system and displayed as actions by the virtual object on the display. ### Claim 10 (depends on 9) The computer implemented method as in claim 9, further comprising: (e) activating an interactive session processed by the computer system, the interactive session includes processing through (b) and (c) at particular times during the interactive session. ### Claim 11 (depends on 10) The computer implemented method as in claim 10, wherein during the interactive session of (e), the method includes, (i) capturing depth data using the camera and analyzing the depth data, the depth data being compared to the geometric defining parameters stored in the recognized object database; and (ii) triggering activation of actions during the interactive session in response to actions performed with the three-dimensional object, as defined by the correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system. ### Claim 12 (depends on 9) The computer implemented method as recited in claim 9, wherein depth data for the three-dimensional object is captured from a plurality of angles by the camera that is a single camera, the plurality of angles allowing the computer system to process the depth data and create the geometric parameters for the three-dimensional object. ### Claim 13 (depends on 9) The computer implemented method as recited in claim 9, wherein the correlations are defined by assigning a particular action performed with the three-dimensional object to a pre-defined particular action to be performed by the computer system. ### Claim 14 (depends on 9) The computer implemented method as recited in claim 9, wherein defining the correlations by having the user interactively move the three-dimensional object in response to a prompt for a pre-defined particular action. ### Claim 15 (depends on 9) The computer implemented method as recited in claim 9, creating macro correlations for a remote control so once a new three-dimensional object is defined, basic movements have a same effect. ### Claim 16 (depends on 9) The computer implemented method as recited in claim 9, saving multiple three-dimensional objects in the recognized object database and switching between them in real-time, each object having customized controls and a corresponding virtual object that is to be displayed on the display in place of the captured depth data of the three-dimensional object. ### Claim 17 (depends on 9) The computer implemented method as in claim 9, wherein the actions performed with the three-dimensional object include moving the object, turning the object, waving the object, tapping the object, shaking the object, pointing with the objects or combinations thereof.