# US 10,369,462 B2 · Client side processing of game controller input > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,369,462 B2 - Document ID: US10369462B2 - Status: Granted - Kind: B2 - Application number: 15/283,891 - Filed: 2016-10-03 - Published: 2017-01-26 - Granted: 2019-08-06 - Earliest claimed: 2012-12-21 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Krassimir Emilov Karamfilov; Emad Barsoum; Charles Claudius Marais; John Raymond Justice; David James Quinn; Roderick Michael Toll - Web page: https://patentgrep.com/patent/US10369462B2 ## CPC - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/20: Input arrangements for video game devices - A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone - A63F 13/30: Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers - A63F 13/35: Details of game servers - A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client - A63F 13/424: involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition - A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes - A63F 13/79: involving player-related data, e.g. identities, accounts, preferences or play histories - A63F 2300/1081: Input via voice recognition - A63F 2300/1087: comprising photodetecting means, e.g. a camera - A63F 2300/538: for performing operations on behalf of the game client, e.g. rendering - A63F 2300/6045: for mapping control signals received from the input arrangement into game commands - A63F 2300/6072: of an input signal, e.g. pitch and rhythm extraction, voice recognition - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR ## Abstract Embodiments of the present invention enable rich control input data to control video games that are remotely executed. Rich control input includes three-dimensional image data, color video, audio, device orientation data, and touch input. A remotely-executed video game is one executed on a server or other computing device that is networked to a client device receiving the rich control input. Rich control input includes more data than can be uploaded to a game server without degrading game performance. Embodiments of the present invention preprocess the rich control data on the client and into data that may be uploaded to the game server. The rich input stream may be processed in a general way or in a game-specific way. ## Claims ### Claim 1 A method of enabling game play over a remote connection using a data input device, the method comprising: receiving, at a client device from a camera, initial controller input for a video game running on a game server, the initial controller input comprising three-dimensional image data; processing the initial controller input on the client device to generate a reduced controller input, wherein the reduced controller input is an environmental delta for objects in an environment described by the three-dimensional image data within the initial controller input, the environmental delta describing changes to an inanimate object in the environment, wherein the initial control input comprises more data than the reduced controller input; communicating over a computer network the reduced controller input to the game server; and receiving, from the game server, video game data that reflects game progress manipulated according to the reduced controller input. ### Claim 2 (depends on 1) The method of claim 1, wherein the processing is performed with code specific to the video game being played. ### Claim 3 (depends on 1) The method of claim 1, wherein the method further comprises not communicating the entire initial controller input to the server. ### Claim 4 (depends on 1) The method of claim 1, wherein the reduced controller input further comprises skeletal geometry for a person described by three-dimensional image data within the initial controller input. ### Claim 5 (depends on 1) The method of claim 1, wherein the reduced controller input further comprises a game command derived by recognizing a gesture described by three-dimensional image data within the initial controller input. ### Claim 6 (depends on 1) The method of claim 1, wherein the initial controller input further comprises an audio signal and the method further comprises performing speech to text translation on the audio signal and communicating the text to the game server. ### Claim 7 (depends on 1) The method of claim 1, wherein the reduced controller input further comprises a motion blob for an object described by three-dimensional image data within the initial controller input, the motion blob describing one or more of an outline or carve out of moving objects in the environment. ### Claim 8 (depends on 1) The method of claim 1, wherein the reduced controller input further comprises a subset of data describing only a portion of a scene that is described by the three-dimensional image data within the initial controller input, the subset describing an object of interest to be analyzed by the video game running on the game server. ### Claim 9 (depends on 1) The method of claim 1, wherein the initial controller input is a point cloud. ### Claim 10 A computing system comprising: a processor; and computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to: receive from a camera initial controller input for a video game running on a game server, the initial controller input comprising a three-dimensional image data having pixels associated with a physical location in three-dimensional space viewed by the camera; process the initial controller input using code specific to the video game to generate a game-specific reduced controller input, wherein the initial control input comprises more data than the game-specific reduced controller input; communicate over a computer network the game-specific reduced controller input to the game server; process the initial controller input using system code to generate a generic reduced controller input; communicate over the computer network the generic reduced controller input to the game server; and receive over the computer network from the game server, video game data that reflects game progress manipulated according to the game-specific reduced controller input and the generic reduced controller input. ### Claim 11 (depends on 10) The computing system of claim 10, wherein the reduced controller input is a game command derived by recognizing a gesture described by three-dimensional image data within the initial controller input. ### Claim 12 (depends on 11) The computing system of claim 11, wherein the game-specific reduced controller input is a subset of the three-dimensional image data describing only a portion of a scene that is described by the three-dimensional image data, the subset describing an object of interest to be analyzed by the video game running on the game server. ### Claim 13 (depends on 12) The computing system of claim 12, wherein the subset is identified on the client using a skeletal geometry for a person described by the three-dimensional image data. ### Claim 14 (depends on 10) The computing system of claim 10, wherein the initial controller input further comprises an audio signal and the method further comprises performing speech to text translation on the audio signal and communicating the text to the game server. ### Claim 15 (depends on 10) The computing system of claim 10, wherein the generic reduced controller input is a player identification for a person described by the three-dimensional image data. ### Claim 16 A computing system comprising: a processor; and computer storage memory having computer-executable instructions stored thereon which, when executed by the processor, configure the computing system to: receive at a client device from a camera, initial controller input for a video game running on a game server that renders at least part of the video game video image, the initial controller input comprising a three-dimensional image data having pixels associated with a physical location in three-dimensional space viewed by the camera; process the initial controller input on the client device using code specific to the video game to generate a game-specific reduced controller input, wherein the initial control input comprises more data than the game-specific reduced controller input, wherein the reduced controller input is a game command derived by recognizing a gesture described by three-dimensional image data within the initial controller input; communicate over a computer network the game-specific reduced controller input to the game server located in a data center; and receive over the computer network, from the game server, video game data that reflects game progress manipulated according to the game-specific reduced controller input. ### Claim 17 (depends on 16) The computing system of claim 16, further configured to: process the initial controller input on the client device using system code to generate a generic reduced controller input, wherein the initial control input comprises more data than the generic reduced controller input; communicate over the computer network the generic reduced controller input to the game server; and wherein the video game data reflects game progress manipulated according to the game-specific reduced controller input and the generic reduced controller input. ### Claim 18 (depends on 17) The computing system of claim 17, wherein the generic reduced controller input is a skeletal geometry for a person described by the three-dimensional image data and wherein the game-specific reduced controller input is a subset of the three-dimensional image data describing only a portion of a scene that is described by the three-dimensional image data, the subset describing an object of interest to be analyzed by the video game running on the game server. ### Claim 19 (depends on 18) The computing system of claim 18, wherein the subset is identified on the client using the skeletal geometry for a person described by the three-dimensional image data. ### Claim 20 (depends on 19) The computing system of claim 19, wherein the initial controller input further comprises an audio signal and the method further comprises performing speech to text translation on the audio signal and communicating the text to the game server.