# US 11,029,836 B2 · Cross-platform interactivity architecture > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,836 B2 - Document ID: US11029836B2 - Status: Granted - Kind: B2 - Application number: 15/224,389 - Filed: 2016-07-29 - Published: 2017-09-28 - Granted: 2021-06-08 - Earliest claimed: 2016-03-25 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC. - Inventors: Barry Christopher Allyn - Web page: https://patentgrep.com/patent/US11029836B2 ## CPC - G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041) - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 8/38: for implementing user interfaces - G06F 9/44: Arrangements for executing specific programs - G06F 9/451: Execution arrangements for user interfaces - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Maintaining an application for deployment across platforms requires significant engineering effort and an expanded code base, especially when dealing with different input types from the host platforms. To reduce the computational expenses and improve the standardization of the user interface, systems and methods are provided to recognize gestures in a platform agnostic code architecture, which may be deployed on multiple platforms with different input and output methodologies to provide a consistent user experience with a smaller code base to install on those platforms and to maintain. Individual platforms may tailor the size of the architecture to meet their deployment needs and the architecture may be updated independently of the client logic of various applications. ## Claims ### Claim 1 A method for handling user commands from multiple operating system platforms at an architecture via an extensible environment, comprising: recognizing a first gesture by: receiving, at the architecture, a first raw pointer input from a first device operating on a first operating system platform; determining, at the architecture, a first object displayed in a first user interface of the first device that is associated with the first raw pointer input based on a position of the first raw pointer input in the first user interface; identifying, at the architecture, a first set of gestures corresponding to the first operating system platform, the first set of gestures including a number and type of gestures based on expected pointer inputs capable of being performed on the first operating system platform; and recognizing, at the architecture, the first gesture from the first set of gestures based on the first raw pointer input, a pointer status, and locations of the first raw pointer input relative to the first object; recognizing, at the architecture, a second gesture based on a received second raw pointer input associated with a second object displayed in a second user interface of a second device operating on a second operating system platform, the second gesture recognized from a second set of gestures corresponding to the second operating system platform, the second set of gestures including a number and type of gestures identified based on expected pointer inputs capable of being performed on the second operating system platform, wherein the second gesture is a different type of gesture from the first gesture and the second operating system platform is a different operating system platform than the first operating system platform; normalizing, at the architecture, the first raw pointer input and the second raw pointer input when determining the first gesture and the second gesture signify execution of a same command; determining, at the architecture, the same command to be executed on the first device and the second device, the same command being determined based on a respective object and a respective gesture; and transmitting the same command for execution on the first device and the second device when the first raw pointer input received from the first device is different from the second raw pointer input received from the second device. ### Claim 2 (depends on 1) The method of claim 1, wherein the first object associated with the first raw pointer input includes multiple sub-objects. ### Claim 3 (depends on 1) The method of claim 1, wherein the position of the first raw pointer input in the first user interface does not match a position of the first object and the first object is determined to be associated with the first raw pointer input as an object closest to the position of the first raw pointer input in the first user interface. ### Claim 4 (depends on 1) The method of claim 1, wherein the first object is replaced in response to recognizing the first gesture signifies execution of a different command. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the same command is further based on knowing an operating system of the first device and the second device. ### Claim 6 (depends on 1) The method of claim 1, wherein transmitting the same command for execution on the first device and the second device comprises updating the first user interface and the second user interface to display the same command at a top z-order relative to the first object and the second object while maintaining a layout of the first user interface and the second user interface. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the same command is further based on a user interface mode. ### Claim 8 (depends on 1) The method of claim 1, wherein the first object is a canvas of the first user interface. ### Claim 9 (depends on 1) The method of claim 1, wherein the same command is composited with a second command in a command stack for transmission to the first device and the second device to execute the same command and the second command. ### Claim 10 (depends on 1) The method of claim 1, wherein the pointer status is chosen from: a mouse click; a mouse double-click; a continuous mouse depression; pen gestures; eye tracking gestures; hover gestures; touch pad gestures; a tap touch input; a double-tap touch input; a continuous touch input; and a multi-touch touch input. ### Claim 11 A system having an architecture for handling user commands from multiple operating system platforms via an extensible environment, the architecture comprising: a processing unit; and a system memory communicatively coupled to the processing unit, wherein the system memory includes instructions, that when executed by the processing unit, cause the processing unit to: recognize a first gesture by: receiving a first raw pointer input associated with a first object displayed in a first user interface from a first device operating on a first operating system platform, wherein the first raw pointer input indicates a pointer status and a location of the first raw pointer input relative to the first object in the first user interface; identifying a first set of gestures corresponding to the first operating system platform, the first set of gestures including a number and type of gestures based on expected pointer inputs capable of being performed on the first operating system platform; and recognizing the first gesture from the first set of gestures based on the first raw pointer input, the pointer status, and the location of the first raw pointer input relative to the first object; recognize a second gesture based on a received second raw pointer input associated with a second object displayed in a second user interface from a second device operating on a second operating system platform, the second gesture recognized from a second set of gestures corresponding to the second operating system platform, the second set of gestures including a number and type of gestures identified based on expected pointer inputs capable of being performed on the second operating system platform, wherein the second gesture is a different type of gesture from the first gesture and the second operating system platform is a different operating system platform than the first operating system platform; normalize the first raw pointer input and the second raw pointer input when determining the first gesture and the second gesture signify execution of a same command; determine the same command to be executed on the first device and the second device, the same command being determined based on a respective object and a respective gesture; and transmit the same command for execution on the first device and the second device when the first raw pointer input received from the first device is different from the second raw pointer input received from the second device. ### Claim 12 (depends on 11) The system of claim 11, wherein components of the architecture are tailored in size and composition to a client logic providing the first user interface and the second user interface at the first device and the second device. ### Claim 13 (depends on 11) The system of claim 11, further comprising: an object application program interface in communication with the system, wherein the object application program interface provides on-object user interface elements related to the same command that is executed. ### Claim 14 A method for handling user commands from multiple operating system platforms at an architecture via an extensible environment, comprising: recognizing a first gesture by: receiving, at the architecture, from a first device operating on a first operating system platform, a first raw pointer input associated with a first object displayed in a first user interface and hit test results, wherein the hit test results identify the first object displayed in the first user interface; identifying, at the architecture, a first set of gestures corresponding to the first operating system platform, the first set of gestures including a number and type of gestures based on expected pointer inputs capable of being performed on the first operating system platform; and recognizing, at the architecture, the first gesture from the first set of gestures based on the first raw pointer input and the hit test results; recognizing, at the architecture, a second gesture based on a received second raw pointer input associated with a second object displayed in a second user interface of a second device operating on a second operating system platform, the second gesture recognized from a second set of gestures corresponding to the second operating system platform, the second set of gestures including a number and type of gestures identified based on expected pointer inputs capable of being performed on the second operating system platform, wherein the second gesture is a different type of gesture from the first gesture and the second operating system platform is a different operating system platform than the first operating system platform; normalizing, at the architecture, the first raw pointer input and the second raw pointer input when determining the first gesture and the second gesture signify execution of a same command; determining, at the architecture, the same command to be executed on the first device and the second device, the same command being determined based on a respective gesture to perform on a respective object; and transmitting the same command to a command stack for execution on the first device and the second device when the first raw pointer input received from the first device is different from the second raw pointer input received from the second device. ### Claim 15 (depends on 14) The method of claim 14, wherein in cases that the hit test results identify multiple objects displayed in the first user interface, the same command is directed to a given object of the multiple objects based on the first gesture being associated with the given object. ### Claim 16 (depends on 14) The method of claim 14, wherein more than one command related to the respective object is transmitted to the command stack. ### Claim 17 (depends on 16) The method of claim 16, further comprising: receiving an undo command from one or more of the first device and the second device; and reverting one or more of the first user interface and the second user interface to a state prior to executing the same command. ### Claim 18 (depends on 14) The method of claim 14, wherein the same command executed at the first device and the second device overrides a command performed by the architecture of the extensible environment on the respective object.