# US 2019/0318002 A1 · Virtual assistant system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318002 A1 - Document ID: US20190318002A1 - Status: Published application - Kind: A1 - Application number: 16/454,005 - Filed: 2019-06-26 - Published: 2019-10-17 - Earliest claimed: 2014-01-30 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Vishal SHARMA; Elhum AMJADI - Web page: https://patentgrep.com/patent/US20190318002A1 ## CPC - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 40/30: Semantic analysis - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06Q 10/02: Reservations, e.g. for tickets, services or events - G06Q 10/40: Business processes related to social networking or social networking services - G06Q 30/0601: Electronic shopping [e-shopping] - G06Q 50/12: Hotels or restaurants - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - H04L 63/06: {for supporting key management in a packet data network (cryptographic mechanisms or cryptographic arrangements for key management H04L9/08)} - H04L 63/08: {for authentication of entities (cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32)} - H04L 63/101: {Access control lists [ACL]} - H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)} - H04L 67/63: Routing a service request depending on the request content or context - H04W 4/20: Services signaling; Auxiliary data signalling, i.e. transmitting data via a non-traffic channel - H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication - H04W 12/082: using revocation of authorisation - H04W 12/088: using filters or firewalls - G10L 2015/223: {Execution procedure of a spoken command} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A mobile device to operate a virtual assistant system to control any wirelessly connected devices and services by receiving an input command to perform a function at the devices and services, translate the input command into a common language sharable between the external devices and services so that such input commands can be understood between the plurality of devices and services, and functions can be performed at the devices and or service in response to the input command. This control can be shared with other mobile devices as instructed by an input command. ## Claims ### Claim 1 An apparatus comprising: a communicator configured to receive a user input command formatted according to a semantic information interchange language; a storage system mapping the received user input command to at least one predefined semantic command, the at least one predefined semantic command comprising a manual gesture; a first plugin configured to issue a first external command to a first external device or service based on the at least one predefined semantic command; and a second plugin configured to issue a second external command to a second external device or service based on the at least one predefined semantic command, the second external command being distinct from the first external command. ### Claim 2 (depends on 1) The apparatus of claim 1, the at least one predefined semantic command further comprising an icon touch or voice data. ### Claim 3 (depends on 1) The apparatus of claim 1, at least one of the first and second external commands comprising a control command for a security system, lighting, a thermostat, an automotive device, a wearable device, or an online service. ### Claim 4 (depends on 1) The apparatus of claim 1, the manual gesture being captured by a camera, motion detector, proximity detector, infrared sensor, temperature sensor, global positioning device, or wearable device. ### Claim 5 (depends on 1) The apparatus of claim 1, the at least one predefined semantic command comprising first and second semantic commands, the first plugin configured to issue primary and secondary first external commands to the first external device or service responsive to the first and second semantic commands, respectively. ### Claim 6 (depends on 5) The apparatus of claim 5, the first external device or service comprising an electronic lock system, the first semantic command comprising closing a fist, the second semantic command comprising closing two fists, the primary first external command comprising locking a door, the secondary first external command comprising engaging a deadbolt. ### Claim 7 (depends on 1) The apparatus of claim 1, the first external device or service comprising a face recognition program, the at least one predefined semantic command comprising a waving gesture, the first external command comprising identifying a person in a picture. ### Claim 8 (depends on 1) The apparatus of claim 1, further comprising an assistant interface configured to translate the received user input command into the at least one predefined semantic command, and a sensor comprising at least one of a camera, motion detector, proximity detector, infrared sensor, temperature sensor, global positioning device, or wearable device for receiving the manual gesture as the user input command. ### Claim 9 (depends on 1) The apparatus of claim 1, the communicator comprising a web communications system for transmitting and receiving information wirelessly over the World Wide Web. ### Claim 10 (depends on 1) The apparatus of claim 1, the first plugin further configured to issue distinct first external commands based on context information analyzed by the assistant server. ### Claim 11 (depends on 10) The apparatus of claim 10, said context information comprising at least one of a location of a mobile device at which the user input command was received, a time of day when the user input command was received, co-presence of specific individuals, involuntary actions by a user, and action sequences taken by a user of the mobile device. ### Claim 12 A method comprising: receiving a user input command formatted according to a semantic information interchange language; mapping the received user input command to at least one predefined semantic command, the at least one predefined semantic command comprising a manual gesture; issuing a first external command to a first external device or service based on the at least one predefined semantic command; and issuing a second external command to a second external device or service based on the at least one predefined semantic command, the second external command being distinct from the first external command. ### Claim 13 (depends on 12) The method of claim 12, the at least one predefined semantic command further comprising an icon touch or voice data. ### Claim 14 (depends on 12) The method of claim 12, at least one of the first and second external commands comprising a control command for a security system, lighting, a thermostat, an automotive device, a wearable device, or an online service. ### Claim 15 (depends on 12) The method of claim 12, the manual gesture being captured by a camera, motion detector, proximity detector, infrared sensor, temperature sensor, global positioning device, or wearable device. ### Claim 16 (depends on 12) The method of claim 12, the at least one predefined semantic command comprising first and second semantic commands, the issuing the first external command comprising issuing primary and secondary first external commands to the first external device or service responsive to the first and second semantic commands, respectively. ### Claim 17 (depends on 16) The method of claim 16, the first external device or service comprising an electronic lock system, the first semantic command comprising closing a fist, the second semantic command comprising closing two fists, the primary first external command comprising locking a door, the secondary first external command comprising engaging a deadbolt. ### Claim 18 (depends on 12) The method of claim 12, the first external device or service comprising a face recognition program, the at least one predefined semantic command comprising a waving gesture, the first external command comprising identifying a person in a picture. ### Claim 19 (depends on 12) The method of claim 12, the receiving comprising receiving the user input command wirelessly over the World Wide Web. ### Claim 20 An apparatus comprising: a processor; a memory storing instructions operable to cause the processor to: receive a user input command formatted according to a semantic information interchange language; map the received user input command to at least one predefined semantic command, the at least one predefined semantic command comprising a manual gesture; issue a first external command to a first external device or service based on the at least one predefined semantic command; and issue a second external command to a second external device or service based on the at least one predefined semantic command, the second external command being distinct from the first external command.