# US 11,907,423 B2 · Systems and methods for contextualized interactions with an environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,423 B2 - Document ID: US11907423B2 - Status: Granted - Kind: B2 - Application number: 17/094,712 - Filed: 2020-11-10 - Published: 2021-05-27 - Granted: 2024-02-20 - Earliest claimed: 2019-11-25 - Assignees: Meta Platforms Technologies, LLC - Applicants: Meta Platforms Technologies, LLC - Inventors: Faizan Muhammad; Daniel Wetmore - Web page: https://patentgrep.com/patent/US11907423B2 ## CPC - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 7/33: using feature-based methods - G06T 19/003: {Navigation within 3D models or images} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 10/945: {User interactive design; Environments; Toolboxes} - G06V 20/20: in augmented reality scenes - G06V 20/64: Three-dimensional [3D] objects - G06T 2207/10016: Video; Image sequence - H04Q 9/00: Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04Q: SELECTING (switches, relays, selectors H01H; wireless communication networks H04W) ## Abstract Computerized systems, methods, apparatuses, and computer-readable storage media are provided for generating a 3D map of an environment and/or for utilizing the 3D map to enable a user to control smart devices in the environment and/or to interact with a person in the environment. To generate the 3D map, perform the control, and/or interact with the person, a plurality of neuromuscular sensors may be worn by the user. The sensors may be arranged on a carrier worn by the user, and may be configured to sense neuromuscular signals from the user. A camera configured to capture information about the environment may be arranged on the carrier worn by the user. The sensors and the camera provide data to a computer processor coupled to a memory. ## Claims ### Claim 1 A computerized system for remote control of devices, the system comprising: a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device structured to be worn by the user to obtain the neuromuscular signals; at least one camera able to capture information about an environment; and at least one computer processor programmed to: generate a map of the environment based on map information about the environment captured by the at least one camera, the map information comprising information for controlling at least one controllable object in the environment, wherein the map information comprises map data representing a physical relationship between two or more controllable objects in the environment, and wherein the map includes a control interface that is specific to the environment, such that different maps for different environments include different control interfaces, and in response to a neuromuscular activity recognized from the neuromuscular signals sensed by the plurality of neuromuscular sensors: control the at least one controllable object using the environment-specific control interface to change from a first state to a second state according to the physical relationship; and detect a specific gesture from the neuromuscular activity, wherein the specific gesture designates a reference object within the environment, and wherein the generated map is identifiable from the designated reference object. ### Claim 2 (depends on 1) The system of claim 1, wherein: the map information of the environment is stored in a memory, and the at least one computer processor retrieves the map information from the memory based on information recognized from the information about the environment captured by the at least one camera. ### Claim 3 (depends on 2) The system of claim 2, wherein the recognized information is visible in the environment and comprises at least one of: a QR code, a graphical symbol, a string of alphanumeric text, a 3D object having a specific shape, or a physical relationship between at least two objects. ### Claim 4 (depends on 3) The system of claim 3, wherein the recognized information comprises a reference object for the environment. ### Claim 5 (depends on 1) The system of claim 1, wherein the map data is 3D panoramic data of objects in the environment, the objects comprising the at least one controllable object. ### Claim 6 (depends on 5) The system of claim 5, wherein the 3D panoramic data comprises a 360° representation of the environment. ### Claim 7 (depends on 5) The system of claim 5, wherein the 3D panoramic data comprises data relative to a single rotational axis. ### Claim 8 (depends on 5) The system of claim 5, wherein the 3D panoramic data comprises a representation of a partial view of the environment. ### Claim 9 (depends on 1) The system of claim 1, wherein the map information comprises a plurality of established center points determined along a specified rotational axis of the camera, and wherein the physical relationship between two or more controllable objects in the environment is determined from the established center points. ### Claim 10 (depends on 1) The system of claim 1, wherein the environment is an extended reality (XR) environment comprising virtual objects and real-world objects, wherein the at least one computer processor is programmed to: determine, from the map information, location information of the virtual objects and location information of the real-world objects, and determine, from the information about the environment captured by the at least one camera, a reference object in the environment, and wherein the map information comprises location information of the at least one controllable object relative to the reference object. ### Claim 11 (depends on 10) The system of claim 10, wherein the neuromuscular activity recognized from the neuromuscular signals sensed by the plurality of neuromuscular sensors results from the user activating a specific motor-unit relative to the at least controllable object while the user is in the environment. ### Claim 12 (depends on 10) The system of claim 10, wherein the neuromuscular activity recognized from the neuromuscular signals sensed by the plurality of neuromuscular sensors results from the user performing at least one gesture relative to the at least controllable object while the user is in the environment. ### Claim 13 (depends on 12) The system of claim 12, wherein the at least one gesture comprises any one or any combination of: the user moving at least one finger relative to the at least one controllable object, the user moving a wrist relative to the at least one controllable object, the user moving an arm relative to the at least one controllable object, the user applying a force without a movement relative to the at least one controllable object, and the user activating a motor unit without a movement relative to the at least one controllable object and without a force relative to the at least one controllable object. ### Claim 14 (depends on 13) The system of claim 13, wherein the at least one gesture comprises at least one of the following: the user using two or more fingers to perform a pinching motion relative to the at least one controllable object; the user tilting the wrist upward or downward relative to the at least one controllable object; the user moving the arm upward or downward relative to the at least one controllable object; or the user moving the at least one finger upward or downward relative to the at least one controllable object. ### Claim 15 (depends on 12) The system of claim 12, wherein: the at least one controllable object comprises a plurality of controllable objects, the at least one gesture comprises a gesture relative to one of the plurality of controllable objects, and the at least one computer processor is programmed to control each of the plurality of controllable objects to change from a first state to a second state. ### Claim 16 A wearable electronic device comprising: a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device structured to be worn by the user to obtain the neuromuscular signals; at least one camera able to capture information about an environment; and at least one computer processor programmed to: generate a map of the environment based on map information about the environment captured by the at least one camera, the map information comprising information for controlling at least one controllable object in the environment, wherein the map information comprises map data representing a physical relationship between two or more controllable objects in the environment, and wherein the map includes a control interface that is specific to the environment, such that different maps for different environments include different control interfaces, and in response to a neuromuscular activity recognized from the neuromuscular signals sensed by the plurality of neuromuscular sensors: control the at least one controllable object using the environment-specific control interface to change from a first state to a second state according to the physical relationship; and detect a specific gesture from the neuromuscular activity, wherein the specific gesture designates a reference object within the environment, and wherein the generated map is identifiable from the designated reference object. ### Claim 17 (depends on 16) The wearable electronic device of claim 16, wherein the environment is at least one of: a room in a home, a room in a business, a story of a multistory building, or an outdoor region. ### Claim 18 (depends on 16) The wearable electronic device of claim 16, wherein the at least one controllable object comprises at least one of: a display device, an electronic game, a window shade, a lamp, a sound system, a lock, or a food or beverage preparation device. ### Claim 19 A computer-implemented method comprising: activating a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device structured to be worn by the user to obtain the neuromuscular signals; activating at least one camera able to capture information about an environment; generating a map of the environment based on map information about the environment captured by the at least one camera, the map information comprising information for controlling at least one controllable object in the environment, wherein the map information comprises map data representing a physical relationship between two or more controllable objects in the environment, and wherein the map includes a control interface that is specific to the environment, such that different maps for different environments include different control interfaces; and in response to a neuromuscular activity recognized from the neuromuscular signals sensed by the plurality of neuromuscular sensors: controlling the at least one controllable object using the environment-specific control interface to change from a first state to a second state according to the physical relationship; and detecting a specific gesture from the neuromuscular activity, wherein the specific gesture designates a reference object within the environment, and wherein the generated map is identifiable from the designated reference object. ### Claim 20 (depends on 19) The computer-implemented method of claim 19, further comprising: storing the map information of the environment in a memory, and retrieving the map information from the memory based on information recognized from the information about the environment captured by the at least one camera.