# US 2023/0069042 A1 · Camera interfaces to interact with iot devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0069042 A1
- Document ID: US20230069042A1
- Status: Published application
- Kind: A1
- Application number: 17/460,228
- Filed: 2021-08-29
- Published: 2023-03-02
- Earliest claimed: 2021-08-29
- Assignees: Snap Inc.
- Inventors: Yu Jiang Tham; Rajan Vaish; Andrés Monroy-Hernández; Sven Kratz; Ana Maria Cardenas Gasca
- Web page: https://patentgrep.com/patent/US20230069042A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06V 20/20: in augmented reality scenes
- H04L 67/125: involving control of end-device applications over a network
- H04L 67/561: Adding application-functional data or data for application control, e.g. adding metadata
- H04N 23/63: by using electronic viewfinders
- H04W 4/70: Services for machine-to-machine communication [M2M] or machine type communication [MTC]
- H04W 8/005: {Discovery of network devices, e.g. terminals}
- G16Y 40/35: Management of things, i.e. controlling in accordance with a policy or in order to achieve specified objectives
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04W 84/18: Self-organising networks, e.g. ad-hoc networks or sensor networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G16Y: INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]

## Abstract

Systems and methods for interfacing augmented reality (AR) camera devices to Internet of Things (IoT) devices include pairing an AR camera device to an IoT device to establish which actions recognized by the AR camera device may control the IoT device, receiving an action identifier representing a gesture or position of a body part that has been recognized by the AR camera device, and sending a command to the IoT device paired with the AR camera device to perform a requested action in response to the action identifier received from the AR camera device. Context information relating to the action identifier from the AR camera may be used to modify the command sent to the IoT device. The AR camera device may be paired with an IoT device pointed at by the AR camera device, selected through a user interface of the AR camera device, or determined from context information.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method of controlling Internet of Things (IoT) devices using augmented reality (AR) objects, comprising:
associating an AR object with a real-world position determined from global positioning system (GPS) coordinates;
forwarding the AR object and the GPS coordinates to an AR camera device;
providing an overlay image of the AR object on the AR camera device; and
when a user of the AR camera device arrives at the GPS coordinates and interacts with the AR object on the AR camera device, providing a control signal corresponding to the interaction to at least one IoT device to control an operation of the at least one IoT device.

### Claim 22 (depends on 21)

The method of claim 21, wherein providing the overlay image of the AR object on the AR camera device comprises presenting the overlay image of the AR object on the AR camera device when the AR camera device arrives at the real-world position.

### Claim 23 (depends on 22)

The method of claim 22, further comprising receiving an indication that a user of the AR camera device has completed a specified action on the overlay image of the AR object on the AR camera device before controlling the operation of the IoT device in response to the control signal.

### Claim 24 (depends on 23)

The method of claim 23, wherein the specified action is a game play action in a virtual environment and the control signal is generated when the game play action has been successfully completed.

### Claim 25 (depends on 21)

The method of claim 21, further comprising enabling sharing permissions between respective AR camera devices for controlling the at least one IoT device, the shared permissions enabling a first user of a first AR camera device to enable access to an IoT device of the first user by a second user of a second AR camera device for a period of time.

### Claim 26 (depends on 25)

The method of claim 25, further comprising passing the permissions and trigger data to the second AR camera device of the second user and sending the control signal to the IoT device of the first user when the second user initiates a trigger designated by the trigger data during the period of time.

### Claim 27 (depends on 21)

The method of claim 21, wherein providing the overlay image of the AR object on the AR camera device comprises pairing the AR camera device to at least one IoT device to establish which actions recognized by the AR camera device may control which IoT device, receiving a sensed data signal from the at least one IoT device, presenting an overlay image corresponding to the sensed data signal as the overlay image of the AR object on the AR camera device, receiving response to the overlay image, the response including an action identifier from the AR camera device, and sending a command to the at least one IoT device to perform an action corresponding to the action identifier received from the AR camera device.

### Claim 28 (depends on 21)

The method of claim 21, further comprising pairing at least one lens with the at least one IoT device and providing the control signal to the at least one IoT device when a user interacts with the at least one lens paired with the at least one IoT device.

### Claim 29 (depends on 28)

The method of claim 28, further comprising sending a selection signal to the AR camera device for selection of the at least one lens when a sensor signal is received from the at least one IoT device paired with the at least one lens and presenting an overlay of the at least one lens on the AR camera device.

### Claim 30

A system that controls Internet of Things (IoT) devices using augmented reality (AR) objects, comprising:
an input translator that associates an AR object with a real-world position determined from global positioning system (GPS) coordinates and forwards the AR object and the GPS coordinates to an AR camera device that provides an overlay image of the AR object on the AR camera device; and
a command generator that provides, when a user of the AR camera device arrives at the GPS coordinates and interacts with the AR object on the AR camera device, a control signal corresponding to the interaction to at least one IoT device to control an operation of the at least one IoT device.

### Claim 31 (depends on 30)

The system of claim 30, wherein the overlay image of the AR object is presented on the AR camera device when the AR camera device arrives at the real-world position.

### Claim 32 (depends on 31)

The system of claim 31, wherein the command generator receives an indication that a user of the AR camera device has completed a specified action on the overlay image of the AR object on the AR camera device before sending the control signal to control the operation of the IoT device.

### Claim 33 (depends on 32)

The system of claim 32, wherein the specified action is a game play action in a virtual environment and the control signal is generated by the command generator when the game play action has been successfully completed.

### Claim 34 (depends on 30)

The system of claim 30, wherein the input translator enables sharing permissions between respective AR camera devices for controlling the at least one IoT device, the shared permissions enabling a first user of a first AR camera device to enable access to an IoT device of the first user by a second user of a second AR camera device for a period of time.

### Claim 35 (depends on 34)

The system of claim 34, wherein the input translator further passes the permissions and trigger data to the second AR camera device of the second user and the command generator sends the control signal to the IoT device of the first user when the second user initiates a trigger designated by the trigger data during the period of time.

### Claim 36 (depends on 30)

The system of claim 30, wherein the input translator receives a sensed data signal from the at least one IoT device, instructs the AR camera device to present an overlay image corresponding to the sensed data signal as the overlay image of the AR object on the AR camera device, receives a response to the overlay image from the AR camera device, the response including an action identifier from the AR camera device, and the command generator sends a command to the at least one IoT device to perform an action corresponding to the action identifier received from the AR camera device.

### Claim 37 (depends on 30)

The system of claim 30, further comprising pairing means for pairing at least one lens with the at least one IoT device, wherein the command generator provides the control signal to the at least one IoT device when a user interacts with the at least one lens paired with the at least one IoT device.

### Claim 38 (depends on 37)

The system of claim 37, wherein the input translator sends a selection signal to the AR camera device for selection of the at least one lens when a sensor signal is received from the at least one IoT device paired with the at least one lens and instructs the AR camera device to present an overlay of the at least one lens on the AR camera device.

### Claim 39

A non-transitory computer readable medium having stored thereon instructions, wherein execution of the instructions by one or more processors cause the one or more processors to controls Internet of Things (IoT) devices using augmented reality (AR) objects by performing operations including:
associating an AR object with a real-world position determined from global positioning system (GPS) coordinates;
forwarding the AR object and the GPS coordinates to an AR camera device;
providing an overlay image of the AR object on the AR camera device; and
when a user of the AR camera device arrives at the GPS coordinates and interacts with the AR object on the AR camera device, providing a control signal corresponding to the interaction to at least one IoT device to control an operation of the at least one IoT device.

### Claim 40 (depends on 39)

The non-transitory computer readable medium of claim 39, further comprising instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising receiving an indication that a user of the AR camera device has completed a specified action on the overlay image of the AR object on the AR camera device before sending the control signal to control the operation of the IoT device.
