Back to results
US 10,241,588 B1Granted
System for localizing devices in a room
AssigneePiccolo Labs Inc.
InventorsMarlon Misra, Neil Raina
Life of this patent
Estimating the term…
Abstract
A method and system may be used to provide a user interface allowing a user to identify electronic devices by placing bounding boxes on a video of scene. The bounding boxes may identify one or more electronic devices. Coordinates of the bounding boxes may be stored to allow determining when a user is gesturing towards one of the bounding boxes. The coordinates of the bounding boxes may be updated by a computer vision process that determines whether one or more electronic devices have been moved. When it is detected that a user is gesturing towards a bounding box, an electronic device associated with the bounding box may be controlled.
Claims13
A method for determining and updating a stored location of an electronic device in a gesture control system:
- capturing video with a video camera on a mobile device;
- displaying the video captured from the mobile device to a user in real-time on the mobile device;
- tracking the location of the user as the user moves to different locations in a room;
- building a three-dimensional representation of the room based on image frames of the video captured from the mobile device and one or more locations of the user;
- displaying user interface elements allowing the placing of a bounding box on the video captured from the mobile device;
- receiving input from the user on the mobile device indicating a bounding box, where the bounding box encloses an electronic device displayed in the video captured from the mobile device, wherein the electronic device is separate from the mobile device;
- capturing video with a video camera on a stationary hardware sensor device, the hardware sensor device being separate from the mobile device and communicably connected to the mobile device by a wireless connection;
- transmitting the video from the stationary hardware sensor device to the mobile device;
- cropping the video from the hardware sensor device based on the bounding box to create an image of the electronic device, wherein the cropping removes pixels outside of the bounding box and does not change the size of the image inside the bounding box;
- storing coordinates of the bounding box associated with the image of the electronic device;
- repeatedly comparing the image of the electronic device with a portion of the video captured from the hardware sensor device at the coordinates of the bounding box;
- when the difference between the portion of the video captured from the hardware sensor device at the coordinates of the bounding box and the image of the electronic device exceeds a threshold, comparing additional regions of the video captured from the hardware sensor device with the image of the electronic device to identify a new location of the electronic device;
- updating the coordinates of the bounding box to the new location;
- using the coordinates of the bounding box in a gesture control system;
- detecting that a user gesture is pointing to the coordinates of the bounding box;
- transmitting a signal to control the electronic device.
13 more elements
A method for determining and updating a stored location of an electronic device in a gesture control system:
- capturing video with a video camera on a stationary hardware sensor device;
- transmitting the video from the stationary hardware sensor device to a mobile device of a user, the hardware sensor device being separate from the mobile device and communicably connected to the mobile device by a wireless connection;
- displaying the video to the user in real-time on the mobile device;
- displaying user interface elements allowing the placing of a bounding box on the video;
- receiving input from the user on the mobile device indicating a bounding box, where the bounding box encloses an electronic device displayed in the video, wherein the electronic device is separate from the mobile device;
- cropping the video based on the bounding box to create an image of the electronic device;
- storing coordinates of the bounding box associated with the image of the electronic device, wherein the cropping removes pixels outside of the bounding box and does not change the size of the image inside the bounding box;
- repeatedly comparing the image of the electronic device with a portion of video at the coordinates of the bounding box;
- when the difference between the portion of video at the coordinates of the bounding box and the image of the electronic device exceeds a threshold, comparing additional regions of the video with the image of the electronic device to identify a new location of the electronic device;
- updating the coordinates of the bounding box to the new location;
- using the coordinates of the bounding box in a gesture control system;
- detecting that a user gesture is pointing to the coordinates of the bounding box;
- transmitting a signal to control the electronic device.
10 more elements
Dependencies are inferred from the claim text.