# US 11,445,107 B2 · Supervised setup for control device with imager > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,107 B2 - Document ID: US11445107B2 - Status: Granted - Kind: B2 - Application number: 16/987,776 - Filed: 2020-08-07 - Published: 2021-02-11 - Granted: 2022-09-13 - Earliest claimed: 2019-08-08 - Assignees: Qorvo US, Inc. - Applicants: 7hugs Labs SAS - Inventors: Julien Colafrancesco; Simon Tchedikian; Simon Guillot; Louis Xiao Wang - Web page: https://patentgrep.com/patent/US11445107B2 ## CPC - G08C 23/04: using light waves, e.g. infrared - H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image} - G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041) - G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser - G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)} - G06V 20/20: in augmented reality scenes - H04N 23/635: {Region indicators; Field of view indicators} - G08C 2201/20: Binding and programming of remote control devices - G08C 2201/32: Remote control based on movements, attitude of remote control device - G08C 2201/70: Device selection - G08C: TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS (fluid pressure transmitting systems F15B; mechanical means for transferring the output of a sensing member into a different variable G01D5/00; mechanical control systems G05G) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods associated with determining a pointing target of pointing device are disclosed. One embodiment disclosed herein is a method conducted with a control device having a pointing direction and an onboard imager. The field of view of the onboard imager includes the pointing direction. The method includes capturing a reference image, determining a characteristic quantity based on the reference image, generating a feedback message based on the characteristic quantity, and predicating a setup phase termination for the control device on the characteristic quantity. ## Claims ### Claim 1 A system comprising: a control device, wherein the control device has a shape which defines a pointing direction of the control device; an onboard imager located on the control device, wherein the onboard imager has a field of view which includes the pointing direction; and one or more computer readable media storing instructions to: receive a reference image indicative of a zone of interest in a physical location; determine a characteristic quantity based on the reference image, the characteristic quantity being representative of distinctiveness of a signature of the zone of interest; generate a feedback message based on the characteristic quantity; and predicate a setup phase termination for the control device on the characteristic quantity; wherein the system further comprises one of the following features (i) to (iii): (i) the feedback message regards at least one additional reference image to be captured to improve the characteristic quantity, and the feedback message includes instructions on how to capture the at least one additional reference image; (ii) the determining of the characteristic quantity includes determining that the reference image is of insufficient quality, and the feedback message includes instructions to recapture the reference image; or (iii) the one or more computer readable media further storing instructions to disable a capture interface on the control device based on the characteristic quantity, wherein the determining of the characteristic quantity based on the reference image is conducted in real time. ### Claim 2 (depends on 1) The system of claim 1, wherein the one or more computer readable media further store instructions to: receive at least one additional reference image; redetermine the characteristic quantity based on the reference image and the at least one additional reference image; and triggering, after the characteristic quantity has been redetermined, an end to the setup phase for the system based on the characteristic quantity. ### Claim 3 (depends on 1) The system of claim 1, wherein the one or more computer readable media further store instructions to: alter an environment in which the control device is located based on the feedback message; wherein altering the environment comprises changing one of: a luminosity of the environment, a position of a mobile robot in the environment; and an image displayed on a screen in the environment. ### Claim 4 (depends on 1) The system of claim 1, wherein: the feedback message regards at least one additional reference image to be captured to improve the characteristic quantity, and the feedback message includes instructions on how to capture the at least one additional reference image. ### Claim 5 A system comprising: a control device, wherein the control device has a shape which defines a pointing direction of the control device; an onboard imager located on the control device, wherein the onboard imager has a field of view which includes the pointing direction; a companion device having a companion device imager and a display; and one or more computer readable media storing instructions to: receive a reference image captured by the companion device imager; determine a characteristic quantity based on the reference image; generate a feedback message based on the characteristic quantity; predicate a setup phase termination for the control device on the characteristic quantity; and display the reference image on the display. ### Claim 6 (depends on 5) The system of claim 5, wherein: at least one additional reference image is captured by the companion device imager; and the instructions specify at least one position for the companion device to be in when capturing the at least one additional reference image. ### Claim 7 (depends on 1) The system of claim 1, wherein: the determining of the characteristic quantity includes determining that the reference image is of insufficient quality, and the feedback message includes instructions to recapture the reference image. ### Claim 8 (depends on 1) The system of claim 1, the one or more computer readable media further storing instructions to: disable a capture interface on the control device based on the characteristic quantity, wherein the determining of the characteristic quantity based on the reference image is conducted in real time. ### Claim 9 (depends on 1) The system of claim 1, the one or more computer readable media further storing instructions to: generate a signature for a zone of interest using the reference image; associate a communication object with the zone of interest; receive a pointing target image from the on board imager; identify the signature using the pointing target image; and activate a communication interface for the communication object in response to identifying the signature. ### Claim 10 (depends on 1) The system of claim 1, the system further comprising: a display on the control device; and wherein the one or more computer readable media further stores instructions to: generate a signature for a zone of interest using the reference image; associate a controllable object with the zone of interest; receive a pointing target image from the on board imager; identify the signature using the pointing target image; and display a controllable object user interface on the display in response to identifying the signature. ### Claim 11 (depends on 1) The system of claim 1, the system further comprising: a motion tracker on the control device; and wherein the one or more computer readable media further stores instructions to either: determine the characteristic quantity based on the reference image and data from the motion tracker; or generate the feedback message based on the characteristic quantity and data from the motion tracker. ### Claim 12 A computer-implemented method, conducted with a control device having: (i) a shape which defines a pointing direction of the control device; and (ii) an on board imager having a field of view which includes the pointing direction, comprising: capturing a reference image indicative of a zone of interest in a physical location; determining a characteristic quantity based on the reference image, the characteristic quantity being representative of distinctiveness of a signature of the zone of interest; generating a feedback message based on the characteristic quantity; and predicating a setup phase termination for the control device on the characteristic quantity; wherein the method further comprises one of the following features (i) or (ii): (i) the step of determining the characteristic quantity includes determining that the reference image is of insufficient quality, and the feedback message includes instructions to recapture the reference image; or (ii) the method further comprises disabling a capture interface on the control device based on the characteristic quantity, wherein the step of determining the characteristic quantity based on the reference image is conducted in real time. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, further comprising: receiving at least one additional reference image; redetermining the characteristic quantity based on the reference image and the at least one additional reference image; and triggering, after the characteristic quantity has been redetermined, an end to the setup phase based on the characteristic quantity. ### Claim 14 A computer-implemented method, conducted with a control device having: (i) a shape which defines a pointing direction of the control device; and (ii) an on board imager having a field of view which includes the pointing direction, comprising: capturing a reference image using a companion device, wherein the companion device has a companion device imager and a display; determining a characteristic quantity based on the reference image; generating a feedback message based on the characteristic quantity; predicating a setup phase termination for the control device on the characteristic quantity; and displaying the reference image on the display. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein: the feedback message regards at least one additional reference image to be captured to improve the characteristic quantity. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, wherein: the feedback message includes instructions on how to capture the at least one additional reference image. ### Claim 17 (depends on 16) The computer-implemented method of claim 16, further comprising: capturing the at least one additional reference image using the companion device imager; and wherein the instructions specify at least one position for the companion device to be in when capturing the at least one additional reference image. ### Claim 18 (depends on 14) The computer-implemented method of claim 14, further comprising: continuously capturing a live stream of images using the on board imager; and wherein the feedback message includes an augmented reality stream of images generated using the live stream of images. ### Claim 19 (depends on 12) The computer-implemented method of claim 12, wherein: the step of determining the characteristic quantity includes determining that the reference image is of insufficient quality, and the feedback message includes instructions to recapture the reference image. ### Claim 20 (depends on 12) The computer-implemented method of claim 12, wherein the method further comprises: disabling a capture interface on the control device based on the characteristic quantity, and wherein the step of determining the characteristic quantity based on the reference image is conducted in real time. ### Claim 21 (depends on 12) The computer-implemented method of claim 12, further comprising: generating a signature for a zone of interest using the reference image; associating, with the zone of interest, one of: a communication object and a controllable object; receiving a pointing target image from the on board imager; identifying the signature using the pointing target image; and activating a communication interface for the communication object in response to identifying the signature. ### Claim 22 (depends on 12) The computer-implemented method of claim 12, further comprising: determining the characteristic quantity based on the reference image and data from a motion tracker on the control device; or generating the feedback message based on the characteristic quantity and data from the motion tracker. ### Claim 23 A system comprising: a control device, wherein the control device has a pointing direction; an onboard imager located on the control device, wherein the onboard imager has a field of view which includes the pointing direction; and one or more computer readable media storing instructions to: receive a set of reference images indicative of a zone of interest in a physical location; determine a characteristic quantity based on the set of reference images, the characteristic quantity being representative of distinctiveness of a signature of the zone of interest; generate a feedback message based on the characteristic quantity, and continuously capture a live stream of images using the onboard imager, wherein the feedback message includes an augmented reality stream of images generated using the live stream of images; and based on the characteristic quantity, terminate a setup phase for the control device. ### Claim 24 A system comprising: a control device, wherein the control device has a pointing direction; an onboard imager located on the control device, wherein the onboard imager has a field of view which includes the pointing direction; a companion device having a companion device imager and a display; and one or more computer readable media storing instructions to: receive a set of reference images captured by the companion device imager; determine a characteristic quantity based on the set of reference images; based on the characteristic quantity either: (i) generate a feedback message; or (ii) terminate a setup phase for the control device; and display the set of reference images on the display. ### Claim 25 (depends on 24) The system of claim 24, wherein: the feedback message regards at least one additional reference image to be captured to improve the characteristic quantity. ### Claim 26 (depends on 25) The system of claim 25, wherein: the feedback message includes instructions on how to capture the at least one additional reference image. ### Claim 27 (depends on 26) The system of claim 26, wherein: the at least one additional reference image is captured by the companion device imager; and the instructions specify at least one position for the companion device to be in when capturing the at least one additional reference image. ### Claim 28 (depends on 23) The system of claim 23, the one or more computer readable media further storing instructions to: generate a signature for a zone of interest using the set of reference images; associate a communication object with the zone of interest; receive a pointing target image from the on board imager; identify the signature using the pointing target image; and activate a communication interface for the communication object in response to identifying the signature. ### Claim 29 (depends on 23) The system of claim 23, the system further comprising: a motion tracker on the control device; and wherein the one or more computer readable media further stores instructions to either: determine the characteristic quantity based on the set of reference images and data from the motion tracker; or generate the feedback message based on the characteristic quantity and data from the motion tracker.