# US 11,307,762 B2 · Operating user interfaces > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,307,762 B2 - Document ID: US11307762B2 - Status: Granted - Kind: B2 - Application number: 16/770,717 - Filed: 2018-01-31 - Published: 2020-11-12 - Granted: 2022-04-19 - Earliest claimed: 2018-01-31 - Assignees: Hewlett-Packard Development Company, L.P. - Applicants: Hewlett-Packard Development Company, L.P. - Inventors: Andrew Hunter; Madhu Sudan Athreya; Will Allen - Web page: https://patentgrep.com/patent/US11307762B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06F 2203/0383: Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method, an apparatus and a machine-readable medium are disclosed. The method relates to operating a user interface. The method may include obtaining, using a sensor associated with a first processing apparatus, data indicative of a pose of an control element and a gesture performed by the control element; delivering the data indicative of the pose and the gesture from the first processing apparatus to a second processing apparatus, the second processing apparatus associated with a computing device for displaying a user interface, wherein the second processing apparatus is remote from the first processing apparatus; generating, based on the data indicative of the pose, a representation of the control element to be presented on the user interface; presenting the representation of the control element on the user interface; determining, using the second processing apparatus, an operation corresponding to the performed gesture; and performing the operation in respect of the user interface on the computing device. ## Claims ### Claim 1 A method comprising: obtaining, using an optical-based sensor associated with a first processing apparatus, data indicative of a pose of a control element and a gesture performed by the control element, wherein the data indicative of the pose corresponds to an orientation and position of the control element relative to the optical-based sensor and a reference surface; determining a vector corresponding to a change between the data indicative of the pose and gesture and a previous data indicative of a previous pose and gesture; delivering the vector from the first processing apparatus to a second processing apparatus, the second processing apparatus associated with a computing device for displaying a user interface, wherein the second processing apparatus is remote from the first processing apparatus; generating, based on the vector, a representation of the control element to be presented on the user interface; presenting the representation of the control element on the user interface; determining, using the second processing apparatus, an operation corresponding to the performed gesture; and performing the operation in respect of the user interface on the computing device. ### Claim 2 (depends on 1) A method according to claim 1, further comprising: presenting a representation of the user interface for display to the operator; and presenting the representation of the control element on the representation of the user interface. ### Claim 3 (depends on 2) A method according to claim 2, further comprising: delivering the data indicative of the pose and the gesture from the first processing apparatus to a display associated with the first processing apparatus; wherein presenting the representation of the user interface comprises presenting the representation of the user interface on the display. ### Claim 4 (depends on 1) A method according to claim 1, wherein the data indicative of the pose includes at least an outline of the control element. ### Claim 5 (depends on 1) A method according to claim 1, wherein generating the representation of the control element comprises generating a representation through which the user interface remains visible. ### Claim 6 (depends on 1) A method according to claim 1, further comprising: obtaining data indicative of a touch event performed by the control element on a touch input surface associated with the first processing apparatus; determining, using the second processing apparatus, a touch operation corresponding to the touch event; and performing a touch operation in respect of the user interface on the computing device. ### Claim 7 (depends on 1) A method according to claim 1, further comprising: encoding, using the first processing apparatus, the data indicative of the pose and the gesture prior to delivering the data to the second processing apparatus; and decoding, using the second processing apparatus, the data upon receipt of the data at the second processing apparatus. ### Claim 8 (depends on 1) A method according to claim 1, further comprising: performing a biometric evaluation of a feature in the data indicative of a pose of the control element; and determining, based on the biometric evaluation, whether or not the operator is authorized to operate the user interface. ### Claim 9 (depends on 1) A method according to claim 1, wherein generating the representation of the control element comprises: determining, based on the data indicative of the pose, a boundary of the control element; and modifying an appearance of the user interface outside the boundary of the control element. ### Claim 10 (depends on 1) A method according to claim 1, further comprising: determining, based on the obtained data, that the control element is tilted in a particular direction with respect to a reference plane; and causing the representation of the control element to move relative to the user interface in a direction based on the particular direction of tilt. ### Claim 11 (depends on 1) A method according to claim 1, further comprising: determining when the representation of the control element crosses a boundary defining an edge of the user interface; and presenting a representation of at least part of the control element on an auxiliary display; wherein the part of the representation of the control element presented on the auxiliary display comprises a portion of the representation of the control element which has crossed the boundary. ### Claim 12 An apparatus, comprising: a first display unit to display a user interface; and processing circuitry to: receive data representing a pose of a control element and a gesture performed by the control element, the data having been captured by an optical-based sensor remote from the apparatus, wherein the data representing the pose corresponds to an orientation and position of the control element relative to the optical-based sensor and a reference surface; determine a vector corresponding to a change between the data representing a pose and a previous data representing a previous second pose; generate, based on the vector, a portrayal of the control element to be displayed on the first display unit; display the portrayal of the control element on the first display unit; establish, from the received data, an operation associated with the gesture performed by the control element; and execute the operation in respect of the user interface on the apparatus. ### Claim 13 (depends on 12) An apparatus according to claim 12, wherein the processing circuitry is further to: determine when the portrayal of the control element crosses a boundary defining an edge of the user interface; and display a portrayal of at least part of the control element on a second display unit. ### Claim 14 (depends on 13) An apparatus according to claim 13, wherein the part of the control element displayed on the second display unit comprises a portion of the portrayal of the control element which has crossed the boundary.