# US 10,768,692 B2 · Systems and methods for capturing and rendering hand skeletons over documents images for telepresence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,768,692 B2 - Document ID: US10768692B2 - Status: Granted - Kind: B2 - Application number: 15/901,888 - Filed: 2018-02-21 - Published: 2019-08-22 - Granted: 2020-09-08 - Earliest claimed: 2018-02-21 - Assignees: Fuji Xerox Co., Ltd. - Applicants: Fuji Xerox Co., Ltd. - Inventors: Chelhwon Kim; Patrick Chiu; Joseph Andrew Alkuino de la Pena; Laurent Denoue; Jun Shingu - Web page: https://patentgrep.com/patent/US10768692B2 ## CPC - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)} - G06T 11/00: Two-dimensional [2D] image generation - G06T 11/60: Creating or editing images; Combining images with text - G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation - G06V 40/113: {Recognition of static hand signs} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - H04N 7/15: Conference systems - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract In a telepresence scenario with remote users discussing a document or a slide, it can be difficult to follow which parts of the document are being discussed. One way to address this problem is to provide feedback by showing where the user's hand is pointing at on the document, which also enables more expressive gestural communication than a simple remote cursor. An important practical problem is how to transmit this remote feedback efficiently with high resolution document images. This is not possible with standard videoconferencing systems which have insufficient resolution. We propose a method based on using hand skeletons to provide the feedback. The skeleton can be captured using a depth camera or a webcam (with a deep network algorithm), and the small data can be transmitted at a high frame rate (without a video codec). ## Claims ### Claim 1 A computer-implemented method, the method being performed in a computerized system comprising a first camera, a second camera, and a central processing unit, the computer implemented method comprising: a. using the first camera to track a hand of a user and a document on a surface to obtain user hand data and document tracking data; b. using the second camera to capture a high-resolution image of the document based on the document tracking data; c. transmitting the high-resolution image of the document to a remote computing system; d. transmitting the obtained user hand data to the remote computing system; e. on the remote computing system, rendering a representation of the user hand data interposed over the high-resolution image of the document; and f. when the document is changed, repeating steps b. and c. only in response to the change. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the user hand data is separate and distinct from the high-resolution image of the document. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the user hand data comprises coordinates of hand joints of the user. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein rendering the representation of the user hand data comprises: generating a graphical visualization of the hand of the user based on the user hand data and document tracking data; and interposing the only graphical visualization of the hand of the user over the high-resolution image of the document. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the user hand data comprises coordinates of 21 hand joints of the user. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the user hand data is normalized. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the user hand data is hand skeleton data. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising rectifying the captured high-resolution image of the document. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising cropping the captured high-resolution image of the document. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the first camera is a web camera. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the second camera is a depth camera. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein rendering the representation of the user hand data comprising generating a virtual shadow of the user hand. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the rendering of the virtual shadow is based on a distance between the user hand and the document. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein the distance between the user hand and the document is determined using a depth camera. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, further comprising repeating steps a., d. and e. in a loop. ### Claim 16 (depends on 1) The computer-implemented method of claim 1, wherein the rendering is performed in a web browser. ### Claim 17 (depends on 1) The computer-implemented method of claim 1, wherein the hand of the user is tracked using a deep learning based hand pose estimator. ### Claim 18 (depends on 1) The computer-implemented method of claim 1, wherein the document tracking data comprises coordinates of four corners of the document. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, wherein the user hand data comprises coordinates of the hand of the user relative to the four corners of the document and does not include an image of the document. ### Claim 20 (depends on 1) The computer-implemented method of claim 1, wherein the rendering a representation of the user hand data comprises rescaling and translating the user hand data. ### Claim 21 A non-transitory computer-readable medium embodying a set of computer-executable instructions, which, when executed in connection with a computerized system comprising a central processing unit and a first camera and a second camera, cause the computerized system to perform a method comprising: a. using the first camera to track a hand of a user and a document on a surface to obtain user hand data and document tracking data; b. using the second camera to capture a high-resolution image of the document based on the document tracking data; c. transmitting the high-resolution image of the document to a remote computing system; d. transmitting the obtained user hand data to the remote computing system; e. on the remote computing system, rendering a representation of the user hand data interposed over the high-resolution image of the document; and f. when the document is changed, repeating steps b. and c. only in response to the change. ### Claim 22 A computerized system comprising a central processing unit, a first camera, a second camera, and a memory, the memory storing a set of computer-readable instructions causing the computerized system to perform a method comprising: a. using the first camera to track a hand of a user and a document on a surface to obtain user hand data and document tracking data; b. using the second camera to capture a high-resolution image of the document based on the document tracking data; c. transmitting the high-resolution image of the document to a remote computing system; d. transmitting the obtained user hand data to the remote computing system; e. on the remote computing system, rendering a representation of the user hand data interposed over the high-resolution image of the document; and f. when the document is changed, repeating steps b. and c. only in response to the change.