# US 10,481,682 B2 · System and method for generating virtual marks based on gaze tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,481,682 B2 - Document ID: US10481682B2 - Status: Granted - Kind: B2 - Application number: 15/473,104 - Filed: 2017-03-29 - Published: 2017-10-05 - Granted: 2019-11-19 - Earliest claimed: 2016-03-29 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Lewis James Marggraff; Nelson G. Publicover; Spencer James Connaughton - Web page: https://patentgrep.com/patent/US10481682B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/005: {Input arrangements through a video camera} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041) - G06F 3/0354: with detection of two-dimensional [2D] relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 40/171: by use of digital ink - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An electronic system generates at a display virtual writing corresponding to tracked motion of the tip of a pointer with respect to a surface based on proximity of the tip of the pointer to the surface and the gaze of a user's eye. The electronic system determines the location and motion of the tip of the pointer with respect to the surface based on images captured by scene cameras, and determines the focus and gaze direction of the user's eye based on images captured by a user-facing camera. By generating virtual writing at the display corresponding to tracked motion of the tip of the pointer based on proximity of the tip of the pointer to the surface and based on the focus and gaze direction of the user's eye, the electronic system can enable virtual writing and associated collaboration services without the need for a specialized writing surface or pointer. ## Claims ### Claim 1 A method comprising: tracking, by a processing unit, motion of a first pointer in relation to a first virtual writing surface based on images captured by a first scene camera; tracking, by the processing unit, a gaze of first user's eye based on images captured by a first user-facing camera by filtering for saccadic eye movements versus smooth pursuit eye movements in relation to the motion of the first pointer; and recording, by the processing unit, at a display, a first mark of the first pointer on the first virtual writing surface based on: the filtering for the smooth pursuit eye movements; a direction of the first pointer intersecting with the first virtual writing surface; the tracked motion of the first pointer in relation to the first virtual writing surface; and the tracked gaze of the first user's eye, wherein recording the first mark continues recording the first mark on the first virtual writing surface when the tracked gaze of the first user's eye leaves the first pointer for less than a threshold amount of time. ### Claim 2 (depends on 1) The method of claim 1, wherein recording the mark is based on the gaze of the first user's eye focusing on an end of the first pointer. ### Claim 3 (depends on 1) The method of claim 1, wherein recording the mark comprises depositing virtual ink on the first virtual writing surface. ### Claim 4 (depends on 1) The method of claim 1, wherein the first pointer comprises a virtual object. ### Claim 5 (depends on 1) The method of claim 1, further comprising: tracking, by the processing unit, motion of a second pointer in relation to a second virtual writing surface based on images captured by a second scene camera; tracking, by the processing unit, a gaze of second user's eye based on images captured by a second user-facing camera; and recording, by the processing unit, at the display, a mark of the second pointer on the second virtual writing surface based on the tracked motion of the second pointer in relation to the second virtual writing surface and based on the tracked gaze of the second user's eye. ### Claim 6 (depends on 1) The method of claim 1, further comprising: prior to recording the first mark of the first pointer by the processing unit at the display, updating the tracked motion of the first pointer with depth information of the first pointer with depth data from a depth sensor. ### Claim 7 (depends on 1) The method of claim 1, further comprising: emitting electromagnetic radiation at a predetermined rate by the first pointer, wherein tracking the motion of the first pointer includes detecting the emitted electromagnetic radiation. ### Claim 8 (depends on 1) The method of claim 1, wherein tracking the motion of the first pointer includes determining an orientation of the first pointer based on a marking on the first pointer. ### Claim 9 A method, comprising: tracking, by a processing unit, a gaze of a user's eye based on images captured by a first user-facing detector; tracking motion of an end of a first writing tool by the processing unit based on images captured by a first scene camera and filtered for saccadic eye movements versus smooth pursuit eye movements of the user's eye in relation to the motion of the first writing tool; and generating, by the processing unit, a mark at a display corresponding to the tracked motion of an end of the first writing tool with respect to a first virtual surface, wherein: generating the mark is based on a proximity of the end of the first writing tool to the first virtual surface, the filtered smooth pursuit eye movements, and the tracked gaze of a first user's eye; generating the mark includes depositing virtual ink at the display corresponding to the motion of the end of the first writing tool in response to identifying that a direction of the first writing tool intersects with the first virtual surface; and recording the first mark continues recording the first mark on the first virtual writing surface when the gaze of the first user's eye leaves the first pointer for less than a threshold amount of time. ### Claim 10 (depends on 9) The method of claim 9, wherein generating the mark is further in response to identifying that the gaze of the first user's eye is focused on the end of the first writing tool. ### Claim 11 (depends on 9) The method of claim 9, further comprising discontinuing generating the mark in response to identifying that the direction of the first writing tool does not intersect with the first virtual surface and resuming generating the mark in response to identifying that the direction of the first writing tool has re-intersected with the surface. ### Claim 12 (depends on 9) The method of claim 9, further comprising: tracking, with the scene camera, motion of an erase pointer with respect to the first virtual surface; and erasing at least a portion of the mark corresponding to the tracked motion of the erase pointer in response to identifying that eraser intersects with the first virtual surface and that the gaze of the first user's eye is focused on the eraser. ### Claim 13 (depends on 12) The method of claim 12, wherein the erase pointer comprises a virtual eraser. ### Claim 14 (depends on 9) The method of claim 9, further comprising: tracking, by the processing unit, motion of an end of a second writing tool with respect to a second virtual surface based on images captured by a second scene camera; tracking, by the processing unit, a gaze of a second user's eye based on images captured by a second user-facing detector; and generating, at the processing unit, a mark corresponding to the motion of the end of the second writing tool based on a direction of the end of the second writing tool intersecting with the second virtual surface and based on the gaze of the second user's eye. ### Claim 15 An electronic device, comprising: a first scene camera configured to track motion of an end of a first pointer with respect to a first virtual surface; a first user-facing camera configured to track a gaze of a first user's eye; and a processor configured to: filter the gaze of the first user's eye for saccadic eye movements versus smooth pursuit eye movements in relation to the motion of the first pointer; identify that a direction of the first pointer intersects with the first virtual surface; generate at a display a mark corresponding to the tracked motion of the end of the first pointer with respect to the first virtual surface based on a proximity of the end of the first pointer to the first virtual surface and the gaze of the first user's eye filtered for saccadic eye movements versus smooth pursuit eye movements in relation to the motion of the first pointer; and generate the mark by depositing virtual ink at the display corresponding to the motion of the end of the first pointer in response to identifying that the direction of the first pointer intersects with the first virtual surface and based on the filtered smooth pursuit eye movements; wherein generating the mark continues on the first virtual writing surface when the gaze of the first user's eye leaves the first pointer for less than a threshold amount of time. ### Claim 16 (depends on 15) The electronic device of claim 15, wherein the processor is configured to generate the mark in response to identifying that the gaze of the first user's eye is focused on the end of the first pointer. ### Claim 17 (depends on 15) The electronic device of claim 15, wherein the processor is further configured to discontinue generating the mark in response to identifying that the direction of the first pointer does not intersect with the first virtual surface and to resume generating the mark in response to identifying that the direction of the first pointer has re-intersected with the first virtual surface. ### Claim 18 (depends on 15) The electronic device of claim 15, wherein the first scene camera is further configured to track motion of an erase pointer with respect to the first virtual surface and the processor is further configured to erase at least a portion of the mark corresponding to the tracked motion of the erase pointer in response to identifying that erase pointer is in contact with the first virtual surface and that the gaze of the first user's eye is focused on the erase pointer. ### Claim 19 (depends on 15) The electronic device of claim 15, further comprising: a second scene camera configured to track motion of an end of a second pointer with respect to a second virtual surface; and a second user-facing camera configured to track a gaze of a second user's eye, wherein the processor is further configured to generate at the display a mark corresponding to the tracked motion of the end of the second pointer with respect to the second virtual surface based on a proximity of the end of the second pointer to the second virtual surface and the gaze of the second user's eye.