# US 11,907,475 B2 · System and machine learning method for localization of an input device relative to a touch sensitive surface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,475 B2 - Document ID: US11907475B2 - Status: Granted - Kind: B2 - Application number: 17/448,866 - Filed: 2021-09-24 - Published: 2022-03-31 - Granted: 2024-02-20 - Earliest claimed: 2020-09-25 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Hojjat Seyed Mousavi; Behrooz Shahsavari; Bongsoo Suh; Utkarsh Gaur; Nima Ferdosi; Baboo V. Gowreesunker - Web page: https://patentgrep.com/patent/US11907475B2 ## CPC - G06F 3/0418: {for error correction or compensation, e.g. based on parallax, calibration or alignment} - G06F 3/03545: {Pens or stylus} - G06F 3/04162: {for exchanging data with external devices, e.g. smart pens, via the digitiser sensing hardware} - G06F 3/04166: {Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving (Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184)} - G06F 3/0446: {using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/20: Ensemble learning - G06V 10/82: using neural networks - G06F 3/0442: {using active external devices, e.g. active pens, for transmitting changes in electrical potential to be received by the digitiser} - G06F 2203/04101: 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In some examples, an electronic device can use machine learning techniques, such as convolutional neural networks, to estimate the distance between a stylus tip and a touch sensitive surface (e.g., stylus z-height). A subset of stylus data sensed at electrodes closest to the location of the stylus at the touch sensitive surface including data having multiple phases and frequencies can be provided to the machine learning algorithm. The estimated stylus z-height can be compared to one or more thresholds to determine whether or not the stylus is in contact with the touch sensitive surface. In some examples, the electronic device can use machine learning techniques to estimate the (x, y) position and/or tilt and/or azimuth angles of the stylus tip at the touch sensitive surface based on a subset of stylus data. ## Claims ### Claim 1 An electronic device in communication with a touch sensitive surface, the electronic device comprising: memory storing instructions; and one or more processors configured to execute the instructions to cause the electronic device to perform a method comprising: applying, via drive circuitry coupled to the touch sensitive surface, drive signals to one or more first touch electrodes of the touch sensitive surface; sensing, via sense circuitry coupled to the touch sensitive surface, sense signals at one or more second touch electrodes of the touch sensitive surface; estimating, via a machine learning algorithm, a location of a stylus based on at least a subset of the sense signals from a first subset of the second touch electrodes; and estimating, via a second machine learning algorithm, a distance of the stylus from the touch sensitive surface based on at least a second subset of the sense signals sensed from a second subset of the second touch electrodes different from the first subset of the second touch electrodes. ### Claim 2 (depends on 1) The electronic device of claim 1, wherein the method further comprises: selecting at least the subset of the sense signals based on magnitudes of the sense signals. ### Claim 3 (depends on 1) The electronic device of claim 1, wherein the method further comprises: filtering the sense signals to obtain first signals having a first frequency and second signals having a second frequency different from the first frequency, wherein at least the subset of the sense signals includes the first signals and the second signals. ### Claim 4 (depends on 1) The electronic device of claim 1, wherein the method further comprises: filtering the sense signals to obtain first signals having a first phase and second signals having a second phase different from the first phase, wherein at least the subset of the sense signals includes the first signals and the second signals. ### Claim 5 (depends on 1) The electronic device of claim 1, wherein the method further comprises: estimating, via a second machine learning algorithm, a distance of the stylus from the touch sensitive surface based on at least a second subset of the sense signals; and in accordance with a determination that the distance of the stylus from the touch sensitive surface exceeds a predetermined threshold distance, forgoing estimating the location of the stylus, wherein estimating the location of the stylus is in accordance with a determination that the distance of the stylus from the touch sensitive surface is less than the predetermined threshold distance. ### Claim 6 (depends on 1) The electronic device of claim 1, wherein the method further comprises: estimating, via the machine learning algorithm, a tilt angle of the stylus based on at least the subset of the sense signals or an azimuth angle of the stylus based on at least the subset of the sense signals. ### Claim 7 (depends on 1) The electronic device of claim 1, wherein an estimated location of the stylus that is estimated via the machine learning algorithm and based on at least the subset of the sense signals is within 100 micrometers of an actual location of the stylus. ### Claim 8 A method comprising: at an electronic device in communication with a touch sensitive surface: applying, via drive circuitry coupled to the touch sensitive surface, drive signals to one or more first touch electrodes of the touch sensitive surface; sensing, via sense circuitry coupled to the touch sensitive surface, sense signals at one or more second touch electrodes of the touch sensitive surface; estimating, via a machine learning algorithm, a location of a stylus based on at least a subset of the sense signals from a first subset of the second touch electrodes; and estimating, via a second machine learning algorithm, a distance of the stylus from the touch sensitive surface based on at least a second subset of the sense signals sensed from a second subset of the second touch electrodes different from the first subset of the second touch electrodes. ### Claim 9 (depends on 8) The method of claim 8, further comprising: selecting at least the subset of the sense signals based on magnitudes of the sense signals. ### Claim 10 (depends on 8) The method of claim 8, further comprising: filtering the sense signals to obtain first signals having a first frequency and second signals having a second frequency different from the first frequency, wherein at least the subset of the sense signals includes the first signals and the second signals. ### Claim 11 (depends on 8) The method of claim 8, further comprising: filtering the sense signals to obtain first signals having a first phase and second signals having a second phase different from the first phase, wherein at least the subset of the sense signals includes the first signals and the second signals. ### Claim 12 (depends on 8) The method of claim 8, further comprising: estimating, via a second machine learning algorithm, a distance of the stylus from the touch sensitive surface based on at least a second subset of the sense signals; and in accordance with a determination that the distance of the stylus from the touch sensitive surface exceeds a predetermined threshold distance, forgoing estimating the location of the stylus, wherein estimating the location of the stylus is in accordance with a determination that the distance of the stylus from the touch sensitive surface is less than the predetermined threshold distance. ### Claim 13 (depends on 8) The method of claim 8, further comprising: estimating, via the machine learning algorithm, a tilt angle of the stylus based on at least the subset of the sense signals or an azimuth angle of the stylus based on at least the subset of the sense signals. ### Claim 14 (depends on 8) The method of claim 8, wherein an estimated location of the stylus that is estimated via the machine learning algorithm and based on at least the subset of the sense signals is within 100 micrometers of an actual location of the stylus. ### Claim 15 A non-transitory computer readable storage medium storing instructions that, when executed by an electronic device with one or more processors and in communication with a touch sensitive surface cause the electronic device to perform a method comprising: applying, via drive circuitry coupled to the touch sensitive surface, drive signals to one or more first touch electrodes of the touch sensitive surface; sensing, via sense circuitry coupled to the touch sensitive surface, sense signals at one or more second touch electrodes of the touch sensitive surface; estimating, via a machine learning algorithm, a location of a stylus based on at least a subset of the sense signals from a first subset of the second touch electrodes; and estimating, via a second machine learning algorithm, a distance of the stylus from the touch sensitive surface based on at least a second subset of the sense signals sensed from a second subset of the second touch electrodes different from the first subset of the second touch electrodes. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the method further comprises: selecting at least the subset of the sense signals based on magnitudes of the sense signals. ### Claim 17 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the method further comprises: filtering the sense signals to obtain first signals having a first frequency and second signals having a second frequency different from the first frequency, wherein at least the subset of the sense signals includes the first signals and the second signals. ### Claim 18 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the method further comprises: filtering the sense signals to obtain first signals having a first phase and second signals having a second phase different from the first phase, wherein at least the subset of the sense signals includes the first signals and the second signals. ### Claim 19 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the method further comprises: estimating, via a second machine learning algorithm, a distance of the stylus from the touch sensitive surface based on at least a second subset of the sense signals; and in accordance with a determination that the distance of the stylus from the touch sensitive surface exceeds a predetermined threshold distance, forgoing estimating the location of the stylus, wherein estimating the location of the stylus is in accordance with a determination that the distance of the stylus from the touch sensitive surface is less than the predetermined threshold distance. ### Claim 20 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the method further comprises: estimating, via the machine learning algorithm, a tilt angle of the stylus based on at least the subset of the sense signals or an azimuth angle of the stylus based on at least the subset of the sense signals. ### Claim 21 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein an estimated location of the stylus that is estimated via the machine learning algorithm and based on at least the subset of the sense signals is within 100 micrometers of an actual location of the stylus.