# US 11,907,448 B2 · Systems, devices, and methods for physical surface tracking with a stylus device in an AR/VR environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,448 B2
- Document ID: US11907448B2
- Status: Granted
- Kind: B2
- Application number: 18/135,059
- Filed: 2023-04-14
- Published: 2023-09-21
- Granted: 2024-02-20
- Earliest claimed: 2021-02-26
- Assignees: Logitech Europe S.A.
- Applicants: Logitech Europe S.A.
- Inventors: Mario Gutierrez; Philip Waldron; Fabien Zellweger; Damien O'Sullivan
- Web page: https://patentgrep.com/patent/US11907448B2

## CPC

- G06F 3/03545: {Pens or stylus}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06V 30/1423: {the instrument generating sequences of position coordinates corresponding to handwriting (preprocessing or recognising digital ink G06V30/32)}
- G06V 30/347: {Sampling; Contour coding; Stroke extraction}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method includes receiving sensor data from one or more sensors that corresponds to a location of a stylus device and determining that a tip of the stylus device is in contact with a physical surface based on the sensor data. While in contact, the method includes determining a new location of the tip of the stylus device based on the location data, mitigating a tracking error of the determined new location of the tip of the stylus device by translating the new location from a 3D space domain to a 2D space domain that corresponds to the physical surface and adjusting the new location based on a comparison of the new location and historical locations. The stylus device can further ascertain surface characteristics through sensor data and simulate surface characteristics via haptic feedback effects.

## Claims

### Claim 1

A stylus tracking system for use in an AR/VR environment, the stylus tracking system comprising:
one or more processors; and
a set of one or more sensors controlled by the one or more processors and configured to generate sensor data, the sensor data including location data usable to derive a location of a stylus device;
wherein the one or more processors are configured to:
determine a location of a tip of the stylus device based on the location data;
determine a location of a virtual surface in the AR/VR environment;
determine virtual surface characteristics of the virtual surface;
generate a haptic output on the stylus device when the tip of the stylus device makes contact with the virtual surface in the AR/VR environment, the haptic output based on the virtual surface characteristics of the virtual surface;
determine when a scaling of the virtual surface has changed; and
modify the haptic output on the stylus device based on the scaling of the virtual surface.

### Claim 2 (depends on 1)

The stylus tracking system of claim 1 wherein the scaling includes increasing or decreasing a magnification of the virtual surface in the AR/VR environment.

### Claim 3 (depends on 1)

The stylus tracking system of claim 1 wherein the set of one or more sensors includes at least one optical sensor or microphone, wherein the virtual surface characteristics of the virtual surface are based on sensor data of the optical sensor or microphone.

### Claim 4 (depends on 1)

The stylus tracking system of claim 1 wherein the virtual surface characteristics for the virtual surface are based on a selection of known surface types.

### Claim 5

A method of tracking a stylus device in an AR/VR environment, the method comprising:
receiving, by one or more processors from a set of one or more sensors, sensor data that includes location data usable to derive a location of a stylus device;
determining a location of a tip of the stylus device based on the location data;
determining a location of a virtual surface in the AR/VR environment;
determining virtual surface characteristics of the virtual surface;
generating a haptic output on the stylus device when the tip of the stylus device makes contact with the virtual surface in the AR/VR environment, the haptic output based on the virtual surface characteristics of the virtual surface;
determining when a scaling of the virtual surface has changed; and
modifying the haptic output on the stylus device based on the scaling of the virtual surface.

### Claim 6 (depends on 5)

The method of claim 5 wherein the scaling includes increasing or decreasing a magnification of the virtual surface in the AR/VR environment.

### Claim 7 (depends on 5)

The method of claim 5 wherein the set of one or more sensors includes at least one optical sensor or microphone, wherein the virtual surface characteristics of the virtual surface are based on sensor data of the optical sensor or microphone.