# US 2025/0093968 A1 · Handheld Controllers with Surface Marking Capabilities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0093968 A1
- Document ID: US20250093968A1
- Status: Published application
- Kind: A1
- Application number: 18/812,916
- Filed: 2024-08-22
- Published: 2025-03-20
- Earliest claimed: 2023-09-20
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Ian P Colahan; Christopher K Ewy; Yuhao Pan; Tong Chen; David D Dashevsky
- Web page: https://patentgrep.com/patent/US20250093968A1

## CPC

- G06F 3/0317: {in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/03545: {Pens or stylus}
- G06F 3/0383: {Signal control means within the pointing device}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system may include an electronic device such as a head-mounted device and a handheld controller for controlling the electronic device. The handheld controller may have a housing with an elongated shaft extending between first and second tip portions. A removable tip member may be coupled to the first tip portion. A position sensor may be located in the removable tip member and may be used to track surface markings made with the handheld controller on a surface. The position sensor may be an optical flow sensor, a visual inertial odometry camera, an interferometer, or other sensor. A force sensor at the first tip portion may be used to detect when the tip contacts a surface. The position sensor may switch from air input mode to surface marking mode in response to detecting contact with the surface.

## Claims

### Claim 1

A handheld controller, comprising:
a housing having an elongated shaft extending between first and second tip portions;
a removable tip member configured to be coupled to the first tip portion, wherein the removable tip member comprises a position sensor that tracks a position of the first tip portion relative to a surface; and
a force sensor in the first tip portion configured to gather force input through the removable tip member.

### Claim 2 (depends on 1)

The handheld controller defined in claim 1 wherein the force sensor comprises a strain gauge.

### Claim 3 (depends on 1)

The handheld controller defined in claim 1 wherein the position sensor comprises an optical flow sensor.

### Claim 4 (depends on 3)

The handheld controller defined in claim 3 wherein the optical flow sensor comprises an infrared light source that emits a pattern of illumination onto the surface and an image sensor that captures an image of the pattern of illumination on the surface.

### Claim 5 (depends on 3)

The handheld controller defined in claim 3 wherein the optical flow sensor is operable in a first mode in which a first frame rate is used for tracking the position of the first tip portion relative to the surface and a second mode in which a second frame rate is used for keypoint detection, wherein the second frame rate is lower than the first frame rate.

### Claim 6 (depends on 5)

The handheld controller defined in claim 5 wherein the optical flow sensor is adjusted between the first mode and the second mode based on the force input gathered with the force sensor.

### Claim 7 (depends on 6)

The handheld controller defined in claim 6 wherein the optical flow sensor switches from the second mode to the first mode when the force input indicates contact between the removable tip member and the surface.

### Claim 8 (depends on 1)

The handheld controller defined in claim 1 wherein the removable tip member is configured to be coupled to the first tip portion using a friction fit with a magnetic release.

### Claim 9 (depends on 1)

The handheld controller defined in claim 1 wherein the first tip portion has a spring-loaded member that retracts into the housing to form an opening that receives the removable tip member.

### Claim 10 (depends on 1)

The handheld controller defined in claim 1 wherein the position sensor comprises an interferometer.

### Claim 11 (depends on 1)

The handheld controller defined in claim 1 further comprising:
a motion sensor that gathers motion sensor data;
a camera that captures image data; and
control circuitry configured to determine a tilt of the elongated shaft relative to the surface based on at least one of the motion sensor data and the image data, wherein the position of the first tip portion relative to the surface is determined based on the tilt.

### Claim 12

An electronic device, comprising:
a housing having an elongated shaft extending between first and second tip portions;
an optical sensor at the first tip portion and configured to capture image data, wherein the optical sensor is operable in a surface marking mode and an air input mode; and
a force sensor configured to detect contact between the first tip portion and a surface, wherein the optical sensor is operated in the surface marking mode in response to detecting contact between the first tip portion and the surface.

### Claim 13 (depends on 12)

The electronic device defined in claim 12 wherein the optical sensor comprises a camera and wherein the camera has a higher frame rate in the surface marking mode than in the air input mode.

### Claim 14 (depends on 12)

The electronic device defined in claim 12 further comprising control circuitry configured to analyze the image data using key point detection techniques in the air input mode.

### Claim 15 (depends on 12)

The electronic device defined in claim 12 further comprising a motion sensor and a camera, wherein the control circuitry is configured to determine an angle of the elongated shaft relative to the surface based on information from at least one of the motion sensor and the camera.

### Claim 16 (depends on 12)

The electronic device defined in claim 12 wherein the optical sensor is located in a removable part of the first tip portion and the force sensor is located in a non-removable part of the first tip portion.

### Claim 17

A handheld controller, comprising:
a housing having an elongated shaft extending between first and second tip portions;
a removable tip member coupled to the first tip portion, wherein the removable tip member comprises an optical flow sensor for tracking surface markings made by the handheld controller on a surface; and
a spring-loaded member configured to retract into the housing to form an opening that receives a portion of the removable tip member.

### Claim 18 (depends on 17)

The handheld controller defined in claim 17 wherein the first tip portion comprises a first magnet and the removable tip member comprises a second magnet that is attracted to the first magnet.

### Claim 19 (depends on 17)

The handheld controller defined in claim 17 further comprising infrared light sources at the first tip portion that produce a sequence of illumination on the surface, wherein the optical flow sensor captures images of the illumination to determine an angle of the elongated shaft relative to the surface.

### Claim 20 (depends on 17)

The handheld controller defined in claim 17 further comprising a force sensor at the first tip portion and configured to detect contact between the removable tip member and the surface.
