# US 2019/0377435 A1 · Touch sensitive device with a camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377435 A1
- Document ID: US20190377435A1
- Status: Published application
- Kind: A1
- Application number: 16/439,536
- Filed: 2019-06-12
- Published: 2019-12-12
- Earliest claimed: 2018-06-12
- Assignees: Beechrock Limited
- Applicants: Rapt IP Limited
- Inventors: Julien Piot; Owen Drumm
- Web page: https://patentgrep.com/patent/US20190377435A1

## CPC

- G06F 3/042: by opto-electronic means
- G06F 3/0308: {comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/03545: {Pens or stylus}
- G06F 3/0383: {Signal control means within the pointing device}
- G06F 3/0416: {Control or interface arrangements specially adapted for digitisers}
- G06F 3/0421: {by interrupting or reflecting a light beam, e.g. optical touch-screen}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A touch sensitive device includes a touch surface and emitters and detectors. The emitters produce optical beams that propagate across the touch surface and are received by the detectors. Touch events from a touch object disturb the beams propagating toward the detectors. Beam values are recorded by the detectors and used to determine the touch events. The touch sensitive device also includes a camera that captures images of touch objects. The images and variations in the beams are analyzed to continuously track the touch object before, during, and after touch events.

## Claims

### Claim 1

A system comprising:
a touch surface and emitters and detectors, the emitters producing optical beams that propagate across the touch surface and are received by the detectors, wherein touches from a touch object in contact with the touch surface disturb the optical beams;
a camera positioned to capture images of at least one of: the touch object in contact with the touch surface or above the touch surface; and
a controller configured to:
receive beam data from the detectors for optical beams distributed by the touch object;
receive the captured images of the touch object from the camera; and
determine information about the touch object based on the beam data and the captured images.

### Claim 2 (depends on 1)

The system of claim 1, wherein to determine the information, the controller is configured to determine the spatial position of the touch object relative to the touch surface.

### Claim 3 (depends on 2)

The system of claim 2, wherein to determine the spatial position of the touch object, the controller is configured to determine the touch object is in contact with the touch surface based on the beam data and the captured images.

### Claim 4 (depends on 2)

The system of claim 2, wherein to determine the spatial position of the touch object, the controller is configured to:
determine the touch object is in contact with the touch surface based on at least one of the beam data or the captured images;
determine an approximate touch location of the touch object on the touch surface based on the captured images; and
modify the approximate touch location based on the beam data.

### Claim 5 (depends on 2)

The system of claim 2, wherein to determine the spatial position of the touch object, the controller is configured to:
determine the touch object is in contact with the touch surface based on the beam data; and
determine an orientation of the touch object based on the image data.

### Claim 6 (depends on 2)

The system of claim 2, wherein to determine the spatial position of the touch object, the controller is configured to:
determine the touch object is above the touch surface based on the beam data and the captured images; and
determine the spatial position of the touch object above the touch surface based on the captured images.

### Claim 7 (depends on 6)

The system of claim 6, wherein to determine the spatial position of the touch object, the controller is further configured to:
determine an orientation of the touch object based on the image data.

### Claim 8 (depends on 2)

The system of claim 2, wherein to determine the spatial position of the touch object, the controller is configured to:
determine a projected touch location on the touch surface before the touch object contacts the touch surface based on the image data; and
determine an actual touch location on the touch surface after the touch object contacts the touch surface based on the beam data.

### Claim 9 (depends on 8)

The system of claim 8, wherein to determine the actual touch location, the controller is configured to:
monitor beam data associated with beam paths in a region of the touch surface that includes the projected touch location.

### Claim 10 (depends on 2)

The system of claim 2, wherein the controller is further configured to track the spatial position of the touch object as the touch object moves.

### Claim 11 (depends on 10)

The system of claim 10, wherein the spatial position of the touch object is tracked in real-time.

### Claim 12 (depends on 1)

The system of claim 1, wherein to determine the information, the controller is configured to determine a touch object type of the touch object based on the beam data or the captured images.

### Claim 13 (depends on 1)

The system of claim 1, wherein to determine the information the controller is configured to:
determine an operational mode of the touch object based on the captured images and the beam data.

### Claim 14 (depends on 1)

The system of claim 1, wherein the touch object is a stylus, the captured images include light emitted by the stylus, and the controller being configured to determine information about the touch object includes the controller being configured to:
determine the touch object is a stylus based on the light emitted by the stylus; and
determine a touch location of the stylus on the touch surface based on the beam data.

### Claim 15 (depends on 1)

The system of claim 1, wherein the touch object is a stylus, the captured images include light emitted by the stylus, and the controller being configured to determine information about the touch object includes the controller being configured to track the spatial position of the stylus based on the light emitted by the stylus.

### Claim 16 (depends on 1)

The system of claim 1, wherein the touch object is a stylus, the captured images include light emitted by the stylus, and the controller being configured to determine information about the touch object includes the controller being configured to:
determine a touch location of the stylus on the touch surface based on the beam data; and
determine an orientation of the stylus based on the light emitted by the stylus.

### Claim 17 (depends on 1)

The system of claim 1, wherein the touch object is a stylus, the captured images include light emitted by the stylus, and the controller being configured to determine information about the touch object includes the controller being configured to:
determine a touch location of the stylus on the touch surface based on the beam data; and
distinguish the stylus from another touch object based on the light emitted by the stylus.

### Claim 18 (depends on 1)

The system of claim 1, wherein the camera is positioned on the periphery of the touch surface.

### Claim 19 (depends on 1)

The system of claim 1, wherein the camera is positioned below the touch surface.

### Claim 20 (depends on 1)

The system of claim 1, wherein the camera is a time of flight (TOF) camera.
