# US 10,871,820 B2 · Self-mixing based 2D/3D user input detection and scanning laser system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,871,820 B2
- Document ID: US10871820B2
- Status: Granted
- Kind: B2
- Application number: 16/408,295
- Filed: 2019-05-09
- Published: 2020-11-12
- Granted: 2020-12-22
- Earliest claimed: 2019-05-09
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Mehmet Mutlu; Ahmet Fatih Cihan
- Web page: https://patentgrep.com/patent/US10871820B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G01B 9/02007: {Two or more frequencies or sources used for interferometric measurement (using only beat G01B9/02003)}
- G01B 9/02027: {Two or more interferometric channels or interferometers}
- G01B 9/02092: {Self-mixing interferometers, i.e. feedback of light from object into laser cavity}
- G01B 11/14: for measuring distance or clearance between spaced objects or spaced apertures (G01B11/26 takes precedence; rangefinders G01C3/00)
- G01B 11/161: {by interferometric means}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0416: {Control or interface arrangements specially adapted for digitisers}
- G06F 3/0421: {by interrupting or reflecting a light beam, e.g. optical touch-screen}
- G06F 3/0423: {using sweeping light beams, e.g. using rotating or vibrating mirror}
- G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)}
- H04N 9/3194: {including sensor feedback}
- G01B 2290/65: Spatial scanning object beam
- G06F 2203/04108: Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
- H04N 9/3129: {scanning a light beam on the display screen (scanning a light beam on a screen in displays other than projection devices G09G3/02; scanning systems in general G02B26/10; projectors using laser light sources in general H04N9/3161)}
- H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed herein are electronic devices, and methods for their operation, that identify user inputs based on interaction of an object with input surfaces separate from the electronic devices. The electronic devices may include one or more self-mixing interferometry sensors that scan a field of view containing the input surface with a light beam, such as a laser beam emitted laser diode. Self-mixing of the emitted light with reflections can generate a self-mixing interferometry signal. Analysis of the self-mixing interferometry signal can allow for identification of an object, such as a user's finger, in the field of view. Deformation of the finger can be detected with the self-mixing interferometry sensor, and a user input identified therefrom.

## Claims

### Claim 1

A method of operating an electronic device, comprising:
scanning a field of view with a beam of light emitted by a self-mixing interferometry sensor;
identifying, using a self-mixing interferometry signal generated by the self-mixing interferometry sensor, an input surface in the field of view and an object in the field of view;
detecting a first distance between the input surface and the self-mixing interferometry sensor;
detecting a second distance between the object and the self-mixing interferometry sensor, the second distance shorter than the first distance;
detecting a third distance between the object and the self-mixing interferometry sensor, the third distance longer than the second distance and corresponding to an interaction between the object and the input surface; and
identifying a user input to the electronic device by comparing the second distance and the third distance.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining, from the self-mixing interferometry signal and a scan pattern, a location of the interaction between the object and the input surface;
mapping the location of the interaction to a portion of an image projected onto the input surface; and
altering an operation of the electronic device based on the user input; wherein altering the operation of the electronic device comprises at least one of:
changing the image;
executing an action of the electronic device associated with the portion of the image; or
providing a feedback to a user of the electronic device.

### Claim 3 (depends on 2)

The method of claim 2, wherein mapping the location of the interaction includes:
determining at least one of a topography or an orientation of the input surface; and
determining a correspondence between points of the image projected onto the input surface and the topography or the orientation of the input surface.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the beam of light is produced by a laser diode;
the self-mixing interferometry signal is an output of a photodetector associated with the laser diode;
identifying the input surface and the object is based at least on:
a first set of distances from the input surface to the laser diode determined using at least the self-mixing interferometry signal; and
a second set of distances from the object to the laser diode determined using at least the self-mixing interferometry signal;
the first distance corresponds to the first set of distances; and
the second distance and the third distance correspond to the second set of distances.

### Claim 5 (depends on 1)

The method of claim 1, further comprising determining that the object is a finger of a user; wherein a difference between the second distance and the third distance corresponds to a deformation of the finger caused by the finger applying a force against the input surface.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining a magnitude of the force applied by the finger based on the deformation;
determining that the user input is a first user input when the magnitude exceeds a first threshold but is less than a second threshold; and
determining that the user input is a second user input when the magnitude exceeds the second threshold.

### Claim 7 (depends on 5)

The method of claim 5, further comprising determining that the finger ceases applying the force against the input surface based on a reduction of the deformation.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
determining that the object is a finger of a user; wherein detecting the deformation of the finger includes:
determining a height of the finger above the input surface; and
comparing the height of the finger above the input surface to a finger thickness.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
determining a motion of the object within the field of view; and
identifying the user input based at least on the motion.

### Claim 10

An electronic device, comprising:
a self-mixing interferometry sensor operable to scan a field of view with a beam of light, the field of view including an input surface; and
a processor operably linked with the self-mixing interferometry sensor and operable to:
determine first distances from the self-mixing interferometry sensor to the input surface using a self-mixing interferometry signal generated by the self-mixing interferometry sensor;
determine, at a first time, second distances from the self-mixing interferometry sensor to an object in the field of view using the self-mixing interferometry signal generated by the self-mixing interferometry sensor;
determine, at a second time after the first time, third distances from the self-mixing interferometry sensor to the object using the self-mixing interferometry signal generated by the self-mixing interferometry sensor;
identify a deformation of the object due to an interaction of the object with the input surface using at least the first distances, the second distances, and the third distances; and
identify a user input to the electronic device from the identified deformation.

### Claim 11 (depends on 10)

The electronic device of claim 10, further comprising:
an image projector operable to project an image onto the input surface in the field of view; wherein:
the processor is further operable to:
determine, from the self-mixing interferometry signal and a scan pattern of the field of view, a position of the interaction of the object with the input surface;
map the position of the interaction to a portion of the image projected onto the input surface by the image projector; and
alter an operation of the electronic device based on the user input; and
altering the operation of the electronic device comprises at least one of:
changing the image;
executing an action of the electronic device associated with the portion of the image; or
providing feedback to a user of the electronic device.

### Claim 12 (depends on 10)

The electronic device of claim 10, wherein:
the beam of light is a laser light emitted from a laser diode; and
the self-mixing interferometry signal is an output of a photodetector associated with the laser diode.

### Claim 13 (depends on 12)

The electronic device of claim 12, wherein the laser diode emits infrared electromagnetic radiation.

### Claim 14 (depends on 12)

The electronic device of claim 12, wherein the processor is operable to cause a modulation of a bias current to the laser diode, the modulation causing a change in the self-mixing interferometry signal.

### Claim 15 (depends on 14)

The electronic device of claim 14, wherein:
the modulation of the bias current of the laser diode is a triangle wave modulation; and
the processor is operable to determine each of the first distances from the self-mixing interferometry sensor to first locations on the input surface and the second distances or the third distances from the self-mixing interferometry sensor to second locations or third locations on the object using a spectrum analysis of the self-mixing interferometry signal.

### Claim 16 (depends on 14)

The electronic device of claim 14, wherein:
the modulation of the bias current of the laser diode is a sinusoidal wave modulation; and
the processor is operable to determine each of the first distances from the self-mixing interferometry sensor to first locations on the input surface and the second distances or the third distances from the self-mixing interferometry sensor to second locations or third locations on the object using an I/Q analysis of the self-mixing interferometry signal.

### Claim 17 (depends on 10)

The method of claim 10, wherein the first time occurs during a calibration procedure, the calibration procedure comprising:
generating a baseline topographical map of the object with respect to the input surface;
storing the baseline topographical map with a memory of the electronic device; and
comparing the baseline topographical map with subsequent objects positioned proximate to the input surface as detected by the self-mixing interferometry sensor to identify subsequent user inputs.

### Claim 18

A method of operating an electronic device, comprising:
performing a first set of scans of a field of view with a beam of light emitted from a self-mixing interferometry sensor of the electronic device;
determining, from a self-mixing interferometry signal generated by the self-mixing interferometry sensor during the first set of scans, a topography of an input surface within the field of view;
projecting an image from an image projector of the electronic device onto the input surface;
performing a second set of scans of the field of view;
detecting, from the self-mixing interferometry signal measured by the self-mixing interferometry sensor during the second set of scans, an object in the field of view and a three-dimensional position of the object in the field of view;
modulating a bias current of the self-mixing interferometry sensor, the modulation causing a change in the self-mixing interferometry signal;
detecting a deformation of the object as the object interacts with the input surface from the self-mixing interferometry signal; and
determining a user input to the electronic device based on the deformation of the object in the field of view.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
determining a three-dimensional motion of the object in the field of view; and
determining the user input to the electronic device based on the three-dimensional motion of the object in the field of view.

### Claim 20 (depends on 19)

The method of claim 19, wherein:
the object is a first object in the field of view; and
the method further comprises:
detecting, from the self-mixing interferometry signal generated by the self-mixing interferometry sensor during the second set of scans, a second object in the field of view;
determining a three-dimensional motion of the second object in the field of view; and
determining the user input to the electronic device based on the three-dimensional motion of the first object and the three-dimensional motion of the second object.