# US 11,030,237 B2 · Method and apparatus for identifying input features for later recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,030,237 B2
- Document ID: US11030237B2
- Status: Granted
- Kind: B2
- Application number: 16/409,054
- Filed: 2019-05-10
- Published: 2019-10-31
- Granted: 2021-06-08
- Earliest claimed: 2012-05-25
- Assignees: Atheer, Inc.
- Applicants: Atheer, Inc.
- Inventors: Sleiman Itani; Allen Yang Yang
- Web page: https://patentgrep.com/patent/US11030237B2

## CPC

- G06F 16/5854: {using shape and object relationship}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 16/5838: {using colour}
- G06V 40/107: {Static hand or arm}
- G06V 40/193: {Preprocessing; Feature extraction}
- G06T 2207/10012: Stereo images
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20112: Image segmentation details
- G06T 2207/30196: Human being; Person
- G06T 2207/30241: Trajectory
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Disclosed are methods and apparatuses to recognize actors during normal system operation. The method includes defining actor input such as hand gestures, executing and detecting input, and identifying salient features of the actor therein. A model is defined from salient features, and a data set of salient features and/or model are retained, and may be used to identify actors for other inputs. A command such as “unlock” may be executed in response to actor input. Parameters may be applied to further define where, when, how, etc. actor input is executed, such as defining a region for a gesture. The apparatus includes a processor and sensor, the processor defining actor input, identifying salient features, defining a model therefrom, and retaining a data set. A display may also be used to show actor input, a defined region, relevant information, and/or an environment. A stylus or other non-human actor may be used.

## Claims

### Claim 1

A method, comprising:
defining a preliminary geometric feature for a real object;
defining a first region, wherein the first region is a first virtual reality object or a first augmented reality object defined for a position in real space or virtual space;
detecting, by a first imaging device, the real object at a first point in time based on the preliminary geometric feature, the real object being within the first region performing a first action associated with a first instruction;
in response to detecting the first action by the real object, executing the first instruction;
identifying, by a processing device, a defining feature of the real object as the real object performs the first action, wherein the defining feature of the real object is different than the preliminary geometric feature;
generating a model of the real object, wherein the model includes the defining feature of the real object;
detecting, by the first imaging device, the real object at a second point in time based on the model of the real object, the real object being within the first region performing a second action associated with a second instruction, wherein the second action is different than the first action; and
in response to detecting the second action by the real object, executing the second instruction.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
defining a limiting parameter for executing the first instruction or the second instruction; and
executing the first instruction or the second instruction within the liming parameter.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the real object comprises a human hand; and
the first action comprises at least one of a hand gesture or a hand posture.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
detecting, by the first imaging device, the real object at a third point in time based on the model of the real object, the real object being within the first region performing a third action associated with a third instruction, wherein the third action is different than the first action and the second action; and
in response to detecting the third action, executing the third instruction.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first region is substantially fully within a field of view of the first imaging device.

### Claim 6 (depends on 1)

The method of claim 1, wherein the real object comprises an end-effector.

### Claim 7 (depends on 1)

The method of claim 1, further comprising identifying a plurality of defining features of the real object in the first region.

### Claim 8 (depends on 1)

The method of claim 1, further comprising detecting another real object using the preliminary geometric feature.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
defining a second region, wherein the second region is a second virtual reality object or a second augmented reality object defined for a position in the real space or the virtual space;
detecting, by the first imaging device, the real object at a third point in time based on the model, the real object being within the second region performing one of the first action associated with the first instruction, the second action associated with the second instruction, or a third action associated with a third instruction; and
in response to detecting the first action, the second action, or the third action, executing the first instruction, the second action, or the third action, respectively.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
defining a second region, wherein the second region is a second virtual reality object or a second augmented reality object defined for a position in the real space or the virtual space;
detecting, by a second imaging device, the real object based on the model, the real object being within the second region performing one of the first action associated with the first instruction, the second action associated with the second instruction, or a third action associated with a third instruction; and
in response to detecting the first action, the second action, or the third action, executing the first instruction, the second action, or the third action, respectively.

### Claim 11

An apparatus, comprising:
a sensor configured to detect a first real object at a first point in time and a second point in time within a first region, wherein the first region is a first virtual reality object or a first augmented reality object defined for a position in real space or virtual space; and
a processing device coupled to the sensor, wherein the processing device is configured to:
determine the first real object is within the first region at the first point in time based on a preliminary geometric feature of the first real object, the first real object performing a first action is associated with a first instruction;
in response to detecting the first action by the first real object, execute the first instruction;
identify a defining feature of the first real object as the first real object performs the first action, wherein the defining feature of the first real object is different than the preliminary geometric feature;
generate a model of the first real object, wherein the model includes the defining feature of the first real object;
determine the first real object is within the first region at the second point in time based on the model of the object, the first real object performing a second action is associated with a second instruction; and
in response to detecting the second action by the first real object, execute the second instruction.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the processing device is configured to use the preliminary geometric feature to identify a second real object.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the sensor is an image sensor.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein the sensor is an audio sensor.

### Claim 15 (depends on 11)

The apparatus of claim 11, further comprising a display coupled to the processing device, wherein:
the sensor comprises an image sensor, the image sensor adapted to obtain an image in a field of view with the first real object therein; and
the display is configured to display at least a portion of the field of view.

### Claim 16 (depends on 11)

The apparatus of claim 11, wherein the sensor comprises a first image sensor and a second image sensor.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the first image sensor and second image sensor are disposed in a stereo configuration.

### Claim 18 (depends on 11)

The apparatus of claim 11, wherein the first real object comprises a stylus.

### Claim 19

A method, comprising:
defining, by a processing device, a preliminary geometric feature for a real object;
defining, by the processing device, a region, wherein the region is a virtual reality object or an augmented reality object defined for a position in real space or virtual space;
detecting, by an imaging device, the real object at a first point in time based on the preliminary geometrid feature, the real object performing a first action being associated with a first instruction;
in response to detecting the first action by the real object, executing the first instruction;
identifying, by the processing device, a defining feature of the real object as the real object performs the first action;
generating a model of the real object wherein the model includes the defining feature of the real object;
detecting, by the imaging device, the real object performing a second action at a second point in time based on the model of the real object; and
in response to detecting the second action by the real object, executing a second instruction.

### Claim 20 (depends on 19)

The method of claim 19, wherein the defining feature comprises at least one of an edge feature, a shape feature, a motion feature, a convex hull model, a concave hull feature, a joint feature, a convex feature, a concave feature, a complex geometric feature, an abstract geometric feature, or a ratio of dimension.