# US 9,437,002 B2 · Systems and methods for a dual modality sensor system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,437,002 B2
- Document ID: US9437002B2
- Status: Granted
- Kind: B2
- Application number: 14/496,037
- Filed: 2014-09-25
- Published: 2016-03-31
- Granted: 2016-09-06
- Earliest claimed: 2014-09-25
- Assignees: Elwha LLC
- Applicants: Elwha LLC
- Inventors: Jesse R. Cheatham, III; Roderick A. Hyde; Muriel Y. Ishikawa; Jordin T. Kare; Craig J. Mundie; Nathan P. Myhrvold; Robert C. Petroski; Eric D. Rudder; Desney S. Tan; Clarence T. Tegreene; Charles Whitmer; Andrew Wilson; Jeannette M. Wing; Lowell L. Wood, Jr.; Victoria Y. H. Wood
- Web page: https://patentgrep.com/patent/US9437002B2

## CPC

- G06V 20/64: Three-dimensional [3D] objects
- A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone
- A63F 13/424: involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- G01S 13/862: {Combination of radar systems with sonar systems}
- G01S 15/003: {Bistatic sonar systems; Multistatic sonar systems}
- G01S 15/58: Velocity or trajectory determination systems; Sense-of-movement determination systems {(velocity measurement in imaging systems G01S15/8979)}
- G01S 15/74: Systems using reradiation of acoustic waves, e.g. IFF, i.e. identification of friend or foe
- G01S 15/86: Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
- G01S 15/87: Combinations of sonar systems
- G01S 15/89: for mapping or imaging
- G06T 7/11: Region-based segmentation
- G06T 7/207: for motion estimation over a hierarchy of resolutions (multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53)
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/107: {Static hand or arm}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10132: Ultrasound image
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20212: Image combination
- G06T 2207/30196: Human being; Person
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure provides systems and methods for using two imaging modalities for imaging an object at two different resolutions. For example, the system may utilize a first modality (e.g., ultrasound or electromagnetic radiation) to generate image data at a first resolution. The system may then utilize the other modality to generate image data of portions of interest at a second resolution that is higher than the first resolution. In another embodiment, one imaging modality may be used to resolve an ambiguity, such as ghost images, in image data generated using another imaging modality.

## Claims

### Claim 1

A method for generating an image of an object within a region, comprising:
transmitting ultrasound, via a first ultrasonic transmitter of an ultrasonic transmission system, into the region;
receiving, via a first ultrasonic receiver of an ultrasonic receiver system, ultrasonic reflections of the transmitted ultrasound from a plurality of sites on the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;
generating, via a processor, coarse image data of the object at a first resolution based on the received ultrasonic reflections;
identifying a portion of interest on the object based on the coarse image data;
receiving electromagnetic radiation from the identified portion of interest on the object, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band;
generating fine image data of the portion of interest on the object at a second resolution based on the received electromagnetic radiation, wherein the second resolution is greater than the first resolution;
determining a kinematic value associated with the portion of interest on the object based on at least one of the received electromagnetic radiation and the received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and
modifying the state of an entertainment device based on the determined kinematic value associated with the portion of interest on the object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the portion of interest is identified based at least in part on a state of the entertainment device.

### Claim 3 (depends on 1)

The method of claim 1, further comprising transmitting electromagnetic energy into the region via an electromagnetic transmitter, and
wherein receiving electromagnetic radiation from the identified portion of interest on the object comprises receiving a reflected portion of the transmitted electromagnetic radiation.

### Claim 4 (depends on 1)

The method of claim 1, wherein at least one of the transmitted ultrasound and the received reflected ultrasound are rebounded from an auxiliary ultrasonic reflector.

### Claim 5 (depends on 1)

The method of claim 1, wherein calculating the kinematic value associated with the portion of interest on the object comprises a Doppler velocity calculation in which the first velocity (Vdop) is a function of: a calculated shift (Δs), the frequency of the transmitted ultrasound (f0), an arrival velocity of the ultrasound (Vin), and a departure velocity of the ultrasound (Vout).

### Claim 6 (depends on 1)

The method of claim 1, wherein detecting a shift and calculating the first velocity component are performed using a derivation or equivalent equation to the following equation:
f
r
=
(
C
-
V
o
C
+
V
o
)
⁢
f
t
,
where fr is based on the frequency of the received ultrasound, C is based on a velocity of the ultrasound in a medium within the region, Vo is based on a velocity of the site on the object relative to the medium, and ft is based on a frequency of the transmitted ultrasound.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
receiving, via a second ultrasonic receiver of the ultrasonic receiver system that is physically separated from the first ultrasonic receiver, an ultrasonic reflection from the object;
detecting a shift of the ultrasonic reflection received by the second ultrasonic receiver; and
calculating, via the processor, a second kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the second ultrasonic receiver.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
receiving, via a third ultrasonic receiver of the ultrasonic receiver system that is physically separated from the first ultrasonic receiver, an ultrasonic reflection from the object;
detecting a shift of the ultrasonic reflection received by the third ultrasonic receiver; and
calculating, via the processor, a third kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the third ultrasonic receiver.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first, second and third ultrasonic receivers are non-collinear with respect to one another.

### Claim 10 (depends on 1)

The method of claim 1, wherein a plurality of kinematic values are calculated for a corresponding plurality of sites on the object.

### Claim 11 (depends on 10)

The method of claim 10, wherein the plurality of kinematic values are used to determine a translational velocity of the object.

### Claim 12 (depends on 1)

The method of claim 1, wherein the portion of interest on the object comprises a portion of a person selected from the group consisting of a hand, finger, arm, leg, foot, toe, torso, neck, head, mouth, lip, and eye.

### Claim 13 (depends on 1)

The method of claim 1, wherein determining the kinematic value of the portion of interest on the object comprises:
transmitting ultrasound, via the ultrasound transmission system, into the region, wherein the region is bounded by a first surface;
receiving, via the ultrasonic receiver system, the direct ultrasonic reflection from the object;
generating direct positional data associated with the object based on the direct ultrasonic reflection;
receiving the rebounded ultrasonic reflection from the object, wherein the rebounded ultrasonic reflection comprises ultrasound reflected by the object and the first surface prior to being received by the ultrasonic receiver;
generating rebounded positional data using the rebounded ultrasonic reflection of the object from the first surface; and
generating enhanced positional data by combining the direct positional data and the rebounded positional data.

### Claim 14 (depends on 1)

The method of claim 1, wherein determining the kinematic value of the portion of interest on the object comprises:
transmitting ultrasound into the region, wherein the region is bounded by a first surface;
receiving the direct ultrasonic reflection from the object;
generating direct positional data associated with the object based on the direct ultrasonic reflection;
receiving a rebounded ultrasonic reflection from the object, wherein the rebounded ultrasonic reflection comprises ultrasound reflected by the object and the first surface prior to being received;
generating rebounded positional data using the rebounded ultrasonic reflection of the object from the first surface; and
generating enhanced positional data by combining the direct positional data and the rebounded positional data.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
receiving an additional rebounded ultrasonic reflection from the object,
wherein the additional rebounded ultrasonic reflection comprises ultrasound reflected by the object and a second surface bounding the region prior to being received;
generating additional rebounded positional data using the additional rebounded ultrasonic reflection of the object from the second surface; and
supplementing the enhanced positional data with the additional rebounded positional data.

### Claim 16 (depends on 14)

The method of claim 14, wherein transmitting the ultrasound comprises:
transmitting a first ultrasonic pulse that is received as the direct ultrasonic reflection; and
transmitting a second ultrasonic pulse that is received as the rebounded ultrasonic reflection.

### Claim 17

A method for generating an image of an object within a region, comprising:
receiving electromagnetic radiation from a plurality of sites on an object within a region, the electromagnetic radiation comprising electromagnetic waves within a first frequency band;
identifying a portion of interest on the object based on the received electromagnetic radiation;
transmitting ultrasound, via a first ultrasonic transmitter of an ultrasonic transmission system, into the region;
receiving, via a first ultrasonic receiver of an ultrasonic receiver system, ultrasonic reflections of the transmitted ultrasound from the portion of interest on the object within the region, the ultrasonic reflections comprising sound waves within a second frequency band, the second frequency band being different than the first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the portion of interest on the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the portion of interest on the object;
generating, via a processor, image data of the portion of interest on the object at a first resolution based on the received ultrasonic reflections;
determining a kinematic value associated with the portion of interest based on at least one of the received electromagnetic radiation and the received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and
modifying the state of an entertainment device based on the determined kinematic value associated with the portion of interest on the object.

### Claim 18 (depends on 17)

The method of claim 17, further comprising generating image data of the object at a second resolution based on the received electromagnetic radiation, wherein the second resolution is less than the first resolution.

### Claim 19 (depends on 17)

The method of claim 17, further comprising transmitting electromagnetic radiation into the region.

### Claim 20

A method for generating an image of an object within a region, comprising:
transmitting ultrasound, via an ultrasonic transmission system, into the region;
receiving, via an ultrasonic receiver system, ultrasonic reflections of the transmitted ultrasound from a plurality of sites on the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflections including direct ultrasonic reflection and rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;
generating, via a processor, image data of the object at a first resolution based on the received ultrasonic reflections, wherein the image data at the first resolution includes at least one ambiguity;
receiving electromagnetic radiation reflected by the object that is sufficient to resolve the at least one ambiguity, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band;
generating enhanced image data resolving the at least one ambiguity based on the received electromagnetic radiation;
determining a kinematic value associated with the object based on at least one of the received electromagnetic radiation and the received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and
modifying the state of an entertainment device based on the determined kinematic value associated with the portion of interest on the object.

### Claim 21 (depends on 20)

The method of claim 20, wherein resolving the at least one ambiguity comprises determining which of a plurality of images associated with the image data is a ghost image.

### Claim 22 (depends on 20)

The method of claim 20, further comprising directing electromagnetic radiation towards a site on the object associated with the at least one ambiguity.

### Claim 23

A system for generating an image of an object within a region, comprising:
an ultrasonic transmitter configured to transmit ultrasound into the region;
an ultrasonic receiver configured to receive an ultrasonic reflection of the transmitted ultrasound from a plurality of sites on the object within the region, the ultrasonic reflections comprising sound waves within a first frequency band, the ultrasonic reflection including a direct ultrasonic reflection and a rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;
a first imaging module configured to generate coarse image data of the object at a first resolution based on the received ultrasonic reflections;
an identification module configured to identify a portion of interest on the object based on the coarse image data;
an electromagnetic radiation receiver configured to receive electromagnetic radiation from the identified portion of interest on the object, the electromagnetic radiation comprising electromagnetic waves within a second frequency band, the second frequency band being different than the first frequency band;
a second imaging module configured to generate fine image data of the portion of interest on the object at a second resolution based on electromagnetic radiation received by the electromagnetic radiation receiver, wherein the second resolution is greater than the first resolution;
a kinematic determination module configured to determine a kinematic value associated with the portion of interest on the object based on at least one of received electromagnetic radiation and received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and
a modification module configured to modify the state of an entertainment device based on the kinematic value associated with the portion of interest on the object.

### Claim 24 (depends on 23)

The system of claim 23, wherein the kinematic value of the object comprises a velocity of the object.

### Claim 25 (depends on 23)

The system of claim 23, wherein at least one of the transmitted ultrasound and the reflected ultrasound are rebounded from an auxiliary ultrasonic reflector.

### Claim 26 (depends on 23)

The system of claim 23, further comprising:
a second ultrasonic receiver that is physically separated from the first ultrasonic receiver, wherein the second ultrasonic receiver is configured to receive an ultrasonic reflection from the site;
wherein a shift module is configured to detect a shift of the ultrasonic reflection received by the second ultrasonic receiver, and
wherein the kinematic value module is configured to calculate a second kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the second ultrasonic receiver.

### Claim 27 (depends on 26)

The system of claim 26, further comprising:
a third ultrasonic receiver that is physically separated from the first and second ultrasonic receivers, wherein the third ultrasonic receiver is configured to receive an ultrasonic reflection from the site,
wherein the shift module is configured to detect a shift of the ultrasonic reflection received by the third ultrasonic receiver, and
wherein the kinematic value module is configured to calculate a third kinematic value associated with the portion of interest on the object based on the detected shift of the ultrasonic reflection received by the third ultrasonic receiver.

### Claim 28 (depends on 27)

The system of claim 27, further comprising:
a prediction module configured to predict a relative position of the object at a future time based on (1) the first kinematic value associated with the portion of interest on the object, (2) the second kinematic value associated with the portion of interest on the object, and (3) the third kinematic value associated with the portion of interest on the object.

### Claim 29

A system for generating an image of an object within a region, comprising:
an electromagnetic radiation receiver configured to receive electromagnetic radiation from a plurality of sites on an object within a region, the electromagnetic radiation comprising electromagnetic waves within a first frequency band;
an identification module configured to identify a portion of interest on the object based on the received electromagnetic radiation;
an ultrasonic transmitter configured to transmit ultrasound into the region;
an ultrasonic receiver configured to receive an ultrasonic reflection of the transmitted ultrasound from the portion of interest on the object within the region, the ultrasonic reflections comprising sound waves within a second frequency band, the second frequency band being different than the first frequency band, the ultrasonic reflection including a direct ultrasonic reflection and a rebounded ultrasonic reflection, wherein the direct ultrasonic reflection is reflected from a first portion of the object and the rebounded ultrasonic reflection is reflected from a second, different portion of the object;
a first imaging module configured to generate image data of the portion of interest on the object at a first resolution based on the received ultrasonic reflections;
a kinematic determination module configured to determine a kinematic value associated with the portion of interest on the object based on at least one of received electromagnetic radiation and received ultrasonic reflections, wherein positional data is generated based at least on the direct ultrasonic reflection and the rebounded ultrasonic reflection, wherein the kinematic value is determined based at least in part on the positional data; and
a modification module configured to modify the state of an entertainment device based on the kinematic value associated with the portion of interest on the object.

### Claim 30 (depends on 29)

The system of claim 29, wherein the kinematic value of the object comprises a position of the object.

### Claim 31 (depends on 1)

The method of claim 1, wherein the first frequency band is between 20 kilohertz (kHz) and 250 kHz.

### Claim 32 (depends on 1)

The method of claim 1, wherein the first frequency band is between 35 kHz and 45 kHz.

### Claim 33 (depends on 17)

The method of claim 17, wherein the second frequency band is between 20 kilohertz (kHz) and 250 kHz.

### Claim 34 (depends on 18)

The method of claim 18, wherein the second frequency band is between 35 kHz and 45 kHz.

### Claim 35 (depends on 20)

The method of claim 20, wherein the first frequency band is between 20 kilohertz (kHz) and 250 kHz.

### Claim 36 (depends on 20)

The method of claim 20, wherein the first frequency band is between 35 kHz and 45 kHz.

### Claim 37 (depends on 23)

The system of claim 23, wherein the first frequency band is between 20 kilohertz (kHz) and 250 kHz.

### Claim 38 (depends on 23)

The system of claim 23, wherein the first frequency band is between 35 kHz and 45 kHz.

### Claim 39 (depends on 29)

The system of claim 29, wherein the second frequency band is between 20 kilohertz (kHz) and 250 kHz.

### Claim 40 (depends on 29)

The system of claim 29, wherein the second frequency band is between 35 kHz and 45 kHz.