# US 2023/0408251 A1 · Multimodal Proximity and Visuotactile Sensing Through Transmissive Membrane

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0408251 A1
- Document ID: US20230408251A1
- Status: Published application
- Kind: A1
- Application number: 18/327,981
- Filed: 2023-06-02
- Published: 2023-12-21
- Earliest claimed: 2022-06-10
- Assignees: The Trustees of the University of Pennsylvania
- Applicants: The Trustees of the University of Pennsylvania
- Inventors: James Henry Pikul; Mark Yim; Jessica Yin
- Web page: https://patentgrep.com/patent/US20230408251A1

## CPC

- G01B 11/24: for measuring contours or curvatures
- B25J 13/084: {Tactile sensors (in general G01L5/16, G01L5/22)}
- B25J 19/021: {Optical sensing devices}
- H04N 23/11: for generating image signals from visible and infrared light wavelengths
- H04N 23/13: with multiple sensors
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A visuotactile sensor, comprising: a deformable membrane; and an imaging train, the deformable membrane allowing at least partial transmission of light therethrough and onto the imaging train such that the imaging train collects light reflected through the membrane by an object on the opposite side of the membrane or contacting the membrane, and the imaging train further configured to collect light indicative of a deformation of the membrane by the object. A method, comprising: with an imaging train, collecting (a) a first light reflected through a deformable membrane by an object proximate to or contacting the deformable membrane and (b) a second light indicative of a deformation of the deformable membrane; and relating the at least one of the first light and the second light to an estimated position of the object, an estimated motion of the object, and an estimated deformation experienced by the membrane.

## Claims

### Claim 1

A visuotactile sensor, comprising:
a deformable membrane; and
an imaging train,
the deformable membrane allowing at least partial transmission of light therethrough and onto the imaging train such that the imaging train collects light reflected through the membrane by an object on the opposite side of the membrane or contacting the membrane, and
the imaging train further configured to collect light indicative of a deformation of the membrane by the object.

### Claim 2 (depends on 1)

The visuotactile sensor of claim 1, wherein the deformable membrane comprises a pattern that distorts with deformation of the deformable membrane, and wherein the imaging train is configured to (i) illuminate the pattern so as to give rise to ultraviolet, infrared, or visible light emission or reflection from the pattern and (ii) collect an ultraviolet, infrared, or visible light image of the pattern.

### Claim 3 (depends on 2)

The visuotactile sensor of claim 2, wherein the pattern comprises a pigment, the pigment optionally absorbing ultraviolet light and emitting visible light.

### Claim 4 (depends on 1)

The visuotactile sensor of claim 1, wherein (a) the light reflected by an object on the opposite side of the membrane or contacting the membrane is infrared light, (b) wherein the light indicative of a deformation of the membrane by the object is visible light, or both (a) and (b).

### Claim 5 (depends on 1)

The visuotactile sensor of claim 1, wherein the deformable membrane is substantially transparent to infrared light and is substantially opaque to visible light having a wavelength of from about 400 to about 700 nm.

### Claim 6 (depends on 1)

The visuotactile sensor of claim 1, wherein the imaging train comprises any one or more of a an ultraviolet light camera, a visual light camera, an infrared light camera, a source of ultraviolet light, a source of visible light, and a source of infrared light.

### Claim 7 (depends on 1)

The visuotactile sensor of claim 1, further comprising an enclosure, the membrane forming a boundary of the enclosure.

### Claim 8 (depends on 7)

The visuotactile sensor of claim 7, further comprising a source of pressure configured to exert a pressure within the enclosure.

### Claim 9 (depends on 7)

The visuotactile sensor of claim 7, further comprising a pressure sensor configured to measure a pressure within the enclosure.

### Claim 10 (depends on 1)

The visuotactile sensor of claim 1, further comprising a processor configured to (a) relate a pressure within an enclosure of the sensor to a deformation of the membrane, (b) modulate a pressure exerted against the deformable membrane, (c) relate an image of a pattern of the membrane to one or more pattern reference images, (d) relate an image of a pattern of the membrane to a deformation of the membrane, (e) relate a deformation of the membrane to a force exerted by the object on the membrane, (f) relate a deformation of the membrane to a torque exerted by the object on the membrane, (g) relate a deformation of the membrane to a position of the object on the membrane, or (h) any two or more of (a)-(g).

### Claim 11 (depends on 1)

The visuotactile sensor of claim 1, wherein the imaging train is configured to relate at least one of (i) light reflected through the membrane by the object on the opposite side of the membrane or contacting the membrane and (ii) light indicative of a deformation of the membrane by the object to at least one of a torque exerted on the membrane by the object, a force exerted on the membrane by the object, an optical characteristic of the object, a geometric characteristic of the object, a material characteristic of the object, a mechanical characteristic of the object, a position of the object, or any combination thereof.

### Claim 12 (depends on 1)

The visuotactile sensor of claim 1, further comprising a mechanism, the mechanism configured to move the deformable membrane.

### Claim 13 (depends on 1)

The visuotactile sensor of claim 1, wherein the visuotactile sensor is incorporated into a furnishing, a bed, a vehicle, an assembly system, a positioning system, a gripper, a robot, a prosthetic, a wearable device, a computer input device, a shelf, or any combination thereof.

### Claim 14

A system, comprising:
a visuotactile sensor that comprises a deformable membrane and an imaging train,
the imaging train configured to collect (i) light reflected through the deformable membrane by an object proximate to or contacting the membrane and (ii) light indicative of a deformation of the deformable membrane; and
a mechanism,
the system configured to actuate the mechanism in response to a light collected by the imaging train,
the light being indicative of any one or more of a position of the object, a motion of the object, and a deformation experienced by the membrane.

### Claim 15 (depends on 14)

The system of claim 14, wherein the mechanism effects relative motion between the visuotactile sensor and the object in response to a deformation of the membrane that exceeds a threshold magnitude and/or a threshold duration.

### Claim 16 (depends on 14)

The system of any claim 14, wherein the mechanism is configured to move the object.

### Claim 17 (depends on 14)

The system of claim 14, wherein the mechanism is configured to effect motivation or activation of an element exterior to the sensor.

### Claim 18 (depends on 14)

The system of claim 14, wherein the system is comprised in a furnishing, a bed, a vehicle, an assembly system, a positioning system, or any combination thereof.

### Claim 19

A method, comprising:
with an imaging train, collecting (a) a first light reflected through a deformable membrane by an object proximate to or contacting the deformable membrane and (b) a second light indicative of a deformation of the deformable membrane; and
relating the at least one of the first light and the second light to an estimated position of the object, an estimated motion of the object, and an estimated deformation experienced by the membrane.

### Claim 20 (depends on 19)

The method of claim 19, further comprising adjusting a position or motion of the object or of the deformable membrane in response to at least one of the position of the object, the motion of the object, or the deformation experienced by the membrane.

### Claim 21 (depends on 19)

The method of claim 19, further comprising generating a model of a trajectory of the object.

### Claim 22 (depends on 19)

The method of claim 19, further comprising actuating a mechanism in response to any one or more of the estimated position of the object, the estimated motion of the object, and the estimated deformation experienced by the membrane.

### Claim 23 (depends on 19)

The method of claim 19, wherein the mechanism is comprised in a furnishing, a bed, a vehicle, an assembly system, a positioning system, or any combination thereof.
