# US 12,214,487 B2 · Vision-based tactile measurement method and apparatus, chip, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,214,487 B2
- Document ID: US12214487B2
- Status: Granted
- Kind: B2
- Application number: 17/369,837
- Filed: 2021-07-07
- Published: 2021-10-28
- Granted: 2025-02-04
- Earliest claimed: 2019-05-16
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Yu Zheng; Zhongjin Xu; Zhengyou Zhang
- Web page: https://patentgrep.com/patent/US12214487B2

## CPC

- B25J 13/084: {Tactile sensors (in general G01L5/16, G01L5/22)}
- G01B 5/20: for measuring contours or curvatures
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24: Classification techniques
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 7/64: of convexity or concavity
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A vision-based tactile measurement method is provided, performed by a computer device (e.g., a chip) connected to a tactile sensor, the tactile sensor including a sensing face and an image sensing component, and the sensing face being provided with a marking pattern. The method includes: obtaining an image sequence collected by the image sensing component of the sensing face, each image of the image sequence comprising one instance of the marking pattern; calculating a difference feature of the marking patterns in adjacent images of the image sequence; and processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, a quantity of hidden layers in the feedforward neural network being less than a threshold.

## Claims

### Claim 1

A vision-based tactile measurement method performed by a computer device connected to a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern; and the method comprising:
obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern;
calculating a difference feature of the marking patterns in adjacent images of the image sequence; and
processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising:
calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and
calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.

### Claim 2 (depends on 1)

The method according to claim 1, wherein n hidden neurons are provided in the hidden layer, n being an integer;
the hidden layer is constructed based on hidden neurons of a logistic sigmoid function; and
the output layer is constructed based on output neurons of a normalized exponential softmax function or linear output neurons.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the feedforward neural network comprises: a location estimation model, and the location estimation model comprises a first hidden layer and a first output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the first hidden layer in the location estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of a contact location on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the first output layer in the location estimation model to process the feature representation of the contact location, to obtain the contact location on the surface of the object.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the feedforward neural network comprises: a contact force estimation model, and the contact force estimation model comprises a second hidden layer and a second output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the second hidden layer in the contact force estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the contact force on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the second output layer in the contact force estimation model to process the feature representation of the contact force, to obtain three-dimensional information of the contact force on the surface of the object, the three-dimensional information comprising at least one of a magnitude and a direction.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the feedforward neural network comprises: a surface classification model and at least two curvature estimation models; and
the processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result comprises:
calling the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a surface type of the surface of the object; and
calling a target curvature estimation model in the at least two curvature estimation models based on the surface type to perform the curvature prediction on the surface of the object, to obtain the local curvature radius of the surface of the object.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the curvature estimation model comprises: a spherical surface estimation model and a cylindrical surface estimation model; and the calling a target curvature estimation model in the at least two curvature estimation models based on the surface type to perform the curvature prediction on the surface of the object, to obtain the local curvature radius of the surface of the object comprises:
in a case that the surface type is a spherical surface, calling the spherical surface estimation model to perform first curvature prediction on the spherical surface, to obtain the local curvature radius of the spherical surface; and
in a case that the surface type is a cylindrical surface, calling the cylindrical surface estimation model to perform second curvature prediction on the cylindrical surface, to obtain the local curvature radius of the cylindrical surface.

### Claim 7 (depends on 5)

The method according to claim 5, wherein the surface classification model comprises a third hidden layer and a third output layer; and the calling the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a surface type of the surface of the object comprises:
calling the third hidden layer in the surface classification model to perform surface recognition on the difference feature of the marking patterns to obtain a feature representation of the surface type; and
calling the third output layer in the surface classification model to process the feature representation of the surface type, to obtain the surface type of the surface of the object.

### Claim 8 (depends on 6)

The method according to claim 6, wherein the spherical surface estimation model comprises a fourth hidden layer and a fourth output layer; and the calling the spherical surface estimation model to perform first curvature prediction on the spherical surface, to obtain the local curvature radius of the spherical surface comprises:
calling the fourth hidden layer in the spherical surface estimation model to perform the first curvature prediction on the spherical surface, to obtain a feature representation of the curvature prediction of the spherical surface; and
calling the fourth output layer in the spherical surface estimation model to process the feature representation of the curvature prediction of the spherical surface, to obtain the local curvature radius of the spherical surface.

### Claim 9 (depends on 6)

The method according to claim 6, wherein the cylindrical surface estimation model comprises a fifth hidden layer and a fifth output layer; and the calling the cylindrical surface estimation model to perform second curvature prediction on the cylindrical surface, to obtain the local curvature radius of the cylindrical surface comprises:
calling the fifth hidden layer in the cylindrical surface estimation model to perform the second curvature prediction on the cylindrical surface, to obtain a feature representation of the curvature prediction of the cylindrical surface; and
calling the fifth output layer in the cylindrical surface estimation model to process the feature representation of the curvature prediction of the cylindrical surface, to obtain the local curvature radius of the cylindrical surface.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the marking pattern comprises at least two marking points, and the difference feature of the marking patterns comprises at least one of displacement and deformation of the marking points.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises:
determining two closest marking patterns from adjacent images at an ith frame and an (i+1)th frame in the image sequence as the same marking pattern, i being an integer; and
calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the ith frame and the (i+1)th frame.

### Claim 13

A computer device in connection with a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern, the computer device further comprising a memory and a processor, wherein the memory stores a plurality of computer-readable instructions; and when the processor executes the plurality of computer-readable instructions, causes the computer device to perform a plurality of operations including:
obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern;
calculating a difference feature of the marking patterns in adjacent images of the image sequence; and
processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising:
calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and
calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.

### Claim 14 (depends on 13)

The computer device according to claim 13, wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.

### Claim 15 (depends on 13)

The computer device according to claim 13, wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises:
determining two closest marking patterns from adjacent images at an ith frame and an (i+1)th frame in the image sequence as the same marking pattern, i being an integer; and
calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the ith frame and the (i+1)th frame.

### Claim 16 (depends on 13)

The computer device according to claim 13, wherein the feedforward neural network comprises: a location estimation model, and the location estimation model comprises a first hidden layer and a first output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the first hidden layer in the location estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of a contact location on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the first output layer in the location estimation model to process the feature representation of the contact location, to obtain the contact location on the surface of the object.

### Claim 17 (depends on 13)

The computer device according to claim 13, wherein the feedforward neural network comprises: a contact force estimation model, and the contact force estimation model comprises a second hidden layer and a second output layer;
the calling the hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object comprises:
calling the second hidden layer in the contact force estimation model to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the contact force on the surface of the object; and
the calling the output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result comprises:
calling the second output layer in the contact force estimation model to process the feature representation of the contact force, to obtain three-dimensional information of the contact force on the surface of the object, the three-dimensional information comprising at least one of a magnitude and a direction.

### Claim 18

A non-transitory computer-readable storage medium, storing a plurality of computer-readable instructions, wherein the plurality of computer-readable instruction when executed by a processor of a computer device in connection with a tactile sensor, the tactile sensor comprising a sensing face and an image sensing component, and the sensing face being provided with a marking pattern, cause the computer device to perform a plurality of operations including:
obtaining an image sequence collected by the image sensing component of the sensing face that is in physical contact with a surface of an object, each image of the image sequence comprising one instance of the marking pattern;
calculating a difference feature of the marking patterns in adjacent images of the image sequence; and
processing the difference feature of the marking patterns using a feedforward neural network to obtain a tactile measurement result, further comprising:
calling a hidden layer in the feedforward neural network to perform feature extraction on the difference feature of the marking patterns to obtain a feature representation of the surface of the object, the feature presentation further including a curvature prediction on the surface of the object; and
calling an output layer in the feedforward neural network to process the feature representation to obtain the tactile measurement result, the tactile measurement result comprising a local curvature radius of the surface of the object.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the marking pattern comprises a grid, and the difference feature of the marking patterns comprises at least one of displacement of grid points and deformation of grid lines in the grid.

### Claim 20 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the calculating a difference feature of the marking patterns based on the marking patterns in adjacent images of the image sequence comprises:
determining two closest marking patterns from adjacent images at an ith frame and an (i+1)th frame in the image sequence as the same marking pattern, i being an integer; and
calculating the difference feature of the marking patterns based on at least one of locations and deformation of the marking patterns in the images at the ith frame and the (i+1)th frame.