# US 11,079,767 B2 · Lidar based recognition of ride hailing gestures for autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,079,767 B2
- Document ID: US11079767B2
- Status: Granted
- Kind: B2
- Application number: 16/365,094
- Filed: 2019-03-26
- Published: 2020-10-01
- Granted: 2021-08-03
- Earliest claimed: 2019-03-26
- Assignees: GM Cruise Holdings LLC
- Applicants: GM Cruise Holdings LLC
- Inventors: Rajaditya Mukherjee
- Web page: https://patentgrep.com/patent/US11079767B2

## CPC

- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/024: {in combination with a laser (lasers per se H01S)}
- G05D 1/2285: using voice or gesture commands
- G05D 1/242: Means based on the reflection of waves generated by the vehicle (using passive navigation aids external to the vehicle G05D1/244; using signals provided by artificial sources external to the vehicle G05D1/247)
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/107: {Static hand or arm}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G05D 2105/22: of humans
- G05D 2107/13: Spaces reserved for vehicle traffic, e.g. roads, regulated airspace or regulated waters
- G05D 2109/10: Land vehicles
- G05D 2111/17: {Coherent light, e.g. laser signals}
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An AV is described herein. The AV includes a lidar sensor system. The AV additionally includes a computing system that executes a gesture recognition component to determine, based upon lidar sensor data, whether a pedestrian in a driving environment of the AV is performing a hailing gesture. The AV can be configured to initiate a pickup maneuver to approach the pedestrian in response to determining that the pedestrian is performing a hailing gesture.

## Claims

### Claim 1

An autonomous vehicle (AV) comprising:
a lidar sensor; and
a computing system that is in communication with the lidar sensor, wherein the computing system comprises:
a processor; and
memory that stores instructions that, when executed by the processor, cause the processor to perform acts comprising:
receiving lidar data from the lidar sensor, the lidar data being data generated by the lidar sensor responsive to performing a lidar scan of a driving environment of the AV, the lidar data being indicative of a point cloud that includes a first plurality of points, each of the first plurality of points of the point cloud comprising data indicative of a coordinate triple corresponding to a location of an object in the driving environment;
determining that a second plurality of points is representative of a pedestrian in the driving environment, the second plurality of points being a subset of the first plurality of points;
identifying, based upon the second plurality of points, a skeletal model for the pedestrian in the driving environment;
generating a pose similarity score based upon the skeletal model;
determining that the pedestrian is making a hailing signal based upon the pose similarity score; and
controlling operation of the AV in the driving environment based upon determining that the pedestrian is making the hailing signal.

### Claim 2 (depends on 1)

The AV of claim 1, wherein the skeletal model comprises a plurality of joints, each of the joints being assigned data indicative of a position, wherein each of the second points is assigned to a respective joint in the joints.

### Claim 3 (depends on 2)

The AV of claim 2, wherein the position assigned to a first joint in the joints is based upon coordinate triples of the second points that are assigned to the first joint.

### Claim 4 (depends on 2)

The AV of claim 2, wherein the plurality of joints includes at least 15 joints.

### Claim 5 (depends on 4)

The AV of claim 4, wherein the plurality of joints includes less than or equal to 20 joints.

### Claim 6 (depends on 2)

The AV of claim 2, wherein generating the pose similarity score comprises performing a comparison between the skeletal model and a reference skeletal model, the reference skeletal model comprising a second plurality of joints, each of the second plurality of joints corresponding to a different joint in the skeletal model.

### Claim 7 (depends on 6)

The AV of claim 6, wherein performing the comparison between the skeletal model and the reference skeletal model comprises:
performing an affine transformation of the skeletal model to the reference skeletal model; and
computing a distance between each of the joints of the skeletal model and its corresponding joint in the second plurality of joints, wherein the pose similarity score is based upon the computed distances.

### Claim 8 (depends on 7)

The AV of claim 7, wherein generating the pose similarity score comprises computing the L2-norm of the computed distances.

### Claim 9 (depends on 1)

The AV of claim 1, wherein generating the pose similarity score comprises providing the skeletal model to a neural network, wherein responsive to receipt of the skeletal model the neural network computes a probability that the pedestrian is making the hailing signal, the probability being the pose similarity score.

### Claim 10 (depends on 9)

The AV of claim 9, wherein the skeletal model comprises a plurality of joints, each of the joints being assigned data indicative of a position, wherein each of the second points is assigned to a respective joint in the joints.

### Claim 11 (depends on 10)

The AV of claim 10, wherein each of the second points further comprises data indicative of an intensity of a lidar return, and wherein each of the joints is further assigned data indicative of an intensity value, the intensity value of a first joint in the joints being based upon intensities of the second points that are assigned to the first joint.

### Claim 12 (depends on 1)

The AV of claim 1, wherein controlling the operation of the AV comprises causing the AV to initiate a pickup maneuver responsive to determining that the pedestrian is making the hailing signal.

### Claim 13 (depends on 1)

The AV of claim 1, further comprising an imaging system, wherein the imaging system generates an image of the pedestrian, and wherein the acts further comprise:
confirming that the pedestrian is making a hailing signal based upon the image of the pedestrian; and
responsive to confirming that the pedestrian is making the hailing signal, controlling the operation of the AV in the driving environment to initiate a pickup maneuver.

### Claim 14

A method for controlling an autonomous vehicle (AV), the method comprising:
receiving a lidar point cloud from a lidar sensor mounted on the AV, the point cloud generated by the lidar sensor responsive to performing a lidar scan of a driving environment of the AV, the point cloud comprising a first plurality of points, each of the points of the point cloud being assigned data indicative of a coordinate triple corresponding to a location of an object in the driving environment;
identifying a second plurality of points in the point cloud that are representative of a pedestrian;
generating a skeletal model that is representative of a pose of the pedestrian based upon the second plurality of points;
computing a pose similarity score based upon the skeletal model;
determining that the pedestrian is performing a hailing gesture based upon the pose similarity score; and
controlling operation of the AV in the driving environment based upon determining that the pedestrian is making the hailing signal.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
repeating the acts of receiving the lidar point cloud, generating the skeletal model, computing the pose similarity score, and determining that the pedestrian is performing the hailing gesture a plurality of times; and
responsive to determining, a plurality of times, that the pedestrian is performing the hailing gesture, initiating a pickup maneuver to cause the AV to approach the pedestrian.

### Claim 16 (depends on 14)

The method of claim 14, further comprising:
confirming that the pedestrian is performing the hailing gesture based upon sensor data from another sensor mounted on the AV; and
responsive to confirming that the pedestrian is performing the hailing gesture, initiating a pickup maneuver to cause the AV to approach the pedestrian.

### Claim 17

A computing system comprising:
a processor; and
memory that stores instructions that, when executed by the processor, cause the processor to perform acts comprising:
computing a skeletal model for a pedestrian in a driving environment of an autonomous vehicle (AV) based upon a lidar point cloud generated by a lidar sensor included on the AV, the pedestrian represented in the lidar point cloud;
providing the skeletal model as input to a convolutional neural network;
outputting, by way of the convolutional neural network, a probability that the pedestrian is making the hailing signal;
determining that the pedestrian is making the hailing signal based upon the probability output by the convolutional neural network; and
causing the AV to initiate a pickup maneuver to approach the pedestrian responsive to determining that the pedestrian is making the hailing signal.

### Claim 18 (depends on 14)

The method of claim 14, wherein computing the pose similarity score comprises:
providing the skeletal model as input to a neural network; and
outputting, by way of the neural network, a probability that the pedestrian is making the hailing signal, the probability being the pose similarity score.

### Claim 19 (depends on 14)

The method of claim 14, wherein computing the pose similarity score comprises computing a measure of similarity between the skeletal model and a reference skeletal model, the measure of similarity being the pose similarity score, and wherein the reference skeletal model is representative of the hailing gesture.

### Claim 20 (depends on 17)

The computing system of claim 17, the acts further comprising:
determining that a subset of points of the lidar point cloud is representative of the pedestrian, and wherein computing the skeletal model for the pedestrian is based upon the subset of points having been determined to be representative of the pedestrian.