# US 12,110,013 B2 · Prediction of pedestrian road crossing with vehicle sensor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,110,013 B2 - Document ID: US12110013B2 - Status: Granted - Kind: B2 - Application number: 17/961,885 - Filed: 2022-10-07 - Published: 2024-04-11 - Granted: 2024-10-08 - Earliest claimed: 2022-10-07 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Oded Bialer; Siddhartha Gupta - Web page: https://patentgrep.com/patent/US12110013B2 ## CPC - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - G01S 13/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S13/46) - B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering - G01S 7/415: {Identification of targets based on measurements of movement associated with the target} - G01S 13/52: Discriminating between fixed and moving objects or between objects moving at different speeds - G01S 13/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G01S 13/865: {Combination of radar systems with lidar systems} - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/931: of land vehicles - G01S 17/931: of land vehicles - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2554/4029: Pedestrians - B60W 2554/4044: Direction of movement, e.g. backwards - B60W 2554/4045: Intention, e.g. lane change or imminent movement - B60W 2554/801: Lateral distance - B60W 2556/40: High definition maps - B60W 2720/106: Longitudinal acceleration - B60W 2720/24: Direction of travel - G01S 2013/9318: {Controlling the steering} - G01S 2013/93185: {Controlling the brakes} - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract A system of controlling operation of a vehicle includes one or more sensors operatively connected to the vehicle. The sensors are configured to obtain respective data of a scene and include a radar unit. A command unit is adapted to receive the respective data and includes a processor and tangible, non-transitory memory on which instructions are recorded. The command unit is configured to determine an orientation angle of a pedestrian in the scene, a Doppler frequency of the pedestrian and a distance of the pedestrian from a border of a road, based in part on the respective data. The orientation angle is based on a heading of the pedestrian relative to a direction of a road. The command unit is configured to designate a status of the pedestrian as either crossing or not crossing based on the distance, the orientation angle and the Doppler frequency of the pedestrian. ## Claims ### Claim 1 A system of controlling operation of a vehicle, the system comprising: one or more sensors operatively connected to the vehicle and configured to obtain respective data of a scene, the one or more sensors including a radar unit; a command unit adapted to receive the respective data, the command unit including a processor and tangible, non-transitory memory on which instructions are recorded; wherein the command unit is configured to: determine an orientation angle of a pedestrian in the scene based in part on the respective data, the orientation angle being based on a heading of the pedestrian relative to a direction of a road; determine a Doppler frequency of the pedestrian and a distance of the pedestrian from a border of the road based in part on the respective data; and designate a status of the pedestrian as either crossing or not crossing based on the distance, the orientation angle and the Doppler frequency of the pedestrian. ### Claim 2 (depends on 1) The system of claim 1, wherein the command unit is adapted to change a vehicle trajectory and/or decelerate the vehicle based in part on the status of the pedestrian. ### Claim 3 (depends on 1) The system of claim 1, wherein: the status is designated as crossing when the distance is less than a threshold distance and the orientation angle is less than a threshold angle and the Doppler frequency of the pedestrian is above a threshold frequency; and the status is designated as not crossing otherwise. ### Claim 4 (depends on 3) The system of claim 3, wherein the orientation angle is between the heading of the pedestrian and a normal vector perpendicular to the direction of the road. ### Claim 5 (depends on 3) The system of claim 3, wherein the threshold frequency is about 160 Hz and the threshold angle is about 30 degrees. ### Claim 6 (depends on 1) The system of claim 1, wherein the command unit is adapted to: segment the scene into a road portion and a non-road portion, the direction and the border of the road being estimated based in part on the segmentation of the scene; detect a location of the pedestrian in three-dimensions; and generate a bounding box around the pedestrian in the scene, the orientation and the distance of the pedestrian being estimated from a center of the bounding box. ### Claim 7 (depends on 6) The system of claim 6, wherein the one or more sensors include a camera and the respective data from the camera is used to segment the scene. ### Claim 8 (depends on 6) The system of claim 6, further comprising: a geographical map accessible to the command unit, the command unit being adapted to employ information from the geographical map to segment the scene. ### Claim 9 (depends on 6) The system of claim 6, wherein the one or more sensors include a lidar unit and the respective data from the lidar unit is employed to generate the bounding box around the pedestrian. ### Claim 10 (depends on 1) The system of claim 1, wherein the command unit is adapted to: obtain a vehicle Doppler frequency based in part on the respective data from the radar unit; obtain an extracted frequency value at a center point of a bounding box surrounding the pedestrian in the scene; and obtain the Doppler frequency of the pedestrian by subtracting the vehicle Doppler frequency from the extracted frequency value. ### Claim 11 (depends on 10) The system of claim 10, wherein the vehicle Doppler frequency is based in part on a carrier frequency of a signal from the radar unit, a speed of light and a speed of the vehicle. ### Claim 12 A method of controlling operation of a vehicle in real-time, the vehicle having one or more sensors and a command unit with a processor and tangible, non-transitory memory, the method comprising: obtaining respective data of a scene, via the one or more sensors, and including a radar unit in the one or more sensors; determining an orientation angle of a pedestrian in the scene based in part on the respective data, the orientation angle being based on a heading of the pedestrian relative to a direction of a road; determining a Doppler frequency of the pedestrian and a distance of the pedestrian from a border of the road based in part on the respective data; and designating a status of the pedestrian as either crossing or not crossing based on the distance, the orientation angle and the Doppler frequency of the pedestrian. ### Claim 13 (depends on 12) The method of claim 12, further comprising: changing a vehicle trajectory and/or decelerating the vehicle based in part on the status of the pedestrian. ### Claim 14 (depends on 12) The method of claim 12, further comprising: designating the status as crossing when the distance is less than a threshold distance and the orientation angle is less than a threshold angle and the Doppler frequency of the pedestrian is above a threshold frequency; and designating the status as not crossing otherwise. ### Claim 15 (depends on 12) The method of claim 12, further comprising: segmenting the scene into a road portion and a non-road portion; detecting a location of the pedestrian in three-dimensions; and generating a bounding box around the pedestrian in the scene. ### Claim 16 (depends on 15) The method of claim 15, further comprising: including a camera in the one or more sensors; and using the respective data from the camera to segment the scene. ### Claim 17 (depends on 15) The method of claim 15, further comprising: including a lidar unit in the one or more sensors; and using the respective data from the lidar unit to generate the bounding box around the pedestrian. ### Claim 18 (depends on 15) The method of claim 15, further comprising: obtaining a vehicle Doppler frequency based in part on the respective data from the radar unit; obtaining an extracted frequency value at a center point of a bounding box surrounding the pedestrian in the scene; and obtaining the Doppler frequency of the pedestrian by subtracting the vehicle Doppler frequency from the extracted frequency value. ### Claim 19 A system of controlling operation of a vehicle, the system comprising: one or more sensors operatively connected to the vehicle and configured to obtain respective data, the one or more sensors including a radar unit, a camera and a lidar unit; a command unit adapted to receive the respective data, the command unit including a processor and tangible, non-transitory memory on which instructions are recorded; wherein the command unit is configured to determine an orientation angle of a pedestrian based in part on the respective data, the orientation angle being between a heading of the pedestrian and a normal vector perpendicular to a direction of a road; wherein the command unit is configured to determine a Doppler frequency of the pedestrian and a distance of the pedestrian from a border of the road based in part on the respective data; and wherein the command unit is configured to determine a status of the pedestrian in real-time as either crossing or not crossing, the status being crossing when the distance is less than a threshold distance, the orientation angle is less than a threshold angle and the Doppler frequency of the pedestrian is above a threshold frequency. ### Claim 20 (depends on 19) The system of claim 19, wherein the command unit is adapted to: segment the scene into a road portion and a non-road portion, via the respective data from the camera; detect a location of the pedestrian in three-dimensions; and generate a bounding box around the pedestrian in the scene, via the respective data from the lidar unit.