# US 2012/0083960 A1 · System and method for predicting behaviors of detected objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0083960 A1
- Document ID: US20120083960A1
- Status: Published application
- Kind: A1
- Application number: 13/251,705
- Filed: 2011-10-03
- Published: 2012-04-05
- Earliest claimed: 2010-10-05
- Assignees: Google Inc.
- Inventors: Jiajun Zhu; David I. Ferguson; Dmitri A. Dolgov
- Web page: https://patentgrep.com/patent/US20120083960A1

## CPC

- B60R 1/23: with a predetermined field of view
- B60R 1/00: Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60T 7/22: initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle {, or by means of contactless obstacle detectors mounted on the vehicle}
- B60T 8/00: Arrangements for adjusting wheel-braking force to meet varying vehicular or ground-surface conditions, e.g. limiting or varying distribution of braking force (by changing number of effective brake cylinders in power brake systems B60T17/10)
- B60T 8/17: Using electrical or electronic regulation means to control braking {(detecting or indicating faulty operation B60T8/885)}
- B60T 8/885: {using electrical circuitry}
- B60T 17/18: Safety devices; Monitoring
- B60T 17/221: {Procedure or apparatus for checking or keeping in a correct functioning condition of brake systems (hydraulic pressure systems in general F15B19/00, F15B21/04; testing structures or apparatus G01M)}
- B60W 30/08: {Active safety systems} predicting or avoiding probable or impending collision {or attempting to minimise its consequences}
- B60W 40/06: Road conditions
- B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention
- B60W 60/0013: {specially adapted for occupant comfort}
- G01C 21/3617: {using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement}
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/931: of land vehicles
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0055: {with safety arrangements}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/021: {specially adapted to land vehicles}
- G05D 1/0212: {with means for defining a desired trajectory (involving a plurality of land vehicles G05D1/0287)}
- G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)}
- G05D 1/0276: {using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058)}
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06Q 10/02: Reservations, e.g. for tickets, services or events
- G06Q 30/02: Marketing; Price estimation or determination; Fundraising
- G06Q 30/0207: Discounts or incentives, e.g. coupons or rebates
- G06Q 40/08: Insurance
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/223: using block-matching
- G06T 7/231: using full search
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/74: {involving reference images or patches}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/64: Three-dimensional [3D] objects
- G07C 9/00563: {using personal physical data of the operator, e.g. finger prints, retinal images, voicepatterns}
- B60R 2300/30: characterised by the type of image processing
- B60T 2201/022: Collision avoidance systems
- B60T 2210/32: Vehicle surroundings
- B60T 2270/406: Test-mode; Self-diagnosis
- B60W 30/186: excessive wear or burn out of friction elements, e.g. clutches
- B60W 2050/0292: {Fail-safe or redundant systems, e.g. limp-home or backup systems}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2540/043: Identity of occupants
- B60W 2540/30: Driving style
- B60W 2552/05: Type of road, e.g. motorways, local streets, paved or unpaved roads
- B60W 2555/60: Traffic rules, e.g. speed limits or right of way
- B60W 2556/10: Historical data
- B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data
- B62D 6/00: Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits (means for initiating a change in direction B62D1/00; steering valves B62D5/06; combined with means for inclining the vehicle body or wheels on bends B62D9/00)
- G01S 13/865: {Combination of radar systems with lidar systems}
- G01S 13/867: {Combination of radar systems with cameras}
- G05B 2219/2637: Vehicle, car, auto, wheelchair
- G05D 1/024: {in combination with a laser (lasers per se H01S)}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0257: {using a radar (radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931)}
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G05D 1/0278: {using satellite positioning signals, e.g. GPS}
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30236: Traffic on road, railway or crossing
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2207/30261: Obstacle
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60T: VEHICLE BRAKE CONTROL SYSTEMS OR PARTS THEREOF; BRAKE CONTROL SYSTEMS OR PARTS THEREOF, IN GENERAL (electrodynamic brake systems for vehicle, in general B60L; brakes per se, i.e. devices where braking effect occurs, including ultimate brake actuators, F16D); ARRANGEMENT OF BRAKING ELEMENTS ON VEHICLES IN GENERAL; PORTABLE DEVICES FOR PREVENTING UNWANTED MOVEMENT OF VEHICLES; VEHICLE MODIFICATIONS TO FACILITATE COOLING OF BRAKES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Aspects of the invention relate generally to autonomous vehicles. Specifically, the features described may be used alone or in combination in order to improve the safety, use, driver experience, and performance of these vehicles.

## Claims

### Claim 1

A method for controlling a vehicle, the method comprising:
detecting an object external to a vehicle using one or more sensors;
using a processor, determining a classification and a state of the detected object;
predicting a likely behavior of the detected object based on behavior data for other objects having a classification and state similar to the detected object; and
providing a command to orient the vehicle in a position and velocity based at least in part on the likely behavior of the detected object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the classification of the detected object includes one of an automobile, a pedestrian, structure, or a bicycle.

### Claim 3 (depends on 1)

The method of claim 1, wherein the detected object is an automobile, and wherein the classification of the detected object includes the type of automobile.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determining a classification of the detected object is based on identifying at least one of a logo, a bumper sticker, or a license plate.

### Claim 5 (depends on 1)

The method of claim 1, wherein the state of the detected object relates to at least one of: location, traffic lane in which the detected object is traveling, speed, acceleration, entry onto a road, exit off of a road, activation of headlights, activation of taillights, or activation of blinkers.

### Claim 6 (depends on 1)

The method of claim 1, wherein the behavior data is based on movement data for a plurality of other objects at one or more locations.

### Claim 7 (depends on 6)

The method of claim 6, wherein the movement data are tracked using one of: satellite imagery, roadside cameras, on-board GPS data, or sensor data acquired for other nearby vehicles.

### Claim 8 (depends on 1)

The method of claim 1, wherein providing the command to orient the vehicle includes positioning the vehicle at a predetermined distance from the detected object, the predetermined distance being based, at least in part, on the classification of the detected object.

### Claim 9 (depends on 1)

The method of claim 1, wherein the likely behavior of the detected object is provided as a probability of the detected object entering to one or more states.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving updated behavior data; and
wherein predicting the likely behavior of the detected object is based at least in part on the updated behavior data.

### Claim 11 (depends on 10)

The method of claim 10, wherein receiving the updated behavior data comprises accessing a remote server.

### Claim 12

A system for controlling a vehicle having an autonomous operation mode, the system comprising:
one or more sensors for detecting an object in a vehicle's surroundings; and
a processor coupled to the one or more sensors, the processor being operable to:
control operation of the vehicle based on a first control strategy, wherein the first control strategy is not based on the detected object;
determine a classification and a state of the detected object;
access behavior data for other objects having a classification and a state similar to the detected object;
predict a likely behavior of the detected object based in at least in part on the accessed behavior data;
determine a second control strategy, wherein the second control strategy is based at least in part on the likely behavior of the detected object; and
control operation of the vehicle based on a second control strategy.

### Claim 13 (depends on 12)

The system of claim 12, wherein the classification of the detected object includes classifying the detected object as one of: an automobile, a pedestrian, a bicycle, or a structure.

### Claim 14 (depends on 12)

The system of claim 12, wherein the detected object is an automobile, and wherein the classification of the detected object includes identifying the type of automobile.

### Claim 15 (depends on 12)

The system of claim 12, wherein the classification of the detected object is based on at least one of: a logo, a bumper sticker, or a license plate.

### Claim 16 (depends on 12)

The system of claim 12, wherein the state of the detected object relates to at least one of: location, traffic lane in which the detected object is traveling, speed, acceleration, entry onto a road, exit off of a road, activation of headlights, activation of taillights, or activation of blinkers.

### Claim 17 (depends on 12)

The system of claim 12, wherein the behavior data represents information regarding the states of the other objects that have been tracked at one or more locations.

### Claim 18 (depends on 17)

The system of claim 17, wherein the entities have been tracked using at least one of: satellite imagery, roadside cameras, on-board GPS data, or via sensor data acquired from other nearby entities.

### Claim 19 (depends on 12)

The system of claim 12, wherein the command to orient the vehicle includes positioning the vehicle at a predetermined distance from the detected object, the predetermined distance being based, at least in part, on the classification of the detected object.

### Claim 20 (depends on 12)

The system of claim 12, wherein the likely behavior of the detected object is provided as a probability of the detected object entering to one or more states.

### Claim 21 (depends on 12)

The system of claim 12,
wherein the processor is further operable to access updated behavior data; and
wherein the likely behavior of the detected object is based at least in part on the updated behavior data.

### Claim 22 (depends on 21)

The method of claim 21, wherein the vehicle receives the updated behavior data from a remote server.

### Claim 23

A method for providing data in connection with controlling of a vehicle, the system comprising:
storing, at a server, behavior data for a plurality of objects, wherein the behavior data includes classification and state information for at least a subset of the plurality of objects;
receiving identification data from a remote device, wherein the identification data includes data relating to a detected object near the remote device;
determining, at the server, the classification and state of the detected object;
predicting a likely behavior of the detected object based on the behavior data for the plurality of objects having a classification and state similar to the detected object;
transmitting the likely behavior of the detected object to the remote device.
