# US 9,434,309 B1 · Apparatus and method for estimating and using a predicted vehicle speed in an indirect vision driving task

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,434,309 B1
- Document ID: US9434309B1
- Status: Granted
- Kind: B1
- Application number: 14/611,905
- Filed: 2015-02-02
- Published: 2016-08-18
- Granted: 2016-09-06
- Earliest claimed: 2013-03-11
- Assignees: The United States of America as represented by the Secretary of the Army
- Applicants: U.S. Army Research Laboratory ATTN: RDRL-LOC-I
- Inventors: Christopher C. Smyth
- Web page: https://patentgrep.com/patent/US9434309B1

## CPC

- B60R 1/24: in front of the vehicle
- B60R 1/28: with an adjustable field of view
- B60W 30/06: Automatic manoeuvring for parking
- B60W 40/072: Curvature of the road
- G01P 3/38: using photographic means
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/60: Analysis of geometric attributes
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- B60R 2300/207: using multi-purpose displays, e.g. camera image and navigation or video on same display
- B60R 2300/302: combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
- G06T 2207/20112: Image segmentation details
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01P: MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT (measuring or recording blood flow A61B5/02, A61B8/06; monitoring speed or deceleration of electrically-propelled vehicles B60L3/00; vehicle lighting systems adapted to indicate speed B60Q1/54; determining position or course in navigation, measuring ground distance in geodesy or surveying G01C; combined measuring devices for measuring two or more variables of movement G01C23/00; measuring velocity of sound G01H; measuring velocity of light G01J7/00; measuring direction or velocity of solid objects by reception or emission of radiowaves or other waves and based on propagation effects, e.g. Doppler effect, propagation time, direction of propagation, G01S; measuring speed of nuclear radiation G01T; measuring acceleration of gravity G01V; {measuring or recording the speed of trains B61L23/00; speed indicators incorporated in motor vehicles B60K35/00; measuring frequency or phase G01R; traffic control G08G})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method and apparatus for predicting vehicle speed during an indirect vision driving task. A further method and apparatus for optimizing the display of a camera return during an indirect vision driving task based on operator perceived vehicle speed as set by the display characteristics and the field-of-view of the camera. A further method and apparatus for using the perceived speed as a driving task aid, in particular, as an electronic aider for optimizing the driving scene display characteristics of scene compression and camera field-of view. In this manner, the invention adjusts the perceived speed in order to match the operator's cognitive flow to the control dynamics needed from the operator for the task. The invention has application to autonomous driving where manual intervention is incorporated during critical events for particular tasks; and with limited display space within the vehicle, the display format is adjusted by the invention according to the operator's task needs.

## Claims

### Claim 1

A method for aiding an operator during performance of indirect vision driving task in a vehicle, while viewing a display of at least one video camera return of a driving scene via a display device, comprising:
determining parameters for a display being viewed during an indirect vision driving task and of a camera providing a camera return that is being displayed;
determining an actual speed of the vehicle during the indirect vision driving task;
calculating an estimated operator-perceived driving performance for the operator of the vehicle as would be expected to be visually perceived and mentally interpreted by the operator as being the operator's actual driving performance during performance of an indirect vision driving task while viewing scene display on the display device based on one or more of the display parameters, camera parameters, driving course, and the actual vehicle speed; and
adaptively controlling, via a processor, display characteristics of scene compression and/or field-of view for the display of the driving scene on the display device to the operator in a manner so as to mitigate differences between the estimated operator-perceived driving performance and the operator's actual driving performance during the performance of the indirect vision driving task.

### Claim 2 (depends on 1)

The method of claim 1, wherein calculating the estimated operator-perceived driving performance comprises calculating an estimated operator-perceived vehicle speed as would be expected to be visually perceived by the operator and mentally interpreted as being the actual vehicle speed during performance of an indirect vision driving task while viewing scene display on the display device.

### Claim 3 (depends on 2)

The method of claim 2, wherein the estimated operator-perceived vehicle speed is calculated for a plurality of sections of the driving course in the driving scene being displayed, from parameters for both of the display and of the video camera, and from the actual speed of the vehicle during the indirect vision driving task, based on an optic flow locus point seen on the said display for the vehicle by the operator, at a camera viewing distance and look-down angle determined by the display and camera parameters, and the driving course geometrical characteristics.

### Claim 4 (depends on 3)

The method of claim 3, wherein the estimated operator-perceived vehicle speed (Vp) comprises expressions for course sections which include:
a. an expression for a straight course, that is given by: Vp=VM*α+1/3, a function of the actual vehicle speed (VM) and a display scene compression ratio (α) relative to that of the camera;
b. an expression for a circular course with unlimited camera field-of-view (FOV), that is given by: VP=VM*sqrt(1+(η/(R*sin θ′c))2)*α+1/3, a function of the radius of curvature (R), where θ′c=a sin(η*α+2/3/ρ), and where ρ=η/sin θc, is the camera viewing distance to the locus point, where θc is the camera viewing angle to the locus point, and η is the camera height above ground level; and
c. an expression for a circular course with a limited camera horizontal field-of view (FOVL), that is given by: VP=VM*sqrt(1+(η/(R*sin θL))2)*sin2(FOVc/2)*α+1/3/sin2(FOVL), where FOVL<FOVc=2a sin(η/(2R*tan θ′c)), twice the horizontal viewing angle at the camera position to the locus point, and θL=a tan(η/(2R*sin(FOVL/2))), the camera look-down angle to the ground as seen at the camera-viewing limit.

### Claim 5 (depends on 1)

The method of claim 1, further comprising: calculating one or more actual driving task performance offsets of the vehicle from a reference path in the driving course seen by the driving scene camera for display in the driving scene.

### Claim 6 (depends on 5)

The method of claim 5, wherein the reference path is defined as a circular arc segment.

### Claim 7 (depends on 6)

The method of claim 6, wherein calculations of the offsets are based on the arc center position (Po: [xo,yo]) and radius (Ro), and on the vehicle position (Pv: [xv,yv]) and heading (θv) in the terrain course coordinates, such that:
a. the heading angular offset (θe) is expressed as the difference in heading between that of the vehicle and that of the reference arc tangent (θt) at the intersection point of the arc radius extended to the vehicle position, such that: θe=θt−θv; where θt=θr+Sr*η/2, and θr=a tan((yv−yo)/(xv−xo)), where Sr=+1 for a counterclockwise turn and Sr=−1 for a clockwise turn;
b. the position lateral offset (γe) is expressed as: γe=Sr*γem, where γem is the lateral offset magnitude, γem=−Ro+(yv−yo)*sin(θr)+(xr−xo)*cos(θr); and
c. the curvature offset (Ce) is expressed as the difference between the arc path curvature and that of the vehicle path: Ce=1/Ro−1/Rv, with curvature expressed as the reciprocal of the path radius, where in simplification of vehicle mechanics, the vehicle path radius is: Rv=abs(L/sin(θw)), with L the wheel base length and θw the vehicle tire wheel angle.

### Claim 8 (depends on 5)

The method of claim 5, wherein the reference path is defined as a straight line segment.

### Claim 9 (depends on 8)

The method of claim 8, wherein the calculations of the offsets are based on an origin position (Pto: [xto,yto]) on the segment and segment heading (θt), and on the vehicle position (Pv:[xv,yv]) and heading (θv) in the terrain course coordinates, such that:
a. the heading angular offset (θe) is expressed as the difference in heading between that of the vehicle and that of the reference line segment (θt), such that: θe=θt−θv;
b. the position lateral offset (γe) is expressed as: γe=Sr*γem, where γem is the lateral offset magnitude, γem=sqrt(Rti^2+Rtv^2), where Rtv is the straight line distance from the origin point to the vehicle and Rti is the distance along the reference line segment from the segment origin point to the intersection point (Pi:[xi,yi]) of the line with a normal to the line from the vehicle, such that Rti=(xv−xto)*cos(θt)+(yv−yto)*sin(θt), and the coordinates of the intersection point are: xi=Rti*cos(θt)+xto, yi=Rti*sin(θt)+yto; here, Sr=+1 for the vehicle to the right of the reference line and Sr=−1 for the vehicle to the left;
c. the curvature offset (Ce) is expressed as that for the vehicle path: Ce=−1/Rv, with curvature expressed as the reciprocal of the path radius, where in simplification of vehicle mechanics, the vehicle path radius is: Rv=abs(L/sin(θw)), with L the wheel base length and θw the vehicle tire wheel angle; and
d. the time offset for the time to start and end on the reference path.

### Claim 10 (depends on 1)

The method of claim 1, wherein calculating the estimated operator-perceived driving performance comprises calculating an estimated operator-perceived vehicle reference path as would be expected to be visually perceived by the operator and mentally interpreted as being the actual vehicle path to be followed during performance of an indirect vision driving task while viewing scene display on the display device.

### Claim 11 (depends on 10)

The method of claim 10, wherein the estimated operator-perceived vehicle reference path is used for pursuit tracking control by the operator of the vehicle from estimations of path curvature by visual fixations on target points of inflection of the reference path perceived by the operator from the driving scene display.

### Claim 12 (depends on 11)

The method of claim 11, where as a first order approximation, the estimated operator-perceived vehicle reference path is parallel to the line of travel, and includes calculations of:
a. an estimated operator-perceived angular size of a target point expressed as: Φf=Φ/a, where Φ is the angular size as seen in unity-display of the terrain;
b. estimated operator-perceived coordinates of the location of the target point for the x-coordinate direction: xf=α*xo*sin(φ/α)*sin(φ), and for the z-coordinate direction: zf=a*xo*cos(φ/α)*cos(φ);
c. an estimated operator-perceived speed of approach to the said target point, expressed as:
Vf=−Vo*sqrt((α^2−1)*cos(φ)^2+1), with
the speed component along the x-direction, expressed as:
Vfx=Vo*(α*sin(φ/α)*cos(φ)−cos(φ/α)*sin(φ)), and with
the speed component along the z-direction, expressed as:
Vfz=−Vo*(α*cos(φ/α)*cos(φ)+sin(φ/α)*sin(φ)); and
d. an estimated operator-perceived curvature at the location of said target point, expressed as:
Cf=(α^2−1)*sin(φ)^4/(α*xo*((α^2−1)*cos(φ)^3+1)^1.5); where:
α is the ratio to the camera scene FOV to that of display FOV as seen by the operator, xo is the lateral offset magnitude of the vehicle from the reference path in the terrain course coordinates, φ is the bearing from the camera to said visual fixation target point on the reference path Pf: [xo,zo], such that φ=a tan(xo/zo), Vo is the actual speed of the vehicle, and where the x-coordinate axis of the display lies along the lateral direction to the vehicle travel and the z-coordinate axis lies along the direction of vehicle travel originating from the camera position.

### Claim 13 (depends on 10)

The method of claim 10, wherein the estimated operator-perceived vehicle reference path is used for compensatory control by the operator of the vehicle from control offset errors, where the offsets are estimated by the operator from a reference path perceived by the operator from the optical flow as seen on the driving scene display.

### Claim 14 (depends on 13)

The method of claim 13, where as a first order approximation, the estimated operator-perceived vehicle reference path is parallel to the line of travel, and includes calculations of:
a. an estimated operator-perceived heading angular offset (θc), expressed as the arctangent of the ratio of the estimated operator-perceived speed in the x-direction to that in the z-direction: θc=a tan((α*sin(ψ/α)*cos(ψ)−cos(ψ/α)*sin(ψ))/(α*cos(ψ/α)*cos(ψ)+sin(ψ/α)*sin(ψ)));
b. an estimated operator-perceived position lateral offset (γc), expressed as: γc=α*xo*sin(ψ/a)/sin(ψ); and
c. and estimated operator-perceived curvature offset (Cc), expressed as:
Cc=(α^2−1)*sin(ψ)^4/(α*xo*((α^2−1)*cos(ψ)^3+1)^1.5); where:
α is the ratio to the camera scene FOV to that of display FOV as seen by the operator, xo is the lateral offset magnitude of the vehicle from the reference path in the terrain course coordinates, and w is the bearing from the camera to a focal point for the optical flow on the reference path, here estimated as at Pc: [xo,zo], with zo now at the optic flow locus origin point that seen by the operator on the display is at a camera viewing distance and look-down angle determined by the display and camera parameters, and by the driving course characteristics, such that ψ=a tan(xo/zo), and where the lateral and heading offsets of the reference path in the terrain are such that the position lateral offset γe=xo, and the heading angular offset θe=0.

### Claim 15 (depends on 14)

The method of claim 14, wherein the compensatory control uses signals which are weighted by error gains for the estimated operator-perceived heading curvature, heading and lateral offsets.

### Claim 16 (depends on 1)

The method of claim 1, wherein controlling the display of the driving scene on the display device to the operator based on the estimated operator-perceived driving performance comprises using a model of information processing which specifies task rules and corresponding feature sets from a knowledge database indexed by operator task attention states and in evaluation sets up the rules for activation, a rules processor activates the rules directing control, and a procedural processor controls the task execution where the task rules are functions of cognitive loading.

### Claim 17 (depends on 1)

The method of claim 1, wherein controlling the display of the driving scene on the display device to the operator based on the estimated operator-perceived driving performance comprises using the estimated operator-perceived driving performance as a metric of the task cognitive loading so as to control the display of the driving scene.

### Claim 18 (depends on 1)

The method of claim 1, wherein the driving scene that is displayed is controlled in accordance with a control strategy that includes adjustments of one or more of:
a. parameters of the driving scene camera;
b. format parameters of the display of the said camera return; and
c. the actual vehicle speed.

### Claim 19 (depends on 18)

The method of claim 18, wherein the adjustments are made in a manner so as to generate a cognitive flow rate for the operator that is optimal for a task.

### Claim 20 (depends on 18)

The method of claim 18, wherein the control strategy computes associated costs for the estimated operator-perceived driving performance, and selects the minimum cost adjustment.

### Claim 21

Apparatus for aiding an operator during performance of an indirect vision driving task in a vehicle by a vehicle operator while viewing a display of a video camera return of a driving scene via a display device, comprising:
a camera for generating a video camera return signal representative of an driving scene;
a video signal processor for applying the video camera return signal to a display for viewing of the driving scene by the vehicle operator;
a prediction modeler for predicting operator-perceived driving performance for the operator of the vehicle as would be expected to be visually perceived and mentally interpreted by the operator as being the operator's actual driving performance during performance of an indirect vision driving task while viewing scene display on the display device based on one or more of the display parameters, camera parameters, driving course, and the actual vehicle speed; and
an adapter aider for adaptively controlling display characteristics of scene compression and/or field-of view for the display of the driving scene on the display device to the operator in a manner so as to mitigate differences between the predicted operator-perceived driving performance and the operator's actual driving performance during performance of the indirect vision driving task.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the adapter aider controls the display of the driving scene in accordance with a control strategy that includes adjustments of one or more of:
a. parameters of the camera;
b. parameters of the display of the camera return; and
c. the vehicle speed.

### Claim 23 (depends on 21)

The apparatus of claim 21, wherein the adaptive aider determines the control strategy for the scheduling of the adjustments, uses a computational process for the scheduling of the adjustments with access to a database of task cost elements, computes by the information processing model associated cost variables as strategy costs for the predicted operator-perceived vehicle speeds of the adjustment combinations, and selects the minimum cost adjustment schedule, where the task cost elements are composed of corresponding subtask times and cognitive workload indexed by task attention states of the operator.

### Claim 24 (depends on 1)

The method of claim 1, wherein the estimated operator-perceived driving performance for the operator of the vehicle in a driving task with indirect vision differs from what would be the expected performance of the same driving task with direct vision.

### Claim 25 (depends on 1)

The method of claim 1, further comprising:
determining characteristics of one or more sections of a course in the driving scene being displayed to the operator via the display device.