# US 2020/0401821 A1 · Operating assistance method, control unit, operating assistance system and working device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401821 A1
- Document ID: US20200401821A1
- Status: Published application
- Kind: A1
- Application number: 16/977,803
- Filed: 2019-04-11
- Published: 2020-12-24
- Earliest claimed: 2018-05-25
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Jens Roeder; Michael Kessler; Patrick Koegel; Steffen Brueggert
- Web page: https://patentgrep.com/patent/US20200401821A1

## CPC

- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06T 2207/30196: Human being; Person
- G06T 2207/30261: Obstacle
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

An operating assistance method for a working device or for a vehicle. Object boxes for an object in a field of view of the working device are obtained at consecutive times. From object boxes of a given object, for images recorded in succession or direct succession, an instantaneous change of scale of an object box for the specific object, and an instantaneous lateral change of position of the object box for the specific object, are determined. An object box predicted in the future is determined from the current change of scale, and from the instantaneous lateral change in position for an object box for a specific object. The position of the predicted object box and/or the ratio of a lateral extension of the predicted object box to a lateral extension of a covered field of view and/or of the recorded images are determined and evaluated.

## Claims

### Claim at position 1

1-13. (canceled)

### Claim 14

An operating assistance method for a working device, comprising the following steps:
(S1) obtaining, at consecutive times, object boxes for an object in a field of view of the working device, or data characterizing the object boxes;
(S2) from the object boxes of the object for images recorded in succession or direct succession, determining: (i) an instantaneous change of scaling of an object box for the object, or variables derived from the instantaneous change of scaling the object box for the object, and (ii) an instantaneous lateral change of position of the object box for the object;
(S3) determining a predicted object box, predicted in for the future, from the instantaneous change of scaling of the object box for the object or the variables derived from the instantaneous change of scaling of the object box for the object, and from the instantaneous lateral change in position of the object box for the object;
(S4) determining and evaluating: (i) a position of the predicted object box and/or (ii) a ratio of a lateral extension of the predicted object box to a lateral extension of: (a) a covered field of view, and/or (b) the recorded images or a section of the recorded images; and
(S5) as a function of the result of the evaluation, determining: (i) whether or not the object which forms the basis of the predicted object box, is critical with regard to a possible collision, and/or (ii) an operating state of the working device being controlled or regulated.

### Claim 15 (depends on 14)

The operating assistance method as recited in claim 14, wherein in step (S1) includes:
(S1a) optically covering the field of view of the working device two-dimensionally and/or monocularly, by recording temporally consecutive images; and
(S1b) in the consecutive, recorded images, determining the object and the object box associated with the object.

### Claim 16 (depends on 14)

The operating assistance method as recited in claim 14, wherein the predicted object box is determined for at least an image currently recorded last, by determining and updating iterative values for the scaling of the object box, for coordinates of the object box, for a translation of the object box, and for a lateral width of the object box, over a plurality of time increments up to a prediction time span.

### Claim 17 (depends on 16)

The operating assistance method as recited in claim 16, wherein for each of the time increments, the following steps are carried out:
(I1) resetting or presetting values to be calculated, in accordance with the following assignment rules
Scalingold:=Scalingnew
BoxTranslationXold:=BoxTranslationXnew
BoxWidthold:=BoxWidthnew
LeftBoxPositionold:=LeftBoxPositionnew
RightBoxPositionold:=RightBoxPositionnew;
(I2) updating the scaling in accordance with the following assignment rule
Scalingnew:=1/(2−Scalingold)
(I3) updating a horizontal translation of the object box in accordance with the following assignment rule
BoxTranslationXnew:=BoxTranslationXold×Scalingold
(I4) updating a horizontal width of the object box in accordance with the following assignment rule
BoxWidthnew:=RightBoxPositionold−LeftBoxPositionold
(I5) predicting horizontal box positions of the object in accordance with the following assignment rules
LeftBoxPositionnew:=LeftBoxPositionold+BoxTranslationXnew−0.5×BoxWidthnew×(Scalingnew−1)/Scalingnew
RightBoxPositionnew:=RightBoxPositionold+BoxTranslationXnew+0.5×BoxWidthnew×(Scalingnew−1)/Scalingnew;
wherein:
Scalingold and Scalingnew respectively designate an old and new scaling, or values of the old and new scaling, of the object box;
BoxTranslationXold and BoxTranslationXnew respectively designate an old and new displacement, or values of the old and new displacement, of the object box;
BoxWidthold and BoxWidthnew respectively designate an old and new width, or values of the old and new width, of the object box;
LeftBoxPositionold and LeftBoxPositionnew respectively designate an old and new position, or values of the old and new position, of a lower left corner of the object box in the form of a first x-coordinate of the object box; and
RightBoxPositionold and RightBoxPositionnew respectively designate an old and new position, or values of the old and new position, of a lower right corner of the object box in the form of a second x-coordinate of the object box.

### Claim 18 (depends on 16)

The operating assistance method as recited in claim 16, wherein the following computational rule is implemented for ascertaining a new box position of the object box:
LeftBoxPositionnew:=(LeftBoxPositioncurrent+LeftBoxSpeedcurrent*TPrediction)/(1+NormSpeedcurrent*TPrediction)
RightBoxPositionnew:=(RightBoxPositioncurrent+RightBoxSpeedcurrent*TPrediction)/(1+NormSpeedcurrent*TPrediction);
wherein:
LeftBoxPositionnew and LeftBoxPositioncurrent and RightBoxPositionnew and RightBoxPositioncurrent are new and current positions, respectively, of left and right box edges, respectively, of the object box;
LeftBoxSpeedcurrent and RightBoxSpeedcurrent are currently measured angular speeds of the left and right box edges, respectively;
NormSpeedcurrent is a currently measured, normalized box speed of the object box;
TPrediction is a prediction time of step (S4); and
wherein NormSpeedcurrent is derived from the determined change of scaling of the object box.

### Claim 19 (depends on 14)

The operating assistance method as recited in claim 14, wherein the object forming the basis of the predicted object box is determined to be critical with regard to a possible collision and to have a value of criticality of 100%, when a portion of the a width of the predicted object box at a width of an underlying image or a predefined section of the underlying image, exceeds a predetermined, first threshold value.

### Claim 20 (depends on 19)

The operating assistance method as recited in claim 19, wherein the value of criticality determined for the object is reduced by a level, at which the predicted object box for the object is positioned, in the width of the predicted object box, outside of the underlying image or the predefined section.

### Claim 21 (depends on 14)

The operating assistance method as recited in claim 14, wherein the object, which forms the basis of the predicted object box, is determined to be uncritical with regard to a possible collision and to have a value of criticality of 0%, when the predicted object box lies completely outside of an underlying image or a predefined section of the underlying image.

### Claim 22 (depends on 14)

The operating assistance method as recited in claim 14, wherein the object is a pedestrian, a position and motion of the pedestrian in the form of an object are checked and evaluated based on a pedestrian model, an ability of the pedestrian to accelerate is determined on based on a speed ascertained for the pedestrian, and the criticality for the pedestrian in the is determined based on the speed and the ability to accelerate, an object box enveloping the predicted object box or at least laterally or horizontally surrounding the object box is generated, based on the ability to accelerate and taken as a basis in the evaluation of the criticality.

### Claim 23

A control unit for an operating assistance system of a working device, configured to:
(S1) obtain, at consecutive times, object boxes for an object in a field of view of the working device, or data characterizing the object boxes;
(S2) from the object boxes of the object for images recorded in succession or direct succession, determine: (i) an instantaneous change of scaling of an object box for the object, or variables derived from the instantaneous change of scaling the object box for the object, and (ii) an instantaneous lateral change of position of the object box for the object;
(S3) determine a predicted object box, predicted in for the future, from the instantaneous change of scaling of the object box for the object or the variables derived from the instantaneous change of scaling of the object box for the object, and from the instantaneous lateral change in position of the object box for the object;
(S4) determine and evaluate: (i) a position of the predicted object box and/or (ii) a ratio of a lateral extension of the predicted object box to a lateral extension of: (a) a covered field of view, and/or (b) the recorded images or a section of the recorded images; and
(S5) as a function of the result of the evaluation, determine: (i) whether or not the object which forms the basis of the predicted object box, is critical with regard to a possible collision, and/or (ii) an operating state of the working device being controlled or regulated.

### Claim 24 (depends on 23)

The control unit as recited in claim 23, wherein the working device is a vehicle.

### Claim 25

An operating assistance system for a working device, comprising:
a control unit for an operating assistance system of a working device, configured to:
(S1) obtain, at consecutive times, object boxes for an object in a field of view of the working device, or data characterizing the object boxes;
(S2) from the object boxes of the object for images recorded in succession or direct succession, determine: (i) an instantaneous change of scaling of an object box for the object, or variables derived from the instantaneous change of scaling the object box for the object, and (ii) an instantaneous lateral change of position of the object box for the object;
(S3) determine a predicted object box, predicted in for the future, from the instantaneous change of scaling of the object box for the object or the variables derived from the instantaneous change of scaling of the object box for the object, and from the instantaneous lateral change in position of the object box for the object;
(S4) determine and evaluate: (i) a position of the predicted object box and/or (ii) a ratio of a lateral extension of the predicted object box to a lateral extension of: (a) a covered field of view, and/or (b) the recorded images or a section of the recorded images; and
(S5) as a function of the result of the evaluation, determine: (i) whether or not the object which forms the basis of the predicted object box, is critical with regard to a possible collision, and/or (ii) an operating state of the working device being controlled or regulated; and
a camera configured to record the images.

### Claim 26 (depends on 25)

The operating assistance system as recited in claim 25, wherein the working device is a vehicle.

### Claim 27 (depends on 25)

The operating assistance system, as recited in claim 25, wherein the vehicle is a motor vehicle or a passenger car.

### Claim 28 (depends on 14)

The method as recited in claim 14, further comprising:
protecting pedestrians, and/or cyclists based on results of step (S5).

### Claim 29 (depends on 14)

The method as reciting in claim 14, further comprising:
using results of step (S5) for ACC and/or for collision avoidance systems.