# US 2019/0168609 A1 · Operator assistance vision system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0168609 A1 - Document ID: US20190168609A1 - Status: Published application - Kind: A1 - Application number: 15/828,571 - Filed: 2017-12-01 - Published: 2019-06-06 - Earliest claimed: 2017-12-01 - Assignees: Caterpillar Inc. - Applicants: Caterpillar Inc. - Inventors: Lawrence Andrew Mianzo; Michael Hoffelder - Web page: https://patentgrep.com/patent/US20190168609A1 ## CPC - E02F 9/264: {Sensors and their calibration for indicating the position of the work tool} - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60K 35/211: {producing three-dimensional [3D] effects, e.g. stereoscopic images} - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60W 50/08: Interaction between the driver and the control system - E02F 9/261: {Surveying the work-site to be treated} - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - H04N 7/183: {for receiving images from a single remote source} - B60K 35/22: Display screens - B60K 35/265: {Voice} - B60K 35/65: Instruments specially adapted for specific vehicle types or users, e.g. for left- or right-hand drive - B60K 2360/177: Augmented reality - B60K 2360/179: Distances to obstacles or vehicles - B60W 2050/146: {Display means} - B60Y 2400/92: Driver displays - G06T 2210/12: Bounding box - E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00) - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B60Y: INDEXING SCHEME RELATING TO ASPECTS CROSS-CUTTING VEHICLE TECHNOLOGY ## Abstract A vision system for assisting an operator of a machine is provided. An image capturing device, mounted on the machine, is configured to capture an image of an environment of the machine. The vision system includes a controller configured to apply an object detection process to detect an object in the image. The controller determines a bounding box comprising one or more pixels associated with the object. The controller determines a height and a range associated with the object based on the bounding box. The controller extracts the one or more pixels within the bounding box. The controller is further configured to generate a three-dimensional (3D) view comprising the object based on the image captured by the image capturing device. The controller reinserts the one or more pixels as a vertical pop-up element with respect to a ground plane in the 3D view. A display is provided to display the 3D view to the operator of the machine. ## Claims ### Claim 1 A vision system for assisting an operator of a machine, the vision system comprising: an image capturing device mounted on the machine, the image capturing device configured to capture an image of an environment of the machine; a controller communicably coupled to the image capturing device, the controller configured to: apply an object detection process to detect an object in the image; determine a bounding box comprising one or more pixels associated with the object; determine a height and a range associated with the object based on the bounding box; extract the one or more pixels within the bounding box; generate a three-dimensional (3D) view comprising the object based on the image captured by the image capturing device; and reinsert the one or more pixels as a vertical pop-up element with respect to a ground plane at a location of the object in the 3D view; and a display communicably coupled to the controller, the display configured to display the 3D view comprising the object to the operator of the machine. ### Claim 2 (depends on 1) The vision system of claim 1, wherein the controller is configured to apply sliding-window detection process to detect the object in the image. ### Claim 3 (depends on 1) The vision system of claim 1, wherein the controller is configured to determine the height and the range associated with the object based on the bounding box and calibration parameters of the image capturing device. ### Claim 4 (depends on 1) The vision system of claim 1, wherein the controller is configured to generate the 3D view by projecting the image onto a horizontal plane and/or a vertical plane. ### Claim 5 (depends on 1) The vision system of claim 1, wherein the controller is configured to display the height and the range associated with the object in the 3D view. ### Claim 6 (depends on 1) The vision system of claim 1, wherein the controller is further configured to alert the operator in case the object is detected at a distance less than a threshold distance from the machine. ### Claim 7 (depends on 6) The vision system of claim 6, wherein the controller is configured to alert the operator by highlighting the object displayed on the display. ### Claim 8 (depends on 1) The vision system of claim 1, wherein the display is provided with user interface controls to enable the operator to rotate the 3D view and allow visualization from different viewpoints. ### Claim 9 (depends on 1) The vision system of claim 1, wherein the display is communicably coupled to a remote system allowing remote monitoring of the environment of the machine. ### Claim 10 A method for assisting an operator of a machine based on an image of an environment of the machine captured by an image capturing device mounted on the machine, the method comprising: receiving, by a controller, the image of the environment of the machine; applying, by the controller, an object detection process to detect an object in the image; determining, by the controller, a bounding box comprising one or more pixels associated with the object; determining, by the controller, a height and a range associated with the object based on the bounding box; extracting, by the controller, the one or more pixels associated with the object from the image; generating, by the controller, a three-dimensional (3D) view comprising the object based on the image captured by the image capturing device; and reinserting, by the controller, the one or more pixels as a vertical pop-up element with respect to a ground plane at a location of the object in the 3D view. ### Claim 11 (depends on 10) The method of claim 10, wherein applying the object detection process comprises applying sliding-window detection process to detect the object in the image. ### Claim 12 (depends on 10) The method of claim 10, wherein determining the height and the range comprises determining the height and the range associated with the object based on the bounding box and calibration parameters of the image capturing device. ### Claim 13 (depends on 10) The method of claim 10, wherein generating the 3D view comprises projecting the image onto a horizontal plane and/or a vertical plane. ### Claim 14 (depends on 10) The method of claim 10 further comprising displaying the height and the range associated with the object in the 3D view. ### Claim 15 (depends on 10) The method of claim 10 further comprising alerting the operator in case the object is detected at a distance less than a threshold distance from the machine. ### Claim 16 (depends on 15) The method of claim 15, wherein alerting the operator comprises alerting the operator by highlighting the object displayed on the display. ### Claim 17 (depends on 10) The method of claim 10, wherein displaying the 3D view further comprises providing user interface controls to enable the operator to rotate the 3D view and allow visualization from different viewpoints. ### Claim 18 (depends on 10) The method of claim 10, wherein displaying the 3D view further comprises displaying the 3D view on a remote system allowing remote monitoring of the environment of the machine. ### Claim 19 (depends on 10) The method of claim 10, wherein generating the 3D view further comprises correcting the 3D view to remove distortions. ### Claim 20 A computer-program product for use in conjunction with an image capturing device and a display, wherein the image capturing device is configured to capture an image of an environment of a machine, the computer-program product comprising a non-transitory computer-readable storage medium having instructions for causing a processor to: apply an object detection process to detect an object in the image; determine a bounding box comprising one or more pixels associated with the object; determine a height and a range associated with the object based on the bounding box; extract the one or more pixels associated with the object from the image; generate a three-dimensional (3D) view comprising the object based on the image captured by the image capturing device; and reinsert the one or more pixels as a vertical pop-up element with respect to a ground plane at a location of the object in the 3D view.