# US 11,908,076 B2 · Display system and display method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,076 B2 - Document ID: US11908076B2 - Status: Granted - Kind: B2 - Application number: 17/605,272 - Filed: 2020-05-26 - Published: 2022-07-21 - Granted: 2024-02-20 - Earliest claimed: 2019-05-31 - Assignees: Komatsu Ltd. - Applicants: Komatsu Ltd. - Inventors: Kazuhisa Takahama - Web page: https://patentgrep.com/patent/US11908076B2 ## CPC - G06T 17/05: Geographic models - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - E02F 3/7618: {with the scraper blade adjustable relative to the pivoting arms about a horizontal axis} - E02F 9/205: {Remotely operated machines, e.g. unmanned vehicles (E02F3/8866 takes precedence)} - E02F 9/2228: {including an electronic controller} - E02F 9/261: {Surveying the work-site to be treated} - G06T 1/00: General purpose image data processing - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00) ## Abstract A display system for a work machine includes an acquisition unit configured to acquire three-dimensional data of a plurality of measurement points, a conversion unit configured to convert the three-dimensional data into a vehicle body coordinate system, an image generation unit configured to generate a reference image representing a three-dimensional shape of a terrain based on the three-dimensional data converted into the vehicle body coordinate system, and a display processing unit configured to display the reference image on a captured image in a superimposed manner. The image generation unit decides a display form of the reference image at a position of the reference image corresponding to each measurement point depending on a distance of the measurement point in a normal direction with respect to a ground surface of the work machine. ## Claims ### Claim 1 A display system for a work machine, the display system comprising: a processor and a storage unit, wherein the processor has functional elements including: an acquisition unit configured to acquire three-dimensional data of an actual terrain of a plurality of measurement points measured by a distance detection device mounted in the work machine; a conversion unit configured to convert the three-dimensional data into a vehicle body coordinate system defined in association with a vehicle body of the work machine; an image generation unit configured to generate a reference image representing a three-dimensional shape of the actual terrain based on the three-dimensional data converted into the vehicle body coordinate system; and a display processing unit configured to display the reference image on an image captured by an imaging device in a superimposed manner, wherein the image generation unit decides a display form of the reference image at a position of the reference image corresponding to each measurement point depending on a distance of the measurement point in a normal direction with respect to a ground surface of the work machine, in a target on which the work machine travels or performs work. ### Claim 2 (depends on 1) The display system according to claim 1, wherein the reference image includes a plurality of line images. ### Claim 3 (depends on 2) The display system according to claim 2, wherein the reference image includes a lattice by the plurality of line images. ### Claim 4 (depends on 3) The display system according to claim 3, wherein the display form is coloring of the reference image. ### Claim 5 (depends on 3) The display system according to claim 3, wherein the display form is a thickness of a line of the line image. ### Claim 6 (depends on 1) The display system according to claim 1, wherein the display form is coloring of the reference image. ### Claim 7 (depends on 1) The display system according to claim 1, wherein the image generation unit differs the display form from a case where the work machine is grounded on a horizontal plane to a case where the work machine is grounded on a slope surface. ### Claim 8 (depends on 2) The display system according to claim 2, wherein the display form is a thickness of a line of the line image. ### Claim 9 (depends on 2) The display system according to claim 2, wherein the display form is coloring of the reference image. ### Claim 10 A display method comprising the steps of: acquiring three-dimensional data of an actual terrain of a plurality of measurement points measured by a distance detection device mounted in a work machine; converting the three-dimensional data into a vehicle body coordinate system defined in association with a vehicle body of the work machine; generating a reference image representing a three-dimensional shape of the actual terrain based on the three-dimensional data converted into the vehicle body coordinate system; and displaying, on a display device, the reference image on an image captured by an imaging device in a superimposed manner, wherein the step of generating the reference image includes a step of deciding a display form of the reference image at a position of the reference image corresponding to each measurement point depending on a distance of the measurement point in a normal direction with respect to a ground surface of the work machine, in a target on which the work machine travels or performs work. ### Claim 11 (depends on 1) A remote operation system for a work machine comprising: the display system according to claim 1; a detection device configured to detect a posture of the work machine; a display device disposed at a location away from the work vehicle and configured to display an image by processing of the display processing unit; and an operation device disposed at a location away from the work machine and configured to remotely operate the work machine. ### Claim 12 (depends on 2) A remote operation system for a work machine comprising: the display system according to claim 2; a detection device configured to detect a posture of the work machine; a display device disposed at a location away from the work vehicle and configured to display an image by processing of the display processing unit; and an operation device disposed at a location away from the work machine and configured to remotely operate the work machine. ### Claim 13 (depends on 3) A remote operation system for a work machine comprising: the display system according to claim 3; a detection device configured to detect a posture of the work machine; a display device disposed at a location away from the work vehicle and configured to display an image by processing of the display processing unit; and an operation device disposed at a location away from the work machine and configured to remotely operate the work machine. ### Claim 14 (depends on 6) A remote operation system for a work machine comprising: the display system according to claim 6; a detection device configured to detect a posture of the work machine; a display device disposed at a location away from the work vehicle and configured to display an image by processing of the display processing unit; and an operation device disposed at a location away from the work machine and configured to remotely operate the work machine. ### Claim 15 (depends on 8) A remote operation system for a work machine comprising: the display system according to claim 8; a detection device configured to detect a posture of the work machine; a display device disposed at a location away from the work vehicle and configured to display an image by processing of the display processing unit; and an operation device disposed at a location away from the work machine and configured to remotely operate the work machine.