# US 10,503,249 B2 · Method and apparatus for construction machine visualization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,249 B2 - Document ID: US10503249B2 - Status: Granted - Kind: B2 - Application number: 14/789,493 - Filed: 2015-07-01 - Published: 2016-01-07 - Granted: 2019-12-10 - Earliest claimed: 2014-07-03 - Assignees: Topcon Positioning Systems, Inc. - Applicants: Topcon Positioning Systems, Inc. - Inventors: Daniele Pagliani; Nicola Finardi; Richard Jackson; Ivan Di Federico - Web page: https://patentgrep.com/patent/US10503249B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06Q 10/063114: {Status monitoring or status determination for a person or group} - G06Q 50/08: Construction - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 40/193: {Preprocessing; Feature extraction} - H04N 9/31: Projection devices for colour picture display {, e.g. using electronic spatial light modulators [ESLM]} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system for displaying information to an operator of a machine comprises a head tracking system and a projection system. One system for displaying information uses a projection system and a see-through display to present two-dimensional images to an operator. One system uses a projection system and a see-through display to present three-dimensional images to an operator. One system uses a pair of smart glasses to display information to a user based a direction a user is looking. ## Claims ### Claim 1 A method comprising: determining a head position of a user with respect to a windshield of a construction machine; determining a direction of a gaze of the user based on the head position of the user with respect to the windshield; generating a representation of a construction site based on data received at a first time from sensors located on a construction machine; determining a difference between the representation of the construction site and a desired site plan; determining an operation to be performed based on the difference, the operation comprising movement of an implement of the construction machine based on a current position of the implement and a desired position of the implement; determining an image to be displayed to the user based on the head position of the user with respect to the windshield, the direction of the gaze of the user, the representation of the construction site, the desired site plan, and the operation to be performed, the image comprising an excavation area overlay, a volume overlay, and a completion overlay; displaying the image to the user; determining that the construction site has changed; updating the representation of the construction site in response to the determining that the construction site has changed, the updating based on data received at a second time from the sensors located on the construction machine to generate an updated representation; determining an updated image to be displayed to the user based on the direction of the user's gaze, the updated representation of the construction site, the desired site plan, and an additional operation to be performed based on a difference between the updated representation and the desired site plan; and displaying the updated image to the user. ### Claim 2 (depends on 1) The method of claim 1, wherein the direction of the gaze of the user is determined further based on a position of the user's eyes and a direction the user's eyes are looking. ### Claim 3 (depends on 1) The method of claim 1, the determining the image to be displayed to the user comprising: determining a difference between the representation of the construction site and a desired site plan; and generating a value pertaining to a percent of completion of the construction site based on the determining the difference between the representation of the construction site and the desired site plan, wherein the image comprises the percent of completion of the construction site. ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining that the direction of the gaze of the user has changed; determining a new image to be displayed to the user based on the determining that the direction of the gaze of the user has changed; and displaying the new image to the user. ### Claim 5 (depends on 1) The method of claim 1, wherein the displaying the image to the user comprises: transmitting the image to be displayed to the user to a projection system. ### Claim 6 (depends on 1) The method of claim 1, wherein the image displayed to the user comprises an overlay pertaining to the current position of the implement and the desired position of the implement. ### Claim 7 An apparatus comprising: a processor; and a memory to store computer program instructions, the computer program instructions when executed on the processor, cause the processor to perform operations comprising: determining a head position of a user with respect to a windshield of a construction machine; determining a direction of a gaze of the user based on the head position of the user with respect to the windshield; generating a representation of a construction site based on data received at a first time from sensors located on a construction machine; determining a difference between the representation of the construction site and a desired site plan; determining an operation to be performed based on the difference, the operation comprising movement of an implement of the construction machine based on a current position of the implement and a desired position of the implement; determining an image to be displayed to the user based on the head position of the user with respect to the windshield, the direction of the gaze of the user, the representation of the construction site, the desired site plan, and the operation to be performed, the image comprising an excavation area overlay, a volume overlay, and a completion overlay; displaying the image to the user; determining that the construction site has changed; updating the representation of the construction site in response to the determining that the construction site has changed, the updating based on data received at a second time from the sensors located on the construction machine to generate an updated representation; determining an updated image to be displayed to the user based on the direction of the user's gaze, the updated representation of the construction site, the desired site plan, and an additional operation to be performed based on a difference between the updated representation and the desired site plan; and displaying the updated image to the user. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein the direction of the gaze of the user is determined further based on a position of the user's eyes and a direction the user's eyes are looking. ### Claim 9 (depends on 7) The apparatus of claim 7, the determining the image to be displayed to the user comprising: determining a difference between the representation of the construction site and a desired site plan; and generating a value pertaining to a percent of completion of the construction site based on the determining the difference between the representation of the construction site and the desired site plan, wherein the image comprises the percent of completion of the construction site. ### Claim 10 (depends on 7) The apparatus of claim 7, the operations further comprising: determining that the direction of the a gaze of the user has changed; determining a new image to be displayed to the user based on the determining that the direction of the gaze of the user has changed; and displaying the new image to the user. ### Claim 11 (depends on 7) The apparatus of claim 7, wherein the image displayed to the user comprises an overlay pertaining to the current position of the implement and the desired position of the implement. ### Claim 12 A non-transitory computer readable medium storing computer program instructions, which, when executed on a processor, cause the processor to perform operations comprising: determining a head position of a user with respect to a windshield of a construction machine; determining a direction of a gaze of the user based on the head position of the user with respect to the windshield; generating a representation of a construction site based on data received at a first time from sensors located on a construction machine; determining a difference between the representation of the construction site and a desired site plan; determining an operation to be performed based on the difference, the operation comprising movement of an implement of the construction machine based on a current position of the implement and a desired position of the implement; determining an image to be displayed to the user based on the head position of the user with respect to the windshield, the direction of the gaze of the user, the representation of the construction site, the desired site plan, and the operation to be performed, the image comprising an excavation area overlay, a volume overlay, and a completion overlay; displaying the image to the user; determining that the construction site has changed; updating the representation of the construction site in response to the determining that the construction site has changed, the updating based on data received at a second time from the sensors located on the construction machine to generate an updated representation; determining an updated image to be displayed to the user based on the direction of the user's gaze, the updated representation of the construction site, the desired site plan, and an additional operation to be performed based on a difference between the updated representation and the desired site plan; and displaying the updated image to the user. ### Claim 13 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the direction of the gaze of the user is determined further based on a position of the user's eyes and a direction the user's eyes are looking. ### Claim 14 (depends on 12) The non-transitory computer readable medium of claim 12, the determining the image to be displayed to the user comprising: determining a difference between the representation of the construction site and a desired site plan; and generating a value pertaining to a percent of completion of the construction site based on the determining the difference between the representation of the construction site and the desired site plan, wherein the image comprises the percent of completion of the construction site. ### Claim 15 (depends on 12) The non-transitory computer readable medium of claim 12, wherein the image displayed to the user comprises an overlay pertaining to the current position of the implement and the desired position of the implement.