# US 10,481,599 B2 · Methods and systems for controlling an object using a head-mounted display > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,481,599 B2 - Document ID: US10481599B2 - Status: Granted - Kind: B2 - Application number: 15/658,096 - Filed: 2017-07-24 - Published: 2019-01-24 - Granted: 2019-11-19 - Earliest claimed: 2017-07-24 - Assignees: Motorola Solutions, Inc. - Applicants: Motorola Solutions, Inc. - Inventors: Fabrice Saadoun; Dafna Langerman; Pavel Livshits - Web page: https://patentgrep.com/patent/US10481599B2 ## CPC - G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)} - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G05D 1/0033: {by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle} - G05D 1/101: {specially adapted for aircraft} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06F 3/0383: {Signal control means within the pointing device} - G02B 2027/0127: {comprising devices increasing the depth of field} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0141: {characterised by the informative content of the display} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G05D 1/0022: {characterised by the communication link (data switching networks in general H04L12/00)} - G06F 2203/0383: Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and systems for controlling an object using a head-mounted display. One head-mounted display includes a display projector. The head-mounted display further includes an eye tracking assembly configured to monitor an eye. The head-mounted display further includes an electronic processor coupled to the display projector and the eye tracking assembly. The electronic processor is configured to determine a depth of field measurement of the eye and determine that the depth of field measurement is greater than a predetermined distance. The electronic processor is further configured to control the display projector to project a virtual line in response to determining that the depth of field measurement is greater than the predetermined distance. The virtual line is projected such that a length of the virtual line corresponds to a range of measurable depth of field measurements of the eye. ## Claims ### Claim 1 A head-mounted display, comprising: a display projector; an eye tracking assembly configured to monitor an eye; and an electronic processor coupled to the display projector and the eye tracking assembly and configured to determine a depth of field measurement of the eye, determine that the depth of field measurement is greater than a predetermined distance, wherein the predetermined distance is within a range of measurable depth of field measurements of the eye and wherein determining that the depth of field measurement is greater than the predetermined distance indicates that the depth of field measurement is outside the range of measurable depth of field measurements, and control the display projector to project a virtual line in response to determining that the depth of field measurement is greater than the predetermined distance, wherein the virtual line is projected such that a length of the virtual line corresponds to the range of measurable depth of field measurements of the eye. ### Claim 2 (depends on 1) The head-mounted display of claim 1, wherein the electronic processor is further configured to determine an eye gaze location and a second depth of field measurement of the eye; determine that the eye is looking at and is focused on a portion of the virtual line based on the eye gaze location and the second depth of field measurement; and control a network interface to transmit a movement command to a movable device to move the movable device based on the eye gaze location and the second depth of field measurement. ### Claim 3 (depends on 2) The head-mounted display of claim 2, wherein the movable device moves forward or backward in response to the movement command. ### Claim 4 (depends on 2) The head-mounted display of claim 2, wherein the virtual line includes indicators that indicate a scale of distances based on a second distance between the head-mounted display and the movable device. ### Claim 5 (depends on 4) The head-mounted display of claim 4, wherein the electronic processor is further configured to update the indicators to change the scale of distances in response to the second distance changing. ### Claim 6 (depends on 1) The head-mounted display of claim 1, wherein the electronic processor is further configured to: determine a change in direction of a direction that the head-mounted display is facing; and control a network interface to transmit a movement command to a movable device to move the movable device in at least one of up, down, and laterally based on the change in direction. ### Claim 7 (depends on 1) The head-mounted display of claim 1, wherein the electronic processor is further configured to: determine an eye gaze location and a second depth of field measurement of the eye; determine that the eye is looking at or is focused on an area in a field of vision besides the virtual line based on the eye gaze location and the second depth of field measurement; and control a network interface to transmit a movement command to a movable device to move the movable device in at least one of up, down, and laterally based on at least one of the group consisting of the eye gaze location and the depth of field measurement. ### Claim 8 (depends on 1) The head-mounted display of claim 1, wherein the electronic processor is further configured to: determine an eye gaze location and a second depth of field measurement of the eye; determine that the eye is not looking at the virtual line based on the eye gaze location and that the second depth of field measurement is less than or equal to the predetermined distance; control the display projector to cease projecting the virtual line in response to determining that the eye is not looking at the virtual line and that the second depth of field measurement is less than or equal to the predetermined distance; and control a network interface to transmit a movement command to a movable device to move the movable device based on the eye gaze location and the second depth of field measurement. ### Claim 9 (depends on 1) The head-mounted display of claim 1, wherein the electronic processor is further configured to determine an eye gaze location and a second depth of field measurement of the eye; determine that the eye is looking at and is focused on a portion of the virtual line based on the eye gaze location and the second depth of field measurement; and control the display projector such that a virtual object appears to the eye to move based on the eye gaze location and the second depth of field measurement. ### Claim 10 (depends on 1) The head-mounted display of claim 1, wherein the virtual line is projected such that a first end of the virtual line that appears closest to the eye is projected to appear at approximately a minimum depth of field of the eye and a second end of the virtual line that appears farthest from the eye is projected to appear at approximately the predetermined distance. ### Claim 11 A method of controlling an object using a head-mounted display, the method comprising: determining, with an electronic processor, a depth of field measurement of an eye based on information received from an eye tracking assembly monitoring the eye; determining, with the electronic processor, that the depth of field measurement is greater than a predetermined distance, wherein the predetermined distance is within a range of measurable depth of field measurements of the eye and wherein determining that the depth of field measurement is greater than the predetermined distance indicates that the depth of field measurement is outside the range of measurable depth of field measurements; and controlling, with the electronic processor, a display projector to project a virtual line in response to determining that the depth of field measurement is greater than the predetermined distance, the virtual line being projected such that a length of the virtual line corresponds to the range of measurable depth of field measurements of the eye. ### Claim 12 (depends on 11) The method of claim 11, further comprising: determining, with the electronic processor, an eye gaze location and a second depth of field measurement of the eye; determining, with the electronic processor, that the eye is looking at and is focused on a portion of the virtual line based on the eye gaze location and the second depth of field measurement; and controlling, with the electronic processor, a network interface to transmit a movement command to a movable device to move the movable device based on the eye gaze location and the second depth of field measurement. ### Claim 13 (depends on 12) The method of claim 12, wherein the movable device moves forward or backward in response to the movement command. ### Claim 14 (depends on 12) The method of claim 12, wherein the virtual line includes indicators that indicate a scale of distances based on a second distance between the head-mounted display and the movable device. ### Claim 15 (depends on 14) The method of claim 14, further comprising, updating, with the electronic processor, the indicators to change the scale of distances in response to the second distance changing. ### Claim 16 (depends on 11) The method of claim 11, further comprising: determining, with the electronic processor, an eye gaze location and a second depth of field measurement of the eye; determining, with the electronic processor, that the eye is looking at or is focused on an area in a field of vision besides the virtual line based on the eye gaze location and the second depth of field measurement; and controlling, with the electronic processor, a network interface to transmit a movement command to a movable device to move the movable device in at least one of up, down, and laterally based on at least one of the group consisting of the eye gaze location and the depth of field measurement. ### Claim 17 (depends on 11) The method of claim 11, further comprising: determining, with the electronic processor, an eye gaze location and a second depth of field measurement of the eye; determining, with the electronic processor, that the eye is not looking at the virtual line based on the eye gaze location and that the second depth of field measurement is less than or equal to the predetermined distance; controlling, with the electronic processor, the display projector to cease projecting the virtual line in response to determining that the eye is not looking at the virtual line and that the second depth of field measurement is less than or equal to the predetermined distance; and controlling, with the electronic processor, a network interface to transmit a movement command to a movable device to move the movable device based on the eye gaze location and the second depth of field measurement. ### Claim 18 (depends on 11) The method of claim 11, further comprising: determining, with the electronic processor, an eye gaze location and a second depth of field measurement of the eye; determining, with the electronic processor, that the eye is looking at and is focused on a portion of the virtual line based on the eye gaze location and the second depth of field measurement; and controlling, with the electronic processor, the display projector such that a virtual object appears to the eye to move based on the eye gaze location and the second depth of field measurement. ### Claim 19 (depends on 11) The method of claim 11, wherein controlling the display projector to project the virtual line such that the length of the virtual line corresponds to the range of measurable depth of field measurements of the eye includes controlling the display projector to project the virtual line such that a first end of the virtual line that appears closest to the eye is projected to appear at approximately a minimum depth of field of the eye and a second end of the virtual line that appears farthest from the eye is projected to appear at approximately the predetermined distance. ### Claim 20 An apparatus, comprising: an eye tracking assembly configured to determine eye gaze of an eye and determine a depth of field measurement of the eye based on a lens distortion of the eye; a display projector coupled to the eye tracking assembly, wherein the display projector projects a virtual depth guidance element in response to the depth of field measurement exceeding a predetermined threshold, wherein the predetermined threshold is a predetermined distance within a range of measurable depth of field measurements of the eye, wherein the depth of field measurement exceeding the predetermined threshold indicates that the depth of field measurement is outside the range of measurable depth of field measurements of the eye, and wherein the virtual depth guidance element includes one or more virtual indicators each corresponding to a measurable depth of field measurement of the eye within the range of measurable depth of field measurements of the eye; and a network interface configured to transmit a movement command to a movable device, wherein the movement command corresponds to a second depth of field measurement of the eye determined when the eye is looking at and is focused on one of the one or more virtual indicators on the virtual depth guidance element.