# US 2017/0318235 A1 · Head-mounted displaying of magnified images locked on an object of interest

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0318235 A1
- Document ID: US20170318235A1
- Status: Published application
- Kind: A1
- Application number: 15/516,842
- Filed: 2015-10-07
- Published: 2017-11-02
- Earliest claimed: 2014-10-07
- Assignees: Elbit Systems Ltd.
- Applicants: Elbit Systems Ltd.
- Inventors: Ron Schneider; Abraham Zeitouny
- Web page: https://patentgrep.com/patent/US20170318235A1

## CPC

- G02B 27/017: {Head mounted}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0172: {characterised by optical features}
- G02B 30/34: Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. three-dimensional [3D] slide viewers
- G03B 15/14: for taking photographs during medical operations
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 13/344: with head-mounted left-right displays
- H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory}
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

System and method for presenting magnified images locked onto object of interest in operator environment. A camera disposed on head of operator captures images of scene, where camera moves in conjunction with head movements. A head tracker detects the operator LOS by detecting at least head orientation. A processor obtains designated coordinates of object of interest in scene, and determines relative angle between detected operator LOS and object. The processor determines coordinates of object in acquired images, and applies image processing for fine stabilization of images based on previous images so as to compensate for operator head movements. The processor rescales an image region surrounding object of interest, in accordance with at least one display parameter, to produce respective magnified image frames of object. A head-mounted display displays the magnified images to operator such that object of interest appears in a defined position on display regardless of operator head movements.

## Claims

### Claim at position 1

1-36. (canceled)

### Claim 37

A system for presenting magnified images locked onto an object of interest in the environment of an operator, the system comprising:
at least one camera, disposed on the head of said operator such that said camera moves in conjunction with said head of said operator, said camera configured to acquire a sequence of image frames of a scene;
a head tracker, configured to detect the line-of-sight (LOS) of said operator by detecting at least the orientation of the head of said operator;
a processor, coupled with said camera and with said head tracker, said processor configured to obtain designated coordinates of at least one object of interest in said scene, said processor further configured to determine the relative angle between the detected operator LOS and said object of interest, said processor further configured to determine the coordinates of said object of interest in the acquired image frames, said processor further configured to apply image processing for fine stabilization of said image frames based on at least one previous image frame so as to at least compensate for head movements of said operator, and said processor further configured to rescale a region surrounding said object of interest in said image frames, in accordance with at least one display parameter, to produce respective magnified image frames of said object of interest; and
a head-mounted display (HMD) worn by said operator and coupled with said processor, said HMD configured to display said magnified image frames to said operator such that said object of interest appears in a defined position on the display regardless of the head movements of said operator.

### Claim 38 (depends on 37)

The system of claim 37, wherein said processor is further configured to apply image or signal processing for coarse stabilization of said image frames, based on said detected operator LOS.

### Claim 39 (depends on 37)

The system of claim 37, wherein said processor is further configured to crop said region surrounding said object of interest in said image frame prior to rescaling.

### Claim 40 (depends on 37)

The system of claim 37, wherein said processor is further coupled with a secondary imaging device, configured to acquire secondary image frames including said object of interest,
said processor being further configured to determine the coordinates of said object of interest in said secondary image frames, and
said HMD being further configured to selectively display the secondary image frames to said operator.

### Claim 41 (depends on 37)

The system of claim 37, wherein said HMD displays said magnified image frame such that said object of interest appears superimposed at its true geolocation, in relation to the LOS of said operator.

### Claim 42 (depends on 37)

The system of claim 37, wherein said HMD is configured to display supplementary content overlaid onto the magnified image frames displayed by said HMD, wherein said supplementary content appears magnified in relation to the magnification of said object of interest in said magnified image frame.

### Claim 43 (depends on 37)

The system of claim 37, wherein said camera comprises a plurality of cameras, and wherein said HMD is configured to display a different image toward each eye of said operator, providing a stereoscopic view of said object of interest.

### Claim 44 (depends on 43)

The system of claim 43, wherein said processor is further configured to determine the distance between said operator and said object of interest, and to adjust said magnified image frame in accordance with the determined distance.

### Claim 45 (depends on 37)

The system of claim 37, wherein said processor is further configured to track multiple objects of interest in said acquired image frames, and to generate a plurality of magnified image frames respective of each of said objects of interest, wherein said HMD is further configured to selectively display at least one of said generated magnified image frames.

### Claim 46 (depends on 37)

The system of claim 37, further comprising an eye tracker, coupled with said processor, said eye tracker configured to detect the gaze direction of said operator, wherein said LOS of said operator is further determined in accordance with the detected gaze direction.

### Claim 47 (depends on 37)

The system of claim 37, wherein said HMD comprises a transparent display, configured to present a displayed image while allowing a see-through view of said scene in said operator FOV.

### Claim 48 (depends on 37)

The system of claim 37, wherein said HMD is further configured to provide at least one notification selected from the group consisting of:
a notification of an obstruction of said object of interest; and
a notification of said object of interest exceeding the FOV of said camera.

### Claim 49 (depends on 37)

The system of claim 37, further comprising an illuminator, configured to illuminate said object of interest in accordance with the LOS of said operator.

### Claim 50 (depends on 37)

An arrangement of a plurality of systems as claimed in claim 37, said systems being communicatively coupled with one another.

### Claim 51

A method for presenting magnified images locked onto an object of interest in the environment of an operator, the method comprising the procedures of:
acquiring a sequence of image frames of a scene, using at least one camera disposed on the head of said operator such that said camera moves in conjunction with said head of said operator;
detecting the line-of-sight (LOS) of said operator by detecting at least the orientation of the head of said operator;
designating coordinates of at least one object of interest in said scene; determining the relative angle between the detected operator LOS and said object of interest;
determining the coordinates of said object of interest in the acquired image frames;
applying image processing for fine stabilization of said image frames based on at least one previous image frame so as to at least compensate for head movements of said operator;
rescaling a region surrounding said object of interest in said image frames, in accordance with at least one display parameter, to produce respective magnified image frames of said object of interest; and
displaying said magnified image frames on a head-mounted display (HMD) worn by said operator, such that said object of interest appears in a defined position on the display regardless of the head movements of said operator.

### Claim 52 (depends on 51)

The method of claim 51, further comprising the procedure of applying image or signal processing for coarse stabilization of said image frames, based on the detected LOS of said operator.

### Claim 53 (depends on 51)

The method of claim 51, further comprising the procedure of cropping said region surrounding said object of interest in said image frame prior to rescaling.

### Claim 54 (depends on 51)

The method of claim 51, further comprising the procedures of:
acquiring secondary image frames including said object of interest, using at least one secondary imaging device; and
displaying the secondary image frames on said HMD.

### Claim 55 (depends on 51)

The method of claim 51, wherein said procedure of displaying said magnified image frames on a HMD comprises displaying a different image toward each eye of said operator, providing a stereoscopic view of said magnified object of interest.

### Claim 56 (depends on 51)

The method of claim 51, further comprising at least one procedure selected from the group consisting of:
providing a notification of an obstruction of said object of interest; and
providing a notification of said object of interest exceeding the FOV of said camera.
