# US 2026/0039768 A1 · System, method, program and recording medium for locating projection point

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0039768 A1
- Document ID: US20260039768A1
- Status: Published application
- Kind: A1
- Application number: 18/900,550
- Filed: 2024-09-27
- Published: 2026-02-05
- Earliest claimed: 2024-08-05
- Assignees: Deltax Co., Ltd.
- Applicants: Deltax Co., Ltd.
- Inventors: Stephen KIM; Akmaljon A-Lijon Ugli PALVANOV; S M Nadim UDD; Thai Hoa HUYNH; Bunyodbek Fazliddin Ugli IBROKHIMOV; Manoj PADNEY
- Web page: https://patentgrep.com/patent/US20260039768A1

## CPC

- H04N 5/74: Projection arrangements for image reproduction, e.g. using eidophor
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/50: Depth or shape recovery
- G06T 7/73: using feature-based methods
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20132: Image cropping
- G06T 2207/30201: Face
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The projection point locating system includes: an image reception unit which receives an image including a target object to be located at a projection point and a face image of a user; a 3D eye tracking model management unit which generates and trains a 3D eye tracking model for optimally locating the projection point of the target object; an image processing unit which receives the 3D eye tracking model from the 3D eye tracking model management unit, and analyzes a face image of the user and estimates a position and a gaze direction of user's eyes, so as to process the target object such that the target objects appears at optimal coordinates aligned with the position and the gaze direction of the user's eyes; and an image output unit which displays the target object by projecting the target object at the optimal coordinates.

## Claims

### Claim 1

A projection point locating system comprising:
an image reception unit which receives an image including a target object to be located at a projection point from a first camera, and receives a face image of a user who observes the target object from a second camera;
a 3D eye tracking model management unit which generates and trains a 3D eye tracking model for optimally locating the projection point of the target object;
an image processing unit which receives the 3D eye tracking model from the 3D eye tracking model management unit, and analyzes a face image of the user who observes the target object and estimates a position and a gaze direction of user's eyes by using the 3D eye tracking model, so as to process the target object such that the target object appears at optimal coordinates aligned with the position and the gaze direction of the user's eyes; and
an image output unit which displays the target object, which is processed by the image processing unit, by projecting the target object at the optimal coordinates aligned with the position and the gaze direction of the user's eyes.

### Claim 2 (depends on 1)

The projection point locating system of claim 1, wherein the 3D eye tracking model is configured by integrating an eyeball tracking model for estimating pixel coordinates (x, y) of the user's eyes, a depth estimation model for estimating a depth (z) corresponding to the pixel coordinates (x, y) of the user's eyes, and a gaze estimation model for estimating the gaze direction of the user's eyes.

### Claim 3 (depends on 2)

The projection point locating system of claim 2, wherein the gaze direction of the user's eyes, which is estimated by the gaze estimation model, is a yaw direction and a pitch direction.

### Claim 4 (depends on 3)

The projection point locating system of claim 3, wherein the first camera and the second camera are general cameras that generate a 2D image, and images received by the image reception unit are 2D images that do not include depth information, and
the depth information corresponding to the 2D images is estimated by the depth estimation model.

### Claim 5 (depends on 4)

The projection point locating system of claim 4, wherein the image processing unit processes the target object to appear at the optimal coordinates, which are aligned with the position and the gaze direction of the user's eyes, by performing:
a first operation of detecting the face image of the user who observes the target object in an image transmitted from the image reception unit;
a second operation of estimating pixel coordinates (x1, y1) of a left eye of the user and pixel coordinates (x2, y2) of a right eye of the user by analyzing the face image of the user;
a third operation of estimating depths (z1, z2) corresponding to the pixel coordinates (x1, y1) of the left eye and the pixel coordinates (x2, y2) of the right eye by using the depth estimation model;
a fourth operation of cropping the face image of the user;
a fifth operation of estimating gaze directions (yaw, pitch) of the left and right eyes of the user by analyzing the cropped face image of the user; and
a sixth operation of estimating real world coordinates indicating the position and the gaze direction of the user's eyes by fusing the pixel coordinates (x1, y1) of the left eye, the pixel coordinates (x2, y2) of the right eye of the user estimated in the second operation, the depths (z1, z2) estimated in the third operation, and the gaze directions (yaw, pitch) of the left and right eyes of the user estimated in the fifth operation.

### Claim 6 (depends on 5)

The projection point locating system of claim 5, wherein the second operation and the fourth operation are performed in parallel.

### Claim 7 (depends on 5)

The projection point locating system of claim 5, wherein the gaze estimation model is configured to estimate the gaze direction of the user's eyes based on a gaze tracking network architecture, and
the gaze tracking network architecture includes:
a backbone network configured to extract features related to the position and the gaze direction of the user's eyes from the cropped image;
a first fully connected (FC) layer unit configured to receive the features from the backbone network to calculate a weighted sum for estimating a yaw value for the features and apply non-linear transformation through an activation function;
a second FC layer unit configured to receive the features from the backbone network to calculate a weighted sum for estimating a pitch value for the features and apply non-linear transformation through an activation function;
a yaw gaze estimation unit configured to convert an output from the first FC layer unit into a probability through a first softmax, and estimate the yaw value through a first composite loss function in which a first cross entropy loss function and a first regression loss function are combined; and
a pitch gaze estimation unit configured to convert an output from the second FC layer unit into a probability through a second softmax, and estimate the pitch value through a second composite loss function in which a second cross entropy loss function and a second regression loss function are combined.

### Claim 8 (depends on 7)

The projection point locating system of claim 7, wherein the first regression loss function and the second regression loss function are mean square error (MSE) functions.

### Claim 9 (depends on 8)

The projection point locating system of claim 8, wherein the yaw gaze estimation unit is configured to:
calculate a bin classification loss between possibilities output through the first softmax and target bin labels based on the first cross entropy loss function;
acquire a yaw expectation value based on the possibilities output through the first softmax; and
estimate the yaw value by calculating a mean square error for the acquired yaw expectation value based on the first regression loss function and adding the mean square error to the bin classification loss calculated based on the first cross entropy loss function.

### Claim 10 (depends on 9)

The projection point locating system of claim 9, wherein the pitch gaze estimation unit is configured to:
calculate a bin classification loss between possibilities output through the second softmax and target bin labels based on the second cross entropy loss function;
acquire a pitch expectation value based on the possibilities output through the second softmax; and
estimate the pitch value by calculating a mean square error for the acquired pitch expectation value based on the second regression loss function and adding the mean square error to the bin classification loss calculated based on the second cross entropy loss function.

### Claim 11 (depends on 7)

The projection point locating system of claim 7, wherein the backbone network is RestNet-50.

### Claim 12

A projection point locating method for analyzing a face image of a user who observes a target object and estimating a position and a gaze direction of user's eyes so that the target object appears at optimal coordinates aligned with the position and the gaze direction of the user's eyes, the projection point locating method comprising:
a first step of detecting the face image of the user who observes the target object in an image received from a camera;
a second step of estimating pixel coordinates (x1, y1) of a left eye of the user and pixel coordinates (x2, y2) of a right eye of the user by analyzing the face image of the user;
a third step of estimating depths (z1, z2) corresponding to the pixel coordinates (x1, y1) of the left eye and the pixel coordinates (x2, y2) of the right eye by using a depth estimation model;
a fourth step of cropping the face image of the user;
a fifth step of estimating gaze directions (yaw, pitch) of the left and right eyes of the user by analyzing the cropped face image of the user; and
a sixth step of estimating real world coordinates indicating the position and the gaze direction of the user's eyes by fusing the pixel coordinates (x1, y1) of the left eye, the pixel coordinates (x2, y2) of the right eye of the user estimated in the second step, the depths (z1, z2) estimated in the third step, and the gaze directions (yaw, pitch) of the left and right eyes of the user estimated in the fifth step.

### Claim 13 (depends on 12)

The projection point locating method of claim 12, wherein the second step and the fourth step are performed in parallel.

### Claim 14 (depends on 12)

A computer program comprising instructions for performing the method of claim 12.

### Claim 15 (depends on 13)

A computer program comprising instructions for performing the method of claim 13.

### Claim 16 (depends on 12)

A computer-readable recording medium which stores a program including instructions for performing the method of claim 12.

### Claim 17 (depends on 13)

A computer-readable recording medium which stores a program including instructions for performing the method of claim 13.
