# US 2023/0410562 A1 · Method and electronic device for determining user's hand in video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410562 A1
- Document ID: US20230410562A1
- Status: Published application
- Kind: A1
- Application number: 18/457,636
- Filed: 2023-08-29
- Published: 2023-12-21
- Earliest claimed: 2022-01-25
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Yevhenii YAKISHYN
- Web page: https://patentgrep.com/patent/US20230410562A1

## CPC

- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 7/194: involving foreground-background segmentation
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 40/168: {Feature extraction; Face representation}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30201: Face
- G06T 2207/30241: Trajectory
- G06T 2207/30244: Camera pose
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method is provided. The method includes obtaining a video including a plurality of frames, estimating a field of view (FoV) trajectory indicating movements of a FoV of the camera, by using the frames of the video, detecting feature points of one or more objects in each of the frames of the video, estimating object trajectories indicating movements of the one or more objects, based on the feature points, determining hand candidates of a user from among the one or more objects, based on the FoV trajectory of the camera and the object trajectories, obtaining a head movement trajectory indicating movements of a head of the user, determining a hand of the user, based on the head movement trajectory and the object trajectories corresponding to the hand candidates of the user, tracking the determined hand of the user in the video, and recognizing a gesture.

## Claims

### Claim 1

A method, performed by an electronic device, of determining hands of a user in a video, the method comprising:
obtaining a video comprising a plurality of frames, by using a camera;
estimating a field of view (FoV) trajectory indicating movements of a FoV of the camera, by using the frames of the video;
detecting feature points of one or more objects in each of the frames of the video;
estimating object trajectories indicating movements of the one or more objects, based on the feature points of the one or more objects;
determining hand candidates of a user from among the one or more objects, based on the FoV trajectory of the camera and the object trajectories;
obtaining a head movement trajectory indicating movements of a head of the user;
determining a hand of the user, based on the head movement trajectory and the object trajectories corresponding to the hand candidates of the user; and
tracking the determined hand of the user in the video and recognizing a gesture.

### Claim 2 (depends on 1)

The method of claim 1, further comprising obtaining foreground frames and background frames by extracting foreground objects from each of the frames of the video.

### Claim 3 (depends on 2)

The method of claim 2,
wherein the estimating of the FoV trajectory of the camera comprises:
obtaining FoV movement information related to movements of the FoV of the camera, by comparing pixels in the background frames, and
determining the FoV trajectory of the camera, based on the FoV movement information, and
wherein the detecting of the feature points of the one or more objects comprises extracting the feature points of the one or more objects from the foreground objects in the foreground frames.

### Claim 4 (depends on 1)

The method of claim 1, wherein the obtaining of the head movement trajectory comprises:
obtaining first sensor data from a first sensor located on the head of the user; and
generating the head movement trajectory, based on the first sensor data.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
obtaining second sensor data from a second sensor of a second electronic device located on the hand of the user; and
generating a hand movement trajectory indicating movements of the hand of the user, based on the second sensor data.

### Claim 6 (depends on 5)

The method of claim 5, wherein the determining of the hand of the user comprises determining the hand of the user by comparing the head movement trajectory with the hand movement trajectory generated based on the second sensor data.

### Claim 7 (depends on 1)

The method of claim 1, wherein the determining of the hand of the user comprises:
calculating correlation coefficients between the head movement trajectory and the object trajectories corresponding to the hand candidates of the user; and
determining the hand of the user from among the hand candidates of the user, based on whether the correlation coefficients are higher than or equal to a preset value.

### Claim 8 (depends on 3)

The method of claim 3,
wherein the detecting of the feature points of the one or more objects comprises detecting face feature points and hand feature points of a plurality of users from a plurality of foreground objects in the foreground frames,
wherein the obtaining of the head movement trajectory comprises obtaining head movement trajectories of the plurality of users, based on the face feature points of the plurality of users, and
wherein the determining of the hand of the user comprises determining hands of the plurality of users by separately associating hand candidates of the plurality of users with faces of the plurality of users.

### Claim 9 (depends on 1)

The method of claim 1, wherein the determining of the hand of the user comprises determining whether the hand of the user is a left hand or a right hand.

### Claim 10 (depends on 1)

The method of claim 1, further comprising executing a control command for controlling the electronic device, based on the recognized gesture.

### Claim 11

An electronic device for determining hands of a user in a video, the electronic device comprising:
at least one camera;
a memory configured to store one or more instructions; and
at least one processor configured to execute the one or more instructions stored in the memory to:
obtain a video comprising a plurality of frames, by using the at least one camera,
estimate a field of view (FoV) trajectory indicating movements of a FoV of the at least one camera, by using the frames of the video,
detect feature points of one or more objects in each of the frames of the video,
estimate object trajectories indicating movements of the one or more objects, based on the feature points of the one or more objects,
determine hand candidates of a user from among the one or more objects, based on the FoV trajectory of the camera and the object trajectories,
obtain a head movement trajectory indicating movements of a head of the user,
determine a hand of the user, based on the head movement trajectory and the object trajectories corresponding to the hand candidates of the user, and
track the determined hand of the user in the video and recognize a gesture.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to execute the one or more instructions to obtain foreground frames and background frames by extracting foreground objects from each of the frames of the video.

### Claim 13 (depends on 12)

The electronic device of claim 12, wherein the at least one processor is further configured to execute the one or more instructions to:
obtain FoV movement information related to movements of the FoV of the at least one camera, by comparing pixels in the background frames;
determine the FoV trajectory of the camera, based on the FoV movement information; and
extract the feature points of the one or more objects from the foreground objects in the foreground frames.

### Claim 14 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to execute the one or more instructions to:
obtain first sensor data from a first sensor located on the head of the user; and
generate the head movement trajectory, based on the first sensor data.

### Claim 15 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to execute the one or more instructions to:
obtain second sensor data from a second sensor of a second electronic device located on the hand of the user; and
generate a hand movement trajectory indicating movements of the hand of the user, based on the second sensor data.

### Claim 16 (depends on 15)

The electronic device of claim 15, wherein the at least one processor is further configured to execute the one or more instructions to determine the hand of the user by comparing the head movement trajectory with the hand movement trajectory generated based on the second sensor data.

### Claim 17 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to execute the one or more instructions to:
calculate correlation coefficients between the head movement trajectory and the object trajectories corresponding to the hand candidates of the user; and
determine the hand of the user from among the hand candidates of the user, based on whether the correlation coefficients are higher than or equal to a preset value.

### Claim 18 (depends on 13)

The electronic device of claim 13, wherein the at least one processor is further configured to execute the one or more instructions to:
detect face feature points and hand feature points of a plurality of users from a plurality of foreground objects in the foreground frames;
obtain head movement trajectories of the plurality of users, based on the face feature points of the plurality of users; and
determine hands of the plurality of users by separately associating hand candidates of the plurality of users with faces of the plurality of users.

### Claim 19 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to execute the one or more instructions to determine whether the hand of the user is a left hand or a right hand.

### Claim 20 (depends on 1)

A non-transitory computer-readable recording medium having recorded thereon a computer program for executing the method of claim 1, on a computer.
