# US 12,223,769 B2 · Electronic device and operating method of electronic device for correcting error during gaze direction recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,769 B2 - Document ID: US12223769B2 - Status: Granted - Kind: B2 - Application number: 17/718,905 - Filed: 2022-04-12 - Published: 2022-12-29 - Granted: 2025-02-11 - Earliest claimed: 2021-06-29 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Cheoljun Lee - Web page: https://patentgrep.com/patent/US12223769B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) - G06F 3/012: {Head tracking input arrangements} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06T 7/73: using feature-based methods - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/164: {using holistic features} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Disclosed is an electronic device for correcting an error during recognition of a gaze direction. The electronic device includes: a camera module including a camera configured to generate an image frame, a memory, and a processor, and the processor may be configured to: perform face recognition with respect to the image frame, perform extraction of a feature point of the face in response to the face being recognized during the face recognition, recognize a face direction of the face in response to the feature point of the face being extracted, generate gaze direction information by recognizing a gaze direction of the face in response to the recognition of the face direction, generate filtered gaze direction information by performing filtering with respect to the gaze direction information in response to the gaze direction being recognized, and store the filtered gaze direction information in the memory. ## Claims ### Claim 1 An electronic device comprising: a camera module comprising a camera and configured to generate an image frame; memory; and at least one processor, wherein the at least one processor is configured to: perform face recognition of a face with respect to the image frame; perform extraction of a feature point of the face in response to the face being recognized during the face recognition; recognize a face direction of the face in response to the feature point of the face being extracted; generate gaze direction information by recognizing a gaze direction of the face in response to the recognition of the face direction, wherein the gaze direction information comprises consecutive image frames corresponding to the gaze direction of the face; generate filtered gaze direction information by performing filtering that restricts a degree of change of the gaze direction among the consecutive image frames in response to the recognition of the gaze direction; and store the filtered gaze direction information in the memory, wherein the filtering restricts the degree of change of the gaze direction based on the gaze direction information recognized from the image frame, previously filtered gaze direction information recognized from a previous image frame, and a filtering coefficient stored in the memory, and wherein the filtering coefficient is adjusted dynamically. ### Claim 2 (depends on 1) The electronic device of claim 1, wherein the filtering comprises a low pass filter (LPF). ### Claim 3 (depends on 1) The electronic device of claim 1, wherein the at least one processor is configured to recognize the face direction based on a recognition module generated based on appearance-based deep learning and executed by the at least one processor. ### Claim 4 (depends on 1) The electronic device of claim 1, wherein the at least one processor comprises a gaze direction filtering module configured to perform the filtering with respect to the gaze direction information, and wherein the gaze direction filtering module comprises: a delay configured to receive the previously filtered gaze direction information and output the previously filtered gaze direction information with a delay; a first multiplier configured to receive the output of the delay and a first filtering coefficient and output a multiplication of the output of the delay and the first filtering coefficient; a second multiplier configured to receive the gaze direction information of the image frame and a second filtering coefficient and output a multiplication of the gaze direction information of the image frame and the second filtering coefficient; and an adder configured to receive the output of the first multiplier and the output of the second multiplier, and output a sum of the output of the first multiplier and the output of the second multiplier as the filtered gaze direction information of the image frame. ### Claim 5 (depends on 4) The electronic device of claim 4, wherein the first filtering coefficient and the second filtering coefficient are adjusted dynamically. ### Claim 6 (depends on 1) The electronic device of claim 1, wherein the filtering is configured to correct an error that occurs from the gaze direction information based on actions that cause an error, and wherein the actions that cause an error comprise accidental movements that occur by eye blinking, coughs, or tic disorder. ### Claim 7 (depends on 1) The electronic device of claim 1, further comprising: a communication module comprising communication circuitry configured to receive an image from an external device; a display module comprising a display configured to display the image through the communication module; and an input module comprising input circuitry configured to generate a signal from a user, wherein, in response to a first signal being generated from the user, the at least one processor is configured to identify an object on the image corresponding to the filtered gaze direction information. ### Claim 8 (depends on 7) The electronic device of claim 7, wherein, in response to the object on the image corresponding to the filtered gaze direction information being identified, the at least one processor is configured to control the display to display information about the object on the image through the display module. ### Claim 9 (depends on 8) The electronic device of claim 8, further comprising: a second communication module comprising communication circuitry configured to receive information about an object on the image from an external second device, and provide the information about the object on the image to the at least one processor. ### Claim 10 (depends on 9) The electronic device of claim 9, wherein, in response to a second signal being generated from the user, the at least one processor is configured to execute at least one of a first additional service through the second communication module and a second additional service not through the second communication module. ### Claim 11 (depends on 1) The electronic device of claim 1, further comprising: a second camera module comprising a camera configured to generate an image; a display module comprising a display configured to display the image generated by the second camera module; and an input module comprising input circuitry configured to generate a signal from a user, wherein, in response to a first signal being generated from the user, the at least one processor is configured to identify an object on the image corresponding to the filtered gaze direction information. ### Claim 12 (depends on 11) The electronic device of claim 11, further comprising: a communication module comprising communication circuitry configured to communicate with an external device; and an acoustic output module comprising acoustic output circuitry configured to output sound, wherein, in response to an object on the image corresponding to the filtered gaze direction information being identified, the at least one processor is configured to: receive information of the object through the communication module and output information of the object through at least one of the acoustic output module or the display module. ### Claim 13 (depends on 11) The electronic device of claim 11, wherein, in response to an object on the image corresponding to the filtered gaze direction information being identified, the at least one processor is configured to control the second camera module to move a focus to the object on the image. ### Claim 14 A method of operating an electronic device comprising at least one processor, a camera module, and a memory, the method comprising: generating an image frame by the camera module; performing face recognition of a face with respect to the image frame by the at least one processor; performing extraction of a feature point of the face by the at least one processor in response to the face being recognized during the face recognition; recognizing a face direction of the face by the at least one processor in response to the feature point of the face being extracted; generating gaze direction information by recognizing a gaze direction of the face by the at least one processor in response to the recognition of the face direction, wherein the gaze direction information comprises consecutive image frames corresponding to the gaze direction of the face; generating filtered gaze direction information by performing filtering that restricts a degree of change of the gaze direction among the consecutive image frames in response to the recognition of the gaze direction; and storing the filtered gaze direction information in the memory, wherein the filtering restricts the degree of change of the gaze direction based on the gaze direction information recognized from the image frame, previously filtered gaze direction information recognized from a previous image frame, and a filtering coefficient stored in the memory, and wherein the filtering coefficient is adjusted dynamically.