# US 11,308,326 B2 · Electronic apparatus and controlling method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,326 B2 - Document ID: US11308326B2 - Status: Granted - Kind: B2 - Application number: 16/686,570 - Filed: 2019-11-18 - Published: 2020-06-18 - Granted: 2022-04-19 - Earliest claimed: 2018-12-18 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Taehwan Yoo; Kyungsu Kim; Ganghui Lee; Hyungmin Lee - Web page: https://patentgrep.com/patent/US11308326B2 ## CPC - G09B 5/065: {Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/20: in augmented reality scenes - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G09B 19/0092: {Nutrition} - G06V 20/44: {Event detection} - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An electronic apparatus and a controlling method thereof are provided. The controlling method according to an embodiment includes acquiring contents and storing the contents in a memory, identifying a first section of the contents related to a first behavior by analyzing the stored contents, outputting contents of the identified first section, acquiring a user image captured while the contents of the first section is output, identifying whether the first behavior is completed by analyzing the user image, and based on identifying that the user has completed the first behavior, outputting contents of a second section, the second section being a next section of the first section. ## Claims ### Claim 1 A method of controlling an electronic apparatus, comprising: acquiring contents and storing the contents in a memory; identifying a first section of the contents related to a first behavior by analyzing the stored contents; outputting contents of the identified first section; acquiring a user image captured while the contents of the identified first section is output; identifying whether the first behavior is completed by analyzing the user image; and based on identifying that a user has completed the first behavior, outputting contents of a second section of the contents, the second section being a next section of the identified first section, wherein the identifying the first section comprises: inputting an image frame of an image included in the contents to a trained Artificial Intelligence (AI) model and acquiring a feature value of the input image frame; performing scene understanding related to the first behavior based on the acquired feature value; and identifying the first section related to the first behavior based on the performed scene understanding. ### Claim 2 (depends on 1) The method as claimed in claim 1, wherein the acquiring the user image comprises: inputting an image frame of the user image to the trained AI model and acquiring a feature value of the input image frame of the user image; and performing scene understanding regarding user behavior included in the user image based on the acquired feature value of the input image frame of the user image. ### Claim 3 (depends on 2) The method as claimed in claim 2, wherein the identifying whether the first behavior has been completed comprises: comparing a feature value of an image frame of an image of the first section with a feature value of an image frame of the user image. ### Claim 4 (depends on 2) The method as claimed in claim 2, wherein the identifying whether the first behavior has been completed comprises: identifying whether the first behavior has been completed based on scene understanding related to the first behavior and scene understanding regarding the user behavior. ### Claim 5 (depends on 4) The method as claimed in claim 4, wherein the outputting the contents of the identified first section comprises, based on identifying that the first behavior has not been completed, stopping reproduction of an image of the first section or repeatedly reproducing the image of the first section. ### Claim 6 (depends on 5) The method as claimed in claim 5, wherein the outputting the contents of the identified first section comprises, based on a behavior different from the first behavior being performed for a predetermined threshold time or longer while the reproduction of the image of the first section is stopped or the image of the first section is repeatedly reproduced, outputting an image of the second section. ### Claim 7 (depends on 2) The method as claimed in claim 2, further comprising: acquiring and displaying a sentence related to an image of the first section and the user image, respectively, wherein the displaying comprises acquiring a first sentence by performing scene understanding regarding the first behavior included in an image of the first section, acquiring a second sentence by performing scene understanding regarding the user behavior included in the user image, and displaying the acquired first and second sentences in an area of a display in which an image of the first section is displayed. ### Claim 8 (depends on 7) The method as claimed in claim 7, wherein the identifying whether the first behavior has been completed comprises identifying whether the first behavior has been completed based on a similarity level of the first and second sentences. ### Claim 9 (depends on 8) The method as claimed in claim 8, wherein the outputting comprises, based on identifying that the first behavior has not been completed, outputting a message informing that a behavior different from the first behavior is being performed through at least one of an image or audio. ### Claim 10 (depends on 1) The method as claimed in claim 1, wherein the memory includes a buffer and a main memory, and wherein the storing comprises storing the contents in the buffer, and based on a storage capacity of the buffer being insufficient, storing remaining contents which are not stored in the buffer in the main memory. ### Claim 11 An electronic apparatus, comprising: a photographing part comprising image capturing circuitry; a memory configured to acquire contents and store the contents; an outputter comprising output circuitry configured to output the contents; and a processor configured to control the electronic apparatus to: identify a first section of the contents related to a first behavior by analyzing the stored contents, and control the outputter to output contents of the identified first section, acquire a user image captured while the contents of the identified first section is being output, and identify whether the first behavior is completed by analyzing the user image, and based on identifying that a user has completed the first behavior, control the outputter to output contents of a second section of the contents, the second section being a next section of the identified first section, wherein the processor is configured to control the electronic apparatus to: input an image frame of an image included in the contents to a trained Artificial Intelligence (AI) model to acquire a feature value of the input image frame, perform scene understanding related to the first behavior based on the acquired feature value, and identify the first section of the contents related to the first behavior based on the performed scene understanding. ### Claim 12 (depends on 11) The electronic apparatus as claimed in claim 11, wherein the processor is configured to control the electronic apparatus to: input an image frame of the user image to the trained AI model to acquire a feature value of the input image frame of the user image, and perform scene understanding regarding user behavior included in the user image based on the acquired feature value of the input image frame of the user image. ### Claim 13 (depends on 12) The electronic apparatus as claimed in claim 12, wherein the processor is configured to control the electronic apparatus to: identify whether the first behavior has been completed by comparing a feature value of an image frame of an image of the first section with a feature value of an image frame of the user image. ### Claim 14 (depends on 12) The electronic apparatus as claimed in claim 12, wherein the processor is configured to control the electronic apparatus to: identify whether the first behavior has been completed based on scene understanding related to the first behavior and scene understanding regarding the user behavior. ### Claim 15 (depends on 14) The electronic apparatus as claimed in claim 14, wherein the processor is configured to control the outputter to stop reproduction of an image of the first section or to repeatedly reproduce the image of the first section. ### Claim 16 (depends on 12) The electronic apparatus as claimed in claim 12, wherein the processor is configured to control the electronic apparatus to: acquire a first sentence by performing scene understanding regarding the first behavior included in an image of the first section, acquire a second sentence by performing scene understanding regarding the user behavior included in the user image, and control the outputter to display the acquired first and second sentences in an area of a display in which an image of the first section is displayed. ### Claim 17 (depends on 16) The electronic apparatus as claimed in claim 16, wherein the processor is configured to control the electronic apparatus to identify whether the first behavior has been completed based on a similarity level of the first sentence and the second sentence. ### Claim 18 (depends on 17) The electronic apparatus as claimed in claim 17, wherein the processor, based on identifying that the first behavior has not been completed, is configured to control the outputter to output a message informing that a behavior different from the first behavior is being performed through at least one of an image or audio.