# US 10,872,444 B2 · Display apparatus and control method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,444 B2 - Document ID: US10872444B2 - Status: Granted - Kind: B2 - Application number: 16/550,075 - Filed: 2019-08-23 - Published: 2020-03-26 - Granted: 2020-12-22 - Earliest claimed: 2018-09-21 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Jihye Ha; Gwanhyung Kim; Sangho Kim - Web page: https://patentgrep.com/patent/US10872444B2 ## CPC - G06F 3/03545: {Pens or stylus} - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06F 3/04166: {Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving (Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184)} - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06T 11/23: using straight lines or curves - G06V 30/1423: {the instrument generating sequences of position coordinates corresponding to handwriting (preprocessing or recognising digital ink G06V30/32)} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - H04N 23/60: Control of cameras or camera modules - H04N 23/80: Camera processing pipelines; Components thereof - G06F 2203/04808: Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A display apparatus is disclosed. The display apparatus includes a touch display, a camera, and a processor configured to control the camera to capture an image including the touch display, based on an object being identified from an image captured through the camera, obtain a first position of the object on the touch display based on the image, identify whether the first position is within a predetermined distance from a second position where the touch input is sensed based on a touch input of the object being sensed on the touch display, and based on the first position being within the predetermined distance from the second position, control the touch display to display a touch trajectory corresponding to the touch input based on the identified object. ## Claims ### Claim 1 A display apparatus comprising: a touch display; a camera; and a processor configured to: control the camera to capture an image including the touch display, based on an object being identified from an image captured through the camera, obtain a first position of the object on the touch display based on the image, identify whether the first position is within a predetermined distance from a second position where a touch input is sensed based on the touch input of the object being sensed on the touch display, and based on the first position being within the predetermined distance from the second position, control the touch display to display a touch trajectory corresponding to the touch input based on the identified object. ### Claim 2 (depends on 1) The display apparatus of claim 1, wherein the processor is configured to: identify information of at least one of a color, a size, or a shape of the object from the image, and determine at least one of the color, size, or shape of the touch trajectory based on the identified information. ### Claim 3 (depends on 1) The display apparatus of claim 1, wherein the processor is configured to: identify a region where a motion is present from a plurality of consecutive images captured through the camera, and identify a remaining region after removing a region corresponding to at least one of a predetermined shape or a predetermined color from the region where the motion is present as the object. ### Claim 4 (depends on 3) The display apparatus of claim 3, further comprising: a storage configured to store color information by color temperatures, and wherein the processor is configured to: identify a color temperature of the image based on a bezel color of the touch display included in the image, filter a region where the motion is present based on the color information corresponding to the identified color temperature, and identify at least one of the color, size, or shape of the object based on the filtering. ### Claim 5 (depends on 1) The display apparatus of claim 1, wherein the processor is configured to: identify a first region corresponding to the touch display and a second region corresponding to the object from the image, convert the first region to correspond to a shape of the touch display, and change a position of the second region to correspond to the converted first region, and identify the first position based on a changed position of the second region with respect to the converted first region. ### Claim 6 (depends on 1) The display apparatus of claim 1, further comprising: a storage, wherein the processor is configured to: store information on at least one of the color, size, or shape of the object in the storage based on the object being identified from the image, and control the touch display to display a touch trajectory corresponding to the touch input based on information stored in the storage based on a touch input of the object being sensed through the touch display. ### Claim 7 (depends on 1) The display apparatus of claim 1, wherein the processor is configured to: identify a function of the object based on at least one of the color, size, or shape of the object, and control the touch display to display the touch trajectory based on the identified function. ### Claim 8 (depends on 1) The display apparatus of claim 1, wherein the processor is configured to control the touch display to display a touch trajectory corresponding to a shape of the object and the touch input. ### Claim 9 (depends on 1) The display apparatus of claim 1, wherein the processor, based on the first position exceeding the predetermined distance from the second position, is configured to control the touch display to not display the touch trajectory. ### Claim 10 (depends on 1) The display apparatus of claim 1, wherein the camera is positioned on one side of the touch display and is configured to capture a screen of the touch display. ### Claim 11 A method of controlling a display apparatus, the method comprising: capturing an image including a touch display provided in the display apparatus; based on an object being identified from the captured image, obtaining a first position of the object on the touch display based on the image; identifying whether the first position is within a predetermined distance from a second position where a touch input is sensed based on a touch input of the object being sensed on the touch display; and based on the first position being within the predetermined distance from the second position, displaying a touch trajectory corresponding to the touch input based on the identified object. ### Claim 12 (depends on 11) The method of claim 11, wherein the displaying comprises identifying information of at least one of a color, a size, or a shape of the object from the image, and determining at least one of the color, size, or shape of the touch trajectory based on the identified information. ### Claim 13 (depends on 11) The method of claim 11, wherein the obtaining comprises: identifying a region where a motion is present from a plurality of consecutive images captured through the camera; and identifying a remaining region after removing a region corresponding to at least one of a predetermined shape or a predetermined color from the region where the motion is present as the object. ### Claim 14 (depends on 13) The method of claim 13, wherein the identifying as the object comprises: identifying a color temperature of the image based on a bezel color of the touch display included in the image; filtering a region where the motion is present based on the color information corresponding to the identified color temperature; and identifying at least one of the color, size, or shape of the object based on the filtering. ### Claim 15 (depends on 11) The method of claim 11, wherein the obtaining comprises: identifying a first region corresponding to the touch display and a second region corresponding to the object from the image; converting the first region to correspond to a shape of the touch display, and changing a position of the second region to correspond to the converted first region; and identifying the first position based on a changed position of the second region with respect to the converted first region. ### Claim 16 (depends on 11) The method of claim 11, further comprising: storing information on at least one of the color, size, or shape of the object in the storage based on the object being identified from the image, wherein the displaying comprises displaying a touch trajectory corresponding to the touch input based on the stored information. ### Claim 17 (depends on 11) The method of claim 11, wherein the displaying comprises identifying a function of the object based on at least one of the color, size, or shape of the object, and displaying the touch trajectory based on the identified function. ### Claim 18 (depends on 11) The method of claim 11, wherein the displaying comprises displaying a touch trajectory corresponding to a shape of the object and the touch input. ### Claim 19 (depends on 11) The method of claim 11, wherein the displaying comprises not displaying the touch trajectory based on the first position exceeding the predetermined distance from the second position. ### Claim 20 (depends on 11) The method of claim 11, wherein the capturing comprises capturing a screen of the touch display by a camera positioned on one side of the touch display.