# US 2025/0095307 A1 · Video-see-through (vst) device for interacting with objects within a vst environment and method for operating the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0095307 A1 - Document ID: US20250095307A1 - Status: Published application - Kind: A1 - Application number: 18/967,138 - Filed: 2024-12-03 - Published: 2025-03-20 - Earliest claimed: 2023-08-02 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Biju Mathew NEYYAN; Vivek SRIDHAR; Arshed V. HAKEEM; Shubhankar BHINGARE - Web page: https://patentgrep.com/patent/US20250095307A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 40/18: Eye characteristics, e.g. of the iris - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G06T 2219/2016: Rotation, translation, scaling - G06V 2201/07: Target detection - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Provided are video-see-through (VST) device for interacting with a spatial object and an operating method the VST device. The VST device receives a user gesture of a user, wherein the user gesture is for selecting a spatial region of interest (ROI) within a field of view of the user; recognizes the spatial ROI and an object located within the selected spatial ROI; generates a virtual bounding region enclosing the recognized object located within the selected spatial ROI; determines an associated modality for enabling an interaction with the object located within the generated virtual bounding region; and generates a prompt corresponding to the associated modality. ## Claims ### Claim 1 A method operated by a video-see-through (VST) device, the method comprising: receiving at least one user gesture of a user for selecting a spatial region of interest (ROI) within a field of view of the user; recognizing the spatial ROI and at least one object located within the spatial ROI; generating at least one virtual bounding region enclosing the at least one recognized object located within the selected spatial ROI; determining at least one associated modality for enabling an interaction with the at least one object located within the at least one virtual bounding region; and generating at least one prompt corresponding to at least one associated modality for interaction with the at least one object wherein the prompt is generated based on a relative position of a hand of the user and the spatial ROI. ### Claim 2 (depends on 1) The method of claim 1, wherein the generating of the at least one virtual bounding region for the spatial ROI comprises: scaling a size of the boundary of the spatial ROI based on relative positions of hands of the user changed by the at least one user gesture. ### Claim 3 (depends on 1) The method of claim 1, wherein the recognizing of the spatial ROI and the at least one object located within the spatial ROI, comprises: detecting, by a head gaze tracker of the VST device, head orientation of the user; and detecting, by an eye gaze tracker of the VST device, eye gaze of the user, wherein the generating of the at least one virtual bounding region for the spatial ROI, comprises: scaling a size of the boundary of the spatial ROI based on at least one of the at least one gesture, the head orientation of the user, or the eye gaze of the user. ### Claim 4 (depends on 1) The method of claim 1, wherein the generating of the at least one virtual bounding region for the spatial ROI, comprises: generating an initial ROI boundary based on a plurality of initial ROI marking points, wherein the initial ROI boundary is at least one of a two-dimensional (2D) and a three-dimensional (3D) initial ROI boundary; and transforming spatially the initial ROI boundary based on a visual analysis of a scene as viewed by the user through the VST device and a plurality of transferal ROI marking points as received from the at least one user gesture. ### Claim 5 (depends on 1) The method of claim 1, wherein the determining of the at least one associated modality for enabling an interaction with the at least one object located within the spatial ROI, comprises: determining at least one likely input command for user interaction with the at least one object located within the spatial ROI, based on the at least one object and at least one of at least a textual element, at least an audio element and at least a visual element located with the at least one object within the spatial ROI; and determining a most likely associated modality by which the user specifies the at least one likely input command. ### Claim 6 (depends on 1) The method of claim 1, wherein the recognizing of the spatial ROI and the at least one object located within the spatial ROI comprises: detecting a hold of the at least one user gesture for a time interval; and recognizing the at least one object in the field of view over the time interval. ### Claim 7 (depends on 1) The method of claim 1, wherein the generating of the at least one prompt corresponding to the at least one associated modality, comprises: generating at least one of: at least one voice prompt, based on the at least one associated modality, for interacting with the at least one object; and at least one visual prompt based on the at least one associated modality, wherein the at least one visual prompt is generated by tracking position of the user and performing a hand reach assessment of the user; adjusting the at least one prompt based on change in the at least one user gesture and change in the at least one object as selected; and rendering at least one of the at least one voice prompt and the at least one visual prompt, and wherein the method further comprises: displaying the rendered at least one of the at least one voice prompt and the at least one visual prompt onto a display of the VST device. ### Claim 8 A video-see-through (VST) device comprising: an user input interface configured to receive gesture input from a user; at least one memory storing one or more instructions; and at least one processor operatively connected to the at least one memory and configured to execute the one or more instructions to cause the VST device to: receive, through the user input interface, at least one user gesture of the user for selecting a spatial region of interest (ROI) within a field of view of the user, recognize the spatial ROI and at least one object located within the spatial ROI; generate at least one virtual bounding region enclosing the at least one recognized object located within the selected spatial ROI, determine at least one associated modality for enabling an interaction with the at least one object located within the at least one virtual bounding region, and generate at least one prompt corresponding to the at least one associated modality for interaction with the at least one object, based on a relative position of a hand of the user and the spatial ROI. ### Claim 9 (depends on 8) The VST device of claim 8, wherein the at least one processor is further configured to execute the one or more instructions to cause the VST device to: scale a size of the boundary of the spatial ROI based on relative positions of hands of the user by change of the at least one user gesture. ### Claim 10 (depends on 8) The VST device of claim 8, further comprising: a head gaze tracker configured to detect head orientation of the user; and an eye gaze tracker configured to detect eye gaze of the user, wherein the at least one processor is further configured to execute the one or more instructions to cause the VST device to: scale a size of the boundary of the spatial ROI based on at least one of the at least one gesture, the head orientation of the user detected by the head gaze tracker, or the eye gaze of the user detected by the eye gaze tracker. ### Claim 11 (depends on 8) The VST device of claim 8, wherein the at least one processor is further configured to execute the one or more instructions to cause the VST device to: generate an initial ROI boundary based on a plurality of initial ROI marking points, wherein the initial ROI boundary may be at least one of a two-dimensional (2D) and a three-dimensional (3D) initial ROI boundary, and transform spatially the initial ROI boundary based on a visual analysis of a scene as viewed by the user through the VST device and a plurality of transferal ROI marking points as received from the at least one user gesture. ### Claim 12 (depends on 8) The VST device of claim 8, wherein the at least one processor is further configured to execute the one or more instructions to cause the VST device to: determine at least one likely input command for user interaction with the at least one object located within the spatial ROI, based on the at least one object and at least one of at least a textual element, at least an audio element and at least a visual element located with the at least one object within the spatial ROI, and determine a most likely associated modality by which the user specifies the at least one likely input command. ### Claim 13 (depends on 8) The VST device of claim 8, wherein the at least one processor is further configured to execute the one or more instructions to cause the VST device to: detect a hold of the at least one user gesture for a time interval, and recognize the at least one object in the field of view over the time interval. ### Claim 14 (depends on 8) The VST device of claim 8, wherein the at least one processor is further configured to execute the one or more instructions to cause the VST device to: generate at least one of: at least one voice prompt, based on the at least one associated modality, for interacting with the at least one object, and at least one visual prompt based on the at least one associated modality, wherein the at least one visual prompt is generated by tracking position of the user and performing a hand reach assessment of the user, adjust the at least one prompt based on change in the at least one user gesture and change in the at least one object as selected, and render at least one of the at least one voice prompt and the at least one visual prompt. ### Claim 15 (depends on 14) The VST device of claim 14, further comprises: A display; and wherein the at least one processor is further configured to execute the one or more instructions to cause the VST device to: control the display to display the rendered at least one of the at least one voice prompt and the at least one visual prompt.