# US 2020/0404159 A1 · Techniques to set focus in camera in a mixed-reality environment with hand gesture interaction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0404159 A1 - Document ID: US20200404159A1 - Status: Published application - Kind: A1 - Application number: 16/933,413 - Filed: 2020-07-20 - Published: 2020-12-24 - Earliest claimed: 2019-05-31 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Maria Cheng LEI; Vishal JAIN; Vikramaditya DANGI - Web page: https://patentgrep.com/patent/US20200404159A1 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G02B 7/28: Systems for automatic generation of focusing signals - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 27/0172: {characterised by optical features} - G03B 13/36: Autofocus systems - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06T 11/60: Creating or editing images; Combining images with text - H04N 23/60: Control of cameras or camera modules - H04N 23/62: Control of parameters via user interfaces - H04N 23/67: Focus control based on electronic image sensor signals - H04N 23/675: {comprising setting of focusing regions} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An adjustable-focus PV (picture/video) camera in a mixed-reality head-mounted display (HMD) device operates with an auto-focus subsystem that is configured to be triggered based on location and motion of a user's hands to reduce the occurrence of auto-focus hunting during camera operations. The HMD device is equipped with a depth sensor that is configured to capture depth data from the surrounding physical environment to detect and track the user's hand location, movements, and gestures in three-dimensions. The hand tracking data from the depth sensor may be assessed to determine hand characteristics—such as which of the user's hands or part of a hand is detected, its size, motion, speed, etc.—within a particular region of interest (ROI) in the field of view of the PV camera. The auto-focus subsystem uses the assessed hand characteristics as an input to control auto-focus of the PV camera to reduce auto-focus hunting occurrences. ## Claims ### Claim at position 1 1-20. (canceled) ### Claim 21 A method performed by a head-mounted display (HMD) device operable by a user to optimize an auto-focus implementation, comprising: enabling auto-focus operations for a camera disposed in the HMD device, the camera being configured to capture a scene in a local physical environment surrounding the HMD device over a field of view (FOV), wherein the camera is a member of a set of one or more sensors that are operatively coupled to the HMD device; providing a gaze detection subsystem, comprising one or more sensors in the HMD device, that is configured to detect locations of focus within the FOV of one or more eyes of the user; and using the detected focus locations to control the auto-focus operations for the camera according to one or more criteria of an auto-focus subsystem. ### Claim 22 (depends on 21) The method of claim 21, in which the set of sensors gather data which describes the local physical environment in which the HMD device operates. ### Claim 23 (depends on 22) The method of claim 22, in which the HMD device includes a see-through mixed-reality display through which a local user observes the local physical environment, and on which the HMD device renders one or more virtual objects. ### Claim 24 (depends on 23) The method of claim 23, in which the scene captured by the camera over the FOV and the rendered virtual objects are transmitted as content to a remote computing device. ### Claim 25 (depends on 23) The method of claim 23, in which the scene captured by the camera over the FOV and the rendered virtual objects are mixed by the HMD device into a composite signal that is recorded. ### Claim 26 (depends on 22) The method of claim 22, further comprising designating a region of interest (ROI) within the FOV, and wherein a criterion of the auto-focus subsystem includes the detected locations indicating that the user is interacting with one or more virtual objects within the ROI. ### Claim 27 (depends on 26) The method of claim 26, in which the ROI includes a three-dimensional space in the local physical environment. ### Claim 28 (depends on 26) The method of claim 26, in which the ROI is dynamically variable in at least one of size, shape, or location. ### Claim 29 (depends on 26) The method of claim 26, in which the set of sensors include a depth sensor and further comprising capturing characteristics of one or more of the user's hands using the depth sensor for tracking one or more of the HMD device user's hands to determine whether the detected locations satisfy one or more criterion of the auto-focus subsystem. ### Claim 30 (depends on 29) The method of claim 29, in which the characteristics of the hands include a pace of hand movements. ### Claim 31 (depends on 29) The method of claim 29, in which the characteristics of the hands include what part of a hand. ### Claim 32 (depends on 29) The method of claim 29, in which the characteristics of the hands include a duration of time that the one or more hands are positioned in the ROI. ### Claim 33 (depends on 21) The method of claim 21, in which the camera comprises a PV (Photo/Video) camera. ### Claim 34 One or more hardware-based non-transitory computer-readable memory devices storing computer-readable instructions which, when executed by one or more processors in a computing device that is employable by a user, cause the computing device to: utilize a gaze detecting subsystem to capture data for locations of focus of a user's eye as the user is employing the computing device in a physical environment; identify a region of interest (ROI) within a field of view (FOV) of the computing device, in which the computing device renders one or more virtual objects on a see-through display coupled to the computing device to enable the user to simultaneously view the physical environment and the one or more virtual objects as a mixed-reality user experience; determine, from the captured focus location data, an extent to which the user is interacting with virtual objects that are located within the ROI; and in response to the determining, trigger or disable an auto-focus operation of a front-facing camera, disposed on the computing device, that is configured to capture a scene comprising at least a portion of the physical environment in the FOV, in which the auto-focus operation is triggered responsive to a determination that user interaction with the virtual objects is a point of focus within the ROI; and the auto-focus operation is disabled responsive to a determination that user interaction with the virtual objects is transitory within the ROI. ### Claim 35 (depends on 34) The one or more hardware-based non-transitory computer-readable memory devices of claim 34, in which the instructions further cause the computing device to capture hand tracking data from a depth sensor that is operatively coupled to the computing device and using the captured hand tracking data to trigger or disable auto-focus operations of the camera. ### Claim 36 (depends on 34) The one or more hardware-based non-transitory computer-readable memory devices of claim 34, in which the computing device includes a head-mounted display (HMD) device, a smartphone, a tablet computer, or a portable computer. ### Claim 37 (depends on 34) The one or more hardware-based non-transitory computer-readable memory devices of claim 34, in which the ROI is located at a central region of the FOV. ### Claim 38 (depends on 37) The one or more hardware-based non-transitory computer-readable memory devices of claim 37, in which the gaze detection subsystem comprises an inward-facing sensor that is configured to capture images of one or more of the user's eyes. ### Claim 39 A computing device configurable to be worn on a head of a user, the computing device configured to reduce unwanted hunting effects of an auto-focus subsystem associated with the computing device, comprising: a display configured to render holograms; an adjustable-focus PV (picture/video) camera that is operatively coupled to the auto-focus subsystem and that is configured to capture adjustably focused images of a physical environment in which the user is located; a gaze detection subsystem configured to detecting the user's gaze in the physical environment comprising one or more of gaze direction, focus direction, or focus location; one or more processors; and one or more hardware-based memory devices storing computer-readable instructions which, when executed by the one or more processors, cause the computing device to: render at least one hologram on the display, the hologram comprising a virtual object that is located in the physical environment at a known location; capture, using the gaze detection subsystem, gaze tracking data for the user's gaze in the physical environment; determine from the gaze tracking data whether the user's one or more hands are interacting with the hologram at the known location; trigger operations of the auto-focus subsystem in response to a determination that the user is not interacting with the hologram; and suppress operations of the auto-focus subsystem in response to a determination that the user is interacting with the hologram. ### Claim 40 (depends on 39) The computing device of claim 39, in which operations of the auto-focus subsystem are suppressed in response to a determination from the gaze tracking data that the user's interactions with the hologram comprise an in-out movement of one or more of the user's hands.