# US 12,602,887 B2 · Augmented reality control surface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,887 B2 - Document ID: US12602887B2 - Status: Granted - Kind: B2 - Application number: 18/679,748 - Filed: 2024-05-31 - Published: 2025-12-04 - Granted: 2026-04-14 - Earliest claimed: 2024-05-31 - Assignees: Avid Technology, Inc. - Applicants: Avid Technology, Inc. - Inventors: William Lurie; Steven H. Milne; Edward Jones; Richard Nevens - Web page: https://patentgrep.com/patent/US12602887B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/0172: {characterised by optical features} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An augmented reality overlay of a physical control surface is displayed on augmented reality glasses worn by an operator of the control surface. The overlay appears to be part of the control surface by simulating a virtual silk-screen or floats over the surface. The overlay serves a variety of functions that assist the operator to mix multiple channels of audio or other aspects of a media project. The overlay may label controls of the control surface, display text or graphics to direct the operator to specific controls and may be dynamically updated based on the current editing context, either in response to an operator prompt or automatically. A 3D model of the control surface is used by the augmented reality system to register the overlay with respect to the control surface and to simulate occlusion of portions of the overlay by components of the control surface and the operator's hands. ## Claims ### Claim 1 A method of assisting an operator of a physical audio control surface, the method comprising: providing augmented reality (AR) glasses for the operator, wherein the AR glasses are in data communication with the physical audio control surface via an AR control system; providing a 3D model of the physical audio control surface, wherein the 3D model is stored in persistent storage of at least one of the physical audio control surface, the AR control system, and the AR glasses; enabling the operator to use physical controls of the physical audio control surface to mix a plurality of audio channels of a media project; and displaying on the AR glasses an overlay, wherein the overlay: is generated by at least one of the AR control system and the AR glasses using the 3D model to register a position of the overlay with respect to the physical control surface; and references one or more physical controls of the physical control surface. ### Claim 2 (depends on 1) The method of claim 1, wherein the overlay appears to be located on a surface of the physical audio control surface. ### Claim 3 (depends on 2) The method of claim 2, wherein at least part of the overlay appears to be printed onto the surface of the physical audio control surface. ### Claim 4 (depends on 1) The method of claim 1, wherein at least part of the overlay appears to float above a surface of the physical audio control surface. ### Claim 5 (depends on 1) The method of claim 1, wherein the overlay corresponds to functions assigned to the physical controls of the physical audio control surface, the functions being determined by a current phase of an audio mixing workflow of the media project. ### Claim 6 (depends on 1) The method of claim 1, wherein the overlay includes instructional text that pertains to one or more of the referenced one or more physical controls of the AR control surface. ### Claim 7 (depends on 1) The method of claim 1, wherein the overlay is dynamically updated based at least in part on a position of the AR glasses with respect to the physical audio control surface. ### Claim 8 (depends on 1) The method of claim 1, wherein the overlay is updated in real time to reflect an adjustment of a position of the referenced one or more physical controls. ### Claim 9 (depends on 1) The method of claim 1, wherein the overlay is updated in real time to simulate occlusion of a portion of the overlay by one or more physical components of the physical audio control surface. ### Claim 10 (depends on 9) The method of claim 9, wherein the physical component is a protruding knob and the occluded portion of the overlay includes a position indicator of the knob. ### Claim 11 (depends on 9) The method of claim 9, wherein the physical component is a fader cap and the occluded portion of the overlay includes a switch label. ### Claim 12 The method of 9, wherein the physical component is a display of the physical audio control surface. ### Claim 13 (depends on 1) The method of claim 1, wherein the overlay is adjusted in real time to simulate occlusion of a portion of the overlay by a hand of the operator. ### Claim 14 (depends on 1) The method of claim 1, further comprising: providing at least one of a hand and finger position sensor; using the at least one of the hand and finger position sensor to detect a gesture made by the operator; and wherein the AR overlay is determined in part by the gesture made by the operator. ### Claim 15 (depends on 1) The method of claim 1, wherein the overlay is generated in response to a prompt by the operator. ### Claim 16 (depends on 1) The method of claim 1, wherein the overlay is generated in response to operation of a control of the physical audio control surface. ### Claim 17 (depends on 1) The method of claim 1, wherein the overlay is determined in part by a current editing context of the media project. ### Claim 18 (depends on 17) The method of claim 17, wherein the editing context is determined by a media processing application controlled by the physical audio control surface. ### Claim 19 (depends on 17) The method of claim 17, wherein the editing context is determined by the physical audio control surface. ### Claim 20 (depends on 17) The method of claim 17, wherein the editing context corresponds to a phase of a multi-phase audio mixing workflow for the media project. ### Claim 21 A computer program product comprising: a non-transitory computer-readable medium with computer-readable instructions encoded thereon, wherein the computer-readable instructions, when processed by a processing device instruct the processing device to perform a method of assisting an operator of a physical audio control surface, the method comprising: providing augmented reality (AR) glasses for the operator, wherein the AR glasses are in data communication with the physical audio control surface via an AR control system; providing a 3D model of the physical audio control surface, wherein the 3D model is stored in persistent storage of at least one of the physical audio control surface, the AR control system, and the AR glasses; enabling the operator to use physical controls of the physical audio control surface to mix a plurality of audio channels of a media project; and displaying on the AR glasses an overlay, wherein the overlay: is generated by at least one of the AR control system and the AR glasses using the 3D model to register a position of the overlay with respect to the physical control surface; and references one or more physical controls of the physical control surface. ### Claim 22 A system comprising: a memory for storing computer-readable instructions; and a processor connected to the memory, wherein the processor, when executing the computer-readable instructions, causes the system to perform a method of assisting an operator of a physical audio control surface, the method comprising: providing augmented reality (AR) glasses for the operator, wherein the AR glasses are in data communication with the physical audio control surface via an AR control system; providing a 3D model of the physical audio control surface, wherein the 3D model is stored in persistent storage of at least one of the physical audio control surface, the AR control system, and the AR glasses; enabling the operator to use physical controls of the physical audio control surface to mix a plurality of audio channels of a media project; and displaying on the AR glasses an overlay, wherein the overlay: is generated by at least one of the AR control system and the AR glasses using the 3D model to register a position of the overlay with respect to the physical control surface; and references one or more physical controls of the physical control surface.