# US 12,181,849 B2 · Control of semiconductor manufacturing equipment in mixed reality environments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,181,849 B2 - Document ID: US12181849B2 - Status: Granted - Kind: B2 - Application number: 18/554,823 - Filed: 2022-04-11 - Published: 2024-06-13 - Granted: 2024-12-31 - Earliest claimed: 2021-04-14 - Assignees: Lam Research Corporation - Applicants: Lam Research Corporation - Inventors: Rainer Unterguggenberger; Christopher Thorgrimsson; Henry T. Chan; Chung-Ho Huang; Terrence George Bernier - Web page: https://patentgrep.com/patent/US12181849B2 ## CPC - G05B 19/0425: {Safety, monitoring} - G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes - G05B 19/418: Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS] or computer integrated manufacturing [CIM] - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 21/44: Program or device authentication - G06F 21/602: {Providing cryptographic facilities or services} - G06T 15/20: Perspective computation - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - H04W 12/03: Protecting confidentiality, e.g. by encryption - H04W 12/06: Authentication - H10P 72/06: {Apparatus for monitoring, sorting, marking, testing or measuring} - G05B 2219/32014: Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command - G05B 2219/45031: Manufacturing semiconductor wafers - G06T 2200/24: involving graphical user interfaces [GUIs] - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - H10P: GENERIC PROCESSES OR APPARATUS FOR THE MANUFACTURE OR TREATMENT OF DEVICES COVERED BY CLASS H10 ## Abstract Various embodiments herein relate to a Mixed Reality (MR) control platform to operate a semiconductor manufacturing tool in an MR environment and to display data associated with the semiconductor manufacturing tool. In son embodiments, the MR control platform comprises an MR control system and an MR headset. The MR control system can obtain sensor data representative of sensor output from a semiconductor manufacturing tool. The MR control system can determine operational information associated with the semiconductor manufacturing tool and based on the sensor data. The MR control system can cause the operational information to be transmitted to the MR headset. The MR headset can receive the operational information associated with the semiconductor manufacturing tool from the MR control system. The MR headset can cause content associated with the operational information and one or more control features to be rendered in an MR environment. ## Claims ### Claim 1 A Mixed Reality (MR) headset configured to control a semiconductor manufacturing tool in an MR environment, the MR headset comprising: one or more processors; a display coupled to the one or more processors; one or more cameras; and one or more memories storing computer-executable instructions which, when executed, cause the one or more processors to: establish a wireless communication channel with an MR control system of a semiconductor manufacturing tool; receive operational information associated with the semiconductor manufacturing tool from the MR control system via the wireless communication channel, wherein the operational information comprises sensor data obtained from one or more sensors of the semiconductor manufacturing tool; and cause content associated with the sensor data and one or more control features to be rendered in an MR environment. ### Claim 2 (depends on 1) The MR headset of claim 1, wherein the instructions, when executed, further cause the one or more processors to initiate the establishment of the wireless communication channel by: identifying a wireless access point associated with the MR control system and a password associated with the wireless access point; and connecting to the wireless access point using the password. ### Claim 3 (depends on 2) The MR headset of claim 2, wherein the instructions, when executed, further cause the one or more processors to identify the wireless access point by: obtaining image data via a camera of the one or more cameras; identifying a machine-readable code in the image data; decoding the machine-readable code to retrieve encrypted information; and decrypting the encrypted information using key information stored in the memory, wherein the decrypted information includes an identifier of the wireless access point and the password. ### Claim 4 (depends on 1) The MR headset of claim 1, wherein the instructions, when executed, further cause the one or more processors to identify a physical position and an orientation of the MR headset based on data from one or more sensors of the MR headset, wherein the content is rendered from a perspective that is based on the identified physical position and orientation of the MR headset. ### Claim 5 (depends on 1) The MR headset of claim 1, wherein the instructions, when executed, further cause the one or more processors to: receive an input that indicates second operational information associated with the semiconductor manufacturing tool, wherein the second operational information is different at least in part from the operational information; and cause a request for the second operational information to be transmitted to the MR control system; receive the second operational information from the MR control system; and cause second content associated with the second operational information to be rendered. ### Claim 6 (depends on 5) The MR headset of claim 5, wherein the input that indicates the second operational information indicates at least one of: a selection of a selectable input included in a menu rendered by the MR headset; or that the second operational information is generated based on an image capture of a machine-readable code that identifies the second operational information. ### Claim 7 (depends on 5) The MR headset of claim 5, wherein the rendered content comprises a user interface element that presents second sensor data obtained from the one or more sensors of the semiconductor manufacturing tool, and wherein the second sensor data corresponds to the requested second operational information. ### Claim 8 (depends on 7) The MR headset of claim 7, wherein the instructions, when executed, further cause the one or more processors to: receive a second input that indicates that the user interface element is to be fixed at a spatial coordinate with respect to the semiconductor manufacturing tool; identify a group of spatial coordinates in a coordinate system that is fixed with respect to the semiconductor manufacturing tool, wherein the group of spatial coordinates indicate borders of the user interface element with respect to a fixed spatial coordinate; identify a physical position and an orientation of the MR headset relative to the coordinate system; and modify presentation of the user interface element based on the physical position and the orientation of the MR headset relative to the fixed spatial coordinate. ### Claim 9 (depends on 1) The MR headset of claim 1, wherein the instructions, when executed, further cause the one or more processors to: receive three-dimensional model information associated with the semiconductor manufacturing tool as part of the operational information; identify a physical position and orientation of the MR headset relative to the semiconductor manufacturing tool; and cause a three-dimensional representation of one or more elements of the semiconductor manufacturing tool to be rendered based on: 1) the three-dimensional model information and 2) the physical position and orientation of the MR headset relative to the semiconductor manufacturing tool as part of the rendered content associated with the operational information in the MR environment. ### Claim 10 (depends on 9) The MR headset of claim 9, wherein the operational information comprises positions of one or more internal elements of the semiconductor manufacturing tool over time, and wherein the rendered content comprises representations of the positions of the one or more internal elements of the semiconductor manufacturing tool over time relative to the semiconductor manufacturing tool. ### Claim 11 (depends on 10) The MR headset of claim 10, wherein the instructions, when executed, further cause the one or more processors to render the content by: generating one or more three-dimensional images representing the one or more internal elements using the three-dimensional model information; and causing the one or more three-dimensional images to be rendered by the MR headset based on the positions of the one or more internal elements relative to the semiconductor manufacturing tool. ### Claim 12 (depends on 1) The MR headset of claim 1, wherein the instructions, when executed further cause the one or more processors to: receive three-dimensional model information associated with the semiconductor manufacturing tool as part of the operational information; and cause the content to be rendered based on the three-dimensional model information, wherein the rendered content comprises content indicative of a state of a component a digital twin represented by the three-dimensional model information, wherein the digital twin represents the semiconductor manufacturing tool. ### Claim 13 A Mixed-Reality (MR) control system comprising: one or more processors; one or more memory devices storing computer-executable instructions which, when executed, cause the one or more processors to: establish a wireless communication channel with an MR headset; obtain sensor data representative of sensor output from a semiconductor manufacturing tool; determine operational information of the semiconductor manufacturing tool based on the sensor data; and cause the operational information comprising the sensor data to be transmitted to the MR headset via the communication channel, wherein the MR headset is configured to cause content associated with the sensor data to be rendered in an MR environment. ### Claim 14 (depends on 13) The MR control system of claim 13, wherein the MR control system is in communication with a digital twin, and wherein the sensor data comprises virtual sensor data obtained from the digital twin. ### Claim 15 (depends on 13) The MR control system of claim 13, wherein the operational information comprises an indication of a state of the digital twin. ### Claim 16 (depends on 15) The MR control system of claim 15, wherein the instructions further cause the one or more processors to transmit three-dimensional model information associated with at least a portion of the semiconductor manufacturing tool to the MR headset, wherein the MR headset renders content indicating the state of the digital twin based on the three-dimensional model information. ### Claim 17 (depends on 13) The MR control system of claim 13, wherein the MR control system is in communication with the semiconductor manufacturing tool, and wherein the sensor data comprises sensor data obtained from one or more physical sensors of the semiconductor manufacturing tool. ### Claim 18 (depends on 13) The MR control system of claim 13, wherein the operational information comprises a state of the semiconductor manufacturing tool. ### Claim 19 (depends on 13) The MR control system of claim 13, wherein the instructions, when executed, further cause the one or more processors to determine positions of one or more internal elements of the semiconductor manufacturing tool over time, wherein the operational information comprises information indicating the positions of the one or more internal elements of the semiconductor manufacturing tool over time relative to the semiconductor manufacturing tool. ### Claim 20 (depends on 19) The MR control system of claim 19 wherein the instructions, when executed, further cause the one or more processors to transmit three-dimensional model information associated with at least a portion of the semiconductor manufacturing tool including at least one of the one or more internal elements, wherein the three-dimensional model information is used by MR headset to render representations of the one or more internal elements based on the three-dimensional model information. ### Claim 21 (depends on 13) The MR control system of claim 13, wherein the instructions, when executed, further cause the one or more processors to: receive, from the MR headset, a request for second operational information that is at least different in part from the operational information; and cause the requested second operational information to be transmitted to the MR headset. ### Claim 22 A method for controlling a semiconductor manufacturing tool via an MR session, the method comprising: establishing, by an MR headset, a wireless communication channel with an MR control system; obtaining, by the MR control system, sensor data representative of sensor output from a semiconductor manufacturing tool; determining, by the MR control system, operational information associated with the semiconductor manufacturing tool and based, at least in part, on the sensor data; transmitting, by the MR control system and responsive to the wireless communication channel being established with an MR headset, the operational information associated with the semiconductor manufacturing tool to the MR headset via the wireless communication channel; receiving, by the MR headset, the operational information associated with the semiconductor manufacturing tool from the MR control system via the wireless communication channel, wherein the operational information comprises the sensor data obtained from one or more sensors of the semiconductor manufacturing tool; and rendering, by the MR headset, content associated with the sensor data and one or more control features in an MR environment. ### Claim 23 (depends on 22) The method of claim 22, further comprising: identifying, by the MR headset, a wireless access point associated with the MR control system and a password associated with the wireless access point; and connecting to the wireless access point using the password. ### Claim 24 (depends on 22) The method of claim 22, wherein the rendered content comprises a user interface element that indicates a sensor value included in the sensor data obtained by the MR control system. ### Claim 25 (depends on 24) The method of claim 24, further comprising: receiving, by the MR headset, a user input that indicates that the user interface element is to be fixed at a spatial coordinate with respect to the semiconductor manufacturing tool; identifying a group of spatial coordinates in a coordinate system that is fixed with respect to the semiconductor manufacturing tool, wherein the group of spatial coordinates indicate borders of the user interface element with respect to a fixed spatial coordinate; identifying a physical position and an orientation of the MR headset relative to the coordinate system; and modifying presentation of the user interface element based on the physical position and the orientation of the MR headset relative to the fixed spatial coordinate. ### Claim 26 (depends on 22) The method of claim 22, wherein the operational information comprises positions of one or more internal elements of the semiconductor manufacturing tool, wherein the rendered content comprises three-dimensional representations of the positions of the one or more internal elements of the semiconductor manufacturing tool over time, and wherein the three-dimensional representations of the positions of the one or more internal elements of the semiconductor manufacturing tool are rendered based on a physical position and orientation of the MR headset relative to the semiconductor manufacturing tool. ### Claim 27 (depends on 26) The method of claim 26, further comprising: transmitting, by the MR control system, three-dimensional model information associated with the semiconductor manufacturing tool as part of the operational information; and receiving, by the MR headset, the three-dimensional model information, wherein the three-dimensional representations of the positions of the one or more internal elements are rendered based on the three-dimensional model information relative to the semiconductor manufacturing tool. ### Claim 28 (depends on 22) The method of claim 22, further comprising: receiving, by the MR headset, one or more input signals associated with an operational instruction for the semiconductor manufacturing tool; transmitting, responsive to receiving the one or more input signals, the operational instruction to the MR control system; receiving, by the MR control system, the operational instruction from the MR headset; and transmitting, by the MR control system, a command to change a state of the semiconductor manufacturing tool or of a digital twin that represents the semiconductor manufacturing tool via a communications interface configured to be communicably connected with the semiconductor manufacturing tool. ### Claim 29 (depends on 22) The method of claim 22, wherein the sensor data comprises virtual sensor data generated by the digital twin, and wherein the instructions stored in the memory of the MR control system cause the one or more processors of the MR control system to receive the virtual sensor data from the digital twin. ### Claim 30 (depends on 22) The method of claim 22, further comprising: receiving, by the MR headset, an input that indicates second operational information associated with the semiconductor manufacturing tool, wherein the second operational information is different at least in part from the operational information; transmitting, by the MR headset, a request for the second operational information to the MR control system; receiving, by the MR control system, the request for the second operational information; transmitting, by the MR control system, the second operational information to the MR headset responsive to receiving the request; and rendering, by the MR headset, second content associated with the second operational information responsive to receiving the second operational information from the MR control system.