# US 2025/0148728 A1 · Augmented reality system including corporal tactile devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0148728 A1 - Document ID: US20250148728A1 - Status: Published application - Kind: A1 - Application number: 19/013,709 - Filed: 2025-01-08 - Published: 2025-05-08 - Earliest claimed: 2023-07-12 - Assignees: University of Central Florida Research Foundation, Inc. - Applicants: University of Central Florida Research Foundation, Inc. - Inventors: Christopher Randles; Ryan McMahan - Web page: https://patentgrep.com/patent/US20250148728A1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G09B 19/24: Use of tools - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip - G09B 23/24: for chemistry - G02B 27/017: {Head mounted} - G06K 19/06037: {multi-dimensional coding} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS ## Abstract The present invention relates to an advanced augmented reality, AR, apparatus featuring a video see-through AR, VST-AR, device that captures images of a real-world environment and displays augmented versions on a screen. The system includes a corporal entity, such as an empty laboratory vessel, equipped with a unique identification marker. This marker provides real-time spatial data for generating visual augmentations representing virtual matter within the vessel. The apparatus integrates various sensors, including an inertial measurement unit, IMU, capacitive tactile sensors, and a thermal diode, to dynamically adjust visual augmentations based on user interactions. Additional sensory enhancements include an olfactory output fan, an eccentric rotating mass, ERM, motor, and auditory outputs. Wireless connectivity via IEEE 802.11 (Wi-Fi) and IEEE 802.15.1 (Bluetooth) ensures seamless communication between components. This AR system is particularly valuable for educational and laboratory settings, offering realistic, interactive simulations. ## Claims ### Claim 1 An augmented reality, AR, apparatus comprising: a) a video see-through AR, VST-AR, device configured to capture images of a real-world environment and display augmented versions of those images on a display screen; b) a corporal entity comprising a physical analog of an otherwise empty laboratory vessel, wherein the corporal entity further comprises an inertial measurement unit, IMU, physically affixed to the vessel and communicatively coupled to the at least one processor, the IMU including an accelerometer, gyroscope, and magnetometer to detect the orientation and movement of the vessel thereby providing data to the at least one processor for dynamic adjustment of the visual augmentations; and c) a spatial registration module accessible by at least one processor wherein a registration indicium captured by the VST-AR device conveys a spatial position and orientation of the vessel in real-time to the processor wherein the at least one processor generates visual augmentations to the display, the augmentations representing virtual matter present within the vessel. ### Claim 2 (depends on 1) The AR apparatus of claim 1 wherein the virtual matter is rendered as a liquid. ### Claim 3 (depends on 2) The AR apparatus of claim 2 wherein the augmentations of the liquid are re-rendered to the display in real time responsive to changes in the spatial position or orientation of the physical manifestation of the vessel. ### Claim 4 (depends on 1) The AR apparatus of claim 1 wherein the registration indicium is a machine-readable code affixed to the vessel and optically retrieved by the VST-AR device. ### Claim 5 (depends on 1) The AR apparatus of claim 1 wherein the registration indicium comprises a plurality of infrared lights affixed to the vessel and optically retrieved by the VST-AR device. ### Claim 6 (depends on 1) The AR apparatus of claim 1, wherein the registration indicium is a machine-readable code affixed to the vessel and optically retrieved by the VST-AR device to initialize the spatial position and orientation of the vessel and then subsequently the IMU conveys changes in the spatial position and orientation whereby subsequent optical occlusion of the code does not impact the ability of the VST-AR device to render visual augmentations for new positions and orientations of the vessel. ### Claim 7 (depends on 1) The AR apparatus of claim 1, further comprising capacitive tactile sensors communicatively coupled to the at least one processor, the sensors affixed to an exterior of the vessel to detect user touch thereby providing feedback to the VST-AR device for enhancing the visual augmentations with tactile responses. ### Claim 8 (depends on 1) The AR apparatus of claim 1, further comprising a thermal diode integrated into the vessel to simulate temperature changes and provided tactile feedback representing exothermic or endothermic reactions in synchronization with the visual augmentations. ### Claim 9 (depends on 1) The AR apparatus of claim 1, further comprising an olfactory output fan within the vessel to emit scents in synchronization with the visual augmentations. ### Claim 10 (depends on 1) The AR apparatus of claim 1, further comprising an eccentric rotating mass, ERM, motor disposed within the vessel to generate vibrations, simulating tactile sensations of stirring or mixing fluids in synchronization with the visual augmentations. ### Claim 11 (depends on 1) The AR apparatus of claim 1, further comprising a sound output in conjunction with the display, the sound output configured to emit audio corresponding to the visual augmentations. ### Claim 12 (depends on 1) The AR apparatus of claim 1, further comprising a fluid flow system including a fluid reservoir and a bidirectional pump to send real, physical fluid to and from the vessel in synchronization with visual augmentations of virtual fluids shown on the display. ### Claim 13 (depends on 12) The AR apparatus of claim 12 wherein the vessel retains the fluids impermeably as they are pumped in or out by the bidirectional pump wherein spilling of the physical fluid by reorientation of the vessel into an inverted state is not possible. ### Claim 14 An augmented reality, AR, apparatus comprising: d) a video see-through AR, VST-AR, device configured to capture images of a real-world environment and display augmented versions of those images on a display screen; e) a corporal entity comprising a physical analog of an otherwise empty laboratory vessel; f) an identification marker affixed to the vessel, the marker uniquely associated with at least the type of vessel, the VST-AR device configured to capture and decode the marker; g) at least one processor communicatively coupled to the VST-AR device and further communicatively coupled to a data store, the data store using the decoded marker as a key to retrieve operational parameters from the data store that define how an augmentation of the vessel by the VST-AR is rendered to the display screen, wherein the virtual matter is rendered as a liquid and an operational parameter is viscosity whereby the augmentation of the virtual matter as the liquid further conveys the liquid's viscosity based on simulated fluid dynamics; and h) a spatial registration module accessible by the at least one processor wherein a registration indicium captured by the VST-AR device conveys distance and orientation of the vessel in real-time to the processor wherein the at least one processor generates visual augmentations to the display, the augmentations representing virtual matter present within the vessel. ### Claim 15 (depends on 14) The AR apparatus of claim 14 wherein an operational parameter is exothermal reactivity whereby the augmentation of the virtual matter as a liquid changes the visual appearance of the matter according to the exothermal reactivity parameter. ### Claim 16 (depends on 15) The AR apparatus of claim 15, further comprising a thermal diode integrated into the vessel whereby an exothermic reaction is simulated by increasing the heat of the vessel by the thermal diode based on the exothermal reactivity parameter in synchronization with the visual augmentations. ### Claim 17 (depends on 14) The AR apparatus of claim 14, further comprising an olfactory output fan within the vessel to emit scents in synchronization with the visual augmentations according to an olfactory operational parameter. ### Claim 18 (depends on 14) The AR apparatus of claim 14, further comprising an eccentric rotating mass, ERM, motor disposed within the vessel to generate vibrations, simulating tactile sensations of stirring or mixing fluids in synchronization with the visual augmentations. ### Claim 19 (depends on 14) The AR apparatus of claim 14, further comprising a sound output in conjunction with the display, the sound output configured to emit audio corresponding to the visual augmentations. ### Claim 20 (depends on 14) The AR apparatus of claim 14, further comprising a fluid flow system including a fluid reservoir and a bidirectional pump to send real, physical fluid to and from the vessel in synchronization with visual augmentations of virtual fluids shown on the display. ### Claim 21 (depends on 20) The AR apparatus of claim 20 wherein the vessel retains the fluids impermeably as they are pumped in or out by the bidirectional pump wherein spilling of the physical fluid by reorientation of the vessel into an inverted state is not possible. ### Claim 22 An augmented reality, AR, apparatus comprising: a) a video see-through AR, VST-AR, device configured to capture images of a real-world environment and display augmented versions of those images on a display screen; b) a laboratory instrument comprising a physical object with integrated sensors and markers for AR enhancement, wherein the laboratory instrument is a micro-pipette and further comprises capacitive tactile sensors affixed to the laboratory instrument to detect user touch and grip, piezoelectric pressure sensors to measure the force applied during liquid transfer, and a mechanical pressure sensor to provide precise feedback on the amount of pressure exerted on the pipette tip; c) an identification marker affixed to the laboratory instrument, the marker uniquely associated with the laboratory instrument, the VST-AR device configured to capture and decode the marker; d) at least one processor communicatively coupled to the VST-AR device and further communicatively coupled to a data store, the data store using the decoded marker as a key to retrieve operational parameters from the data store that define how an augmentation of the laboratory instrument by the VST-AR is rendered to the display screen; and e) a spatial registration module accessible by the at least one processor wherein the unique identification marker optically captured by the VST-AR device conveys distance and orientation of the laboratory instrument in real-time to the processor wherein the at least one processor generates visual augmentations to the display, the augmentations corresponding to the laboratory tasks performed with the instrument.