# US 10,504,294 B2 · System and method for augmented reality discrepancy determination and reporting > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,294 B2 - Document ID: US10504294B2 - Status: Granted - Kind: B2 - Application number: 16/199,780 - Filed: 2018-11-26 - Published: 2019-03-28 - Granted: 2019-12-10 - Earliest claimed: 2014-06-09 - Assignees: Huntington Ingalls Incorporated - Applicants: Huntington Ingalls Incorporated - Inventors: Brian Bare; Jonathan Martin; Patrick Ryan; Mark Lawrence; Paul Sells - Web page: https://patentgrep.com/patent/US10504294B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06T 15/205: {Image-based rendering} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/20: in augmented reality scenes - H04W 4/021: Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A method is provided for providing target object information to a mobile interface device user in a dynamic structural environment. The method includes receiving from the mobile interface device a request for target object information associated with a target object in the dynamic structural environment. A pose of the mobile interface device relative to the target object is determined accounting for spatial differences in the environment coordinate system resulting from changes in the dynamic structure. The method also includes assembling AR target object information for transmission to and display on the mobile interface device and transmitting the AR target object information to the mobile interface device for display in conjunction with a real-time view of the target object. ## Claims ### Claim 1 A method for providing target object information to a mobile interface device user in a dynamic structural environment in the form of a dynamic structure having an environment coordinate system, the method comprising: receiving on a central data processor from the mobile interface device over a communication network a request for target object information associated with a target object in the dynamic structural environment; determining a pose of the mobile interface device relative to the target object accounting for spatial differences in the environment coordinate system resulting from changes in the dynamic structure; assembling augmented reality (AR) target object information for transmission to and display on the mobile interface device; and transmitting the AR target object information to the mobile interface device for display in conjunction with a real-time view of the target object. ### Claim 2 (depends on 1) The method of claim 1 further comprising: receiving first spatial information from a local positioning system and second spatial information from a three dimensional model of the dynamic structure; and using the first and second spatial information to determine the spatial differences in the environment coordinate system resulting from changes in the dynamic structure. ### Claim 3 (depends on 1) The method of claim 1 further comprising: receiving a digital image of the target object captured by the mobile interface device; and receiving mobile interface device location information from a local positioning system, said location information being provided in the environment coordinate system, wherein the location information and the digital image are used in the action of determining the pose of the mobile interface device. ### Claim 4 (depends on 1) The method of claim 1 wherein the AR target object information includes at least one of the set consisting of information on a desired target object parameter, and a graphical representation of a desired target object configuration. ### Claim 5 (depends on 1) The method of claim 1 further comprising: receiving, from a measurement data system, target object data for one or more target object parameters. ### Claim 6 (depends on 5) The method of claim 5 wherein the AR target object information includes a graphical representation of the one or more target object parameters, the graphical representation being constructed using the target object data. ### Claim 7 (depends on 5) The method of claim 5 further comprising: comparing the target object data to desired target object values for the one or more target object parameters to determine target object parameter deviations. ### Claim 8 (depends on 7) The method of claim 7 wherein the AR target object information includes a graphical representation of the target object parameter deviations. ### Claim 9 (depends on 7) The method of claim 7 further comprising: performing at least one of the set consisting of a maintenance operation and a discrepancy correction operation on the target object; and repeating the actions of receiving target object data, comparing the target object data to desired target object values to determine target object parameter deviations, assembling AR target object information, and transmitting the AR target information to the mobile interface device for display. ### Claim 10 (depends on 1) The method of claim 1, wherein the mobile interface device is one of the set consisting of a tablet computer, a smartphone, and a wearable heads-up display. ### Claim 11 (depends on 1) The method of claim 1 wherein the request for target environment information includes a digital image of the target object captured by the mobile interface device. ### Claim 12 (depends on 1) The method of claim 1 wherein the dynamic structure defines an interior space in which the target object is disposed. ### Claim 13 (depends on 1) The method of claim 1 wherein the dynamic structure comprises the target object. ### Claim 14 (depends on 1) The method of claim 1 wherein the spatial changes result from flexure of the dynamic structure. ### Claim 15 An automated system for providing target object information to a mobile interface device for use in inspecting a target object in a dynamic structural environment in the form of a dynamic structure having an environment coordinate system, the system comprising: at least one mobile interface device configured for variable disposition within the dynamic structural environment, capturing information about the target object within the structural environment, and transmitting the target object information over a network; a local positioning system in communication with the at least one mobile interface device via the network; and a central processor comprising at least one data processing machine in communication with the at least one mobile interface device and the local positioning system via the network, the central processor being configured for receiving from a requesting one of the at least one mobile interface device a request for target object information associated with the target object; determining a pose of the requesting mobile interface device relative to the target object accounting for spatial differences in the environment coordinate system resulting from changes in the dynamic structure; assembling augmented reality (AR) target object information for transmission to and display on the mobile interface device; and transmitting the AR target object information to the mobile interface device for display in conjunction with a real-time view of the target object. ### Claim 16 (depends on 15) The automated system of claim 15 wherein the central processor is also configured for receiving first spatial information from a local positioning system and second spatial information from a three dimensional model of the dynamic structure; and using the first and second spatial information to determine the spatial differences in the environment coordinate system resulting from changes in the dynamic structure. ### Claim 17 (depends on 15) The automated system of claim 15 further comprising: a measurement data system configured for capturing target object data for one or more target object parameters. ### Claim 18 (depends on 17) The automated system of claim 17, wherein the central processor is also configured for receiving the target object data, and comparing the target object data to desired target object values for the one or more target object parameters to determine target object parameter deviations, and wherein the AR target object information includes a graphical representation of the target object parameter deviations.