# US 11,442,438 B2 · Automated supervision and inspection of assembly process > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,438 B2 - Document ID: US11442438B2 - Status: Granted - Kind: B2 - Application number: 16/523,762 - Filed: 2019-07-26 - Published: 2020-02-20 - Granted: 2022-09-13 - Earliest claimed: 2018-08-14 - Assignees: The Boeing Company - Applicants: The Boeing Company - Inventors: Huafeng Yu; Daniel S. ReMine; Tyler Charles Staudinger - Web page: https://patentgrep.com/patent/US11442438B2 ## CPC - G05B 19/41805: {characterised by assembly} - G05B 19/41875: {characterised by quality surveillance of production} - G05B 19/4188: {characterised by CIM planning or realisation} - G05B 19/41885: {characterised by modeling, simulation of the manufacturing system} - G06F 18/22: Matching criteria, e.g. proximity measures - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 20/00: Machine learning - G06V 20/20: in augmented reality scenes - G05B 2219/32193: Ann, neural base quality management - G06N 3/08: Learning methods - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and apparatus for performing automated supervision and inspection of an assembly process. The method is implemented using a computer system. Sensor data is generated at an assembly site using a sensor system positioned relative to the assembly site. A current stage of an assembly process for building an assembly at the assembly site is identified using the sensor data. A context for the current stage is identified. A quality report for the assembly is generated based on the sensor data and the context for the current stage. ## Claims ### Claim 1 A method for performing automated supervision and inspection of an assembly process, the method being implemented using a computer system and comprising: generating sensor data at an assembly site using a sensor system positioned relative to the assembly site; identifying, by the computer system, a current stage of a plurality of stages in the assembly process for building an assembly at the assembly site based on the sensor data and assembly process flow data that includes information about a sequence of the plurality of stages, wherein identifying the current stage includes detecting a plurality of objects using the sensor data and matching at least a portion of the plurality of objects to parts for the assembly such that the portion of the plurality of objects matched to the parts is a plurality of matched objects; identifying, by the computer system, a context for the current stage; generating, by the computer system, a quality report for the assembly corresponding to the current stage based on the sensor data and the context for the current stage; and storing, by the computer system, a record corresponding to the current stage of the assembly process in a data repository to thereby build a history of records for the assembly process, wherein the record includes at least a portion of the sensor data, the quality report, an identification of the plurality of objects, and an identification of the plurality of matched objects. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the sensor data comprises: generating imaging data using a set of imaging systems positioned relative to the assembly and the assembly site, wherein the imaging data comprises at least one of two-dimensional imaging data or three-dimensional imaging data. ### Claim 3 (depends on 1) The method of claim 1, wherein the quality report includes an overall stage assessment that identifies a grade for the current stage of the assembly process, and wherein the grade indicates how the current stage has progressed with respect to selected tolerances and is selected from a plurality of grade levels that includes pass, borderline, and fail. ### Claim 4 (depends on 1) The method of claim 1, wherein matching the at least a portion of the plurality of objects to the parts comprises: matching, by the computer system, the at least a portion of the plurality of objects to the parts for the assembly using at least one of images of the parts or models of the parts. ### Claim 5 (depends on 1) The method of claim 1, wherein matching the at least a portion of the plurality of objects to the parts comprises: matching, by the computer system, the at least a portion of the plurality of objects to the parts for the assembly using at least one of machine learning, deep learning, computer vision, a customized learning technique, computational learning, or artificial intelligence learning. ### Claim 6 (depends on 4) The method of claim 4, wherein identifying, by the computer system, the current stage of the assembly process further comprises: determining, by the computer system, at least one of a plurality of locations or a plurality of orientations for the plurality of matched objects with respect to a reference coordinate system for the assembly site; and determining, by the computer system, the current stage of the assembly process based on an identification of the plurality of matched objects and the at least one of the plurality of locations or the plurality of orientations for the plurality of matched objects. ### Claim 7 (depends on 6) The method of claim 6, wherein identifying, by the computer system, the context comprises: determining, by the computer system, the context for the current stage of the assembly process using context information stored in a data repository, wherein the context includes at least one of a nominal condition for the assembly at the current stage, a nominal condition for each part expected to be present at the current stage, selected tolerances for the nominal condition for the assembly at the current stage, an identification of steps completed prior to the current stage, an identification of steps to be completed during the current stage, or an identification of steps to be completed after the current stage. ### Claim 8 (depends on 6) The method of claim 6, wherein generating, by the computer system, the quality report comprises: generating, by the computer system, a matching assessment that indicates whether one or more of the plurality of objects detected is unable to be matched to a corresponding part and whether one or more of the plurality of matched objects does not match a part that is expected to be present at the current stage of the assembly process. ### Claim 9 (depends on 6) The method of claim 6, wherein generating, by the computer system, the quality report comprises: generating, by the computer system, a location assessment that indicates whether each of the plurality of matched objects has been assembled in a correct or incorrect location based on the context for the current stage of the assembly process. ### Claim 10 (depends on 6) The method of claim 6, wherein generating, by the computer system, the quality report comprises: generating, by the computer system, an inconsistency assessment that indicates whether any features have been identified and whether any identified features are outside of selected tolerances, wherein a feature is selected from one of a crack, a dent, a twist, or a surface level feature. ### Claim 11 (depends on 10) The method of claim 10, further comprising: detecting, by the computer system, one or more features in the assembly using at least one of machine learning, deep learning, computer vision, a customized learning technique, computational learning, or artificial intelligence learning. ### Claim 12 (depends on 1) The method of claim 1, further comprising: generating, by the computer system, guidance information for guiding a human operator during the assembly process based on the context for the current stage of the assembly process. ### Claim 13 (depends on 12) The method of claim 12, wherein generating the guidance information comprises: generating, by the computer system, current stage guidance for use by the human operator to perform one or more tasks during the current stage of the assembly process. ### Claim 14 (depends on 12) The method of claim 12, wherein generating, by the computer system, the guidance information comprises: generating, by the computer system, next stage guidance for use by the human operator to perform one or more tasks during a next stage of the assembly process. ### Claim 15 (depends on 12) The method of claim 12, further comprising: displaying, by the computer system, the guidance information on a display system positioned relative to the assembly site, wherein the guidance information includes instructions for use by the human operator to perform at least one task in the assembly process during the current stage. ### Claim 16 (depends on 1) The method of claim 1, further comprising: storing, by the computer system, the quality report as part of a record corresponding to the current stage of the assembly process in a data repository to thereby build a history of records for the assembly process. ### Claim 17 (depends on 1) The method of claim 1, further comprising: determining, by the computer system, whether the quality report for the current stage of the assembly process indicates that an issue of interest is present; and rolling back through the history of records, via the computer system, to identify a root cause of the issue of interest. ### Claim 18 (depends on 17) The method of claim 17, wherein rolling back through the history of records comprises: analyzing, by the computer system, a first record corresponding to a selected stage of the assembly process immediately prior to the current stage of the assembly process. ### Claim 19 (depends on 18) The method of claim 18, wherein rolling back through the history of records further comprises: determining, by the computer system, whether the root cause occurred during the selected stage; and analyzing, by the computer system, a second record corresponding to another stage of the assembly process immediately prior to the selected stage. ### Claim 20 A method for performing automated supervision and inspection of an assembly process, the method being implemented using a computer system and comprising: generating sensor data at an assembly site using a sensor system positioned relative to the assembly site; identifying, by the computer system, a current stage of a plurality of stages in the assembly process for building an assembly at the assembly site using the sensor data, assembly information stored in a data repository, and at least one of machine learning, deep learning, computer vision, a customized learning technique, computational learning, or artificial intelligence learning, wherein the assembly information includes assembly process flow data that includes information about a sequence of the plurality of stages; and wherein identifying the current stage includes detecting a plurality of objects using the sensor data and matching at least a portion of the plurality of objects to parts for the assembly such that the portion of the plurality of objects matched to the parts is a plurality of matched objects; identifying, by the computer system, a context for the current stage; generating, by the computer system, a quality report for the assembly corresponding to the current stage based on the sensor data and the context for the current stage, wherein the quality report includes an overall stage assessment that indicates a quality of the current stage of the assembly process; and storing, by the computer system, a record corresponding to the current stage of the assembly process in the data repository to thereby build a history of records for the assembly process, wherein the record includes at least a portion of the sensor data, an identification of the plurality of objects, an identification of the plurality of matched objects, and the quality report. ### Claim 21 (depends on 20) The method of claim 20, further comprising: determining, by the computer system, whether the quality report for the current stage of the assembly process indicates that at least one issue of interest is present, wherein the at least one issue of interest is selected from one of an improper part, an improper part location, an improper orientation, a crack, a dent, a twist, and a surface inconsistency. ### Claim 22 (depends on 21) The method of claim 21, further comprising: rolling back through the history of records, via the computer system, to identify a root cause of the at least one issue of interest. ### Claim 23 (depends on 20) The method of claim 20, further comprising: generating, by the computer system, at least one of current stage guidance or next stage guidance for use by a human operator during the assembly process. ### Claim 24 (depends on 23) The method of claim 23, further comprising: displaying, by the computer system, the at least one of the current stage guidance or the next stage guidance on a display system. ### Claim 25 (depends on 20) The method of claim 20, further comprising: presenting, by the computer system, at least a portion of the quality report on a display system, wherein the overall stage assessment includes a grade that indicates how the current stage has progressed with respect to selected tolerances. ### Claim 26 A system for automated supervision and inspection of an assembly process, the system comprising: a sensor system positioned relative to an assembly site and an assembly being built at the assembly site; and a computer system that: identifies a current stage of a plurality of stages in the assembly process for building the assembly at the assembly site based on sensor data generated by the sensor system and assembly process flow data that includes information about a sequence of the plurality of stages, wherein identifying the current stage includes detecting a plurality of objects using the sensor data and matching at least a portion of the plurality of objects to parts for the assembly such that the portion of the plurality of objects matched to the parts is a plurality of matched objects; identifies a context for the current stage; generates a quality report for the assembly corresponding to the current stage based on the sensor data and the context for the current stage, and stores a record corresponding to the current stage of the assembly process in a data repository to thereby build a history of records for the assembly process, wherein the record includes at least a portion of the sensor data, the quality report, an identification of the plurality of objects, and an identification of the plurality of matched objects. ### Claim 27 (depends on 26) The system of claim 26, wherein the sensor system includes at least one camera positioned relative to the assembly site and wherein the camera generates two-dimensional imaging data. ### Claim 28 (depends on 26) The system of claim 26, wherein the computer system identifies the current stage of the assembly process using the sensor data, assembly information stored in a data repository, and at least one of machine learning, deep learning, computer vision, a customized learning technique, computational learning, or artificial intelligence learning. ### Claim 29 (depends on 28) The system of claim 28, wherein the assembly information includes at least one of a part images library, a part models library, assembly process flow data, assembly site model data, assembly model data, assembly video data, or feature data. ### Claim 30 (depends on 26) The system of claim 26, wherein the context includes at least one of a nominal condition for the assembly at the current stage, a nominal condition for each part expected to be present at the current stage, selected tolerances for the nominal condition for the assembly at the current stage, an identification of steps completed prior to the current stage, an identification of steps to be completed during the current stage, or an identification of steps to be completed after the current stage.