# US 2025/0086079 A1 · Method and apparatus with test result reliability verification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0086079 A1 - Document ID: US20250086079A1 - Status: Published application - Kind: A1 - Application number: 18/828,022 - Filed: 2024-09-09 - Published: 2025-03-13 - Earliest claimed: 2023-09-08 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Keunseo KIM; Jaecheol LEE; Sangkyu PARK; Sangdo PARK; Dokyoung KIM; Minkyu YANG; Young LEE; Jinsic JANG - Web page: https://patentgrep.com/patent/US20250086079A1 ## CPC - G06F 11/2273: {Test methods} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method for verifying reliability of a test for products performed by test equipment includes: receiving result images generated from preprocessing of result data of the test, the result data of the test including labels for a plurality of scale levels and the received result images including first result images belonging to a first scale level of the plurality of scale levels and second result images belonging to a second scale level of the plurality of scale levels; making a first determination, from the first result images, whether the first scale level is normal or abnormal; making a second determination, from the second result images, whether the second scale level is normal or abnormal; and determining that no error occurred in the test in response to both the first scale level and the second scale level being determined to be normal; or determining that an error occurred in the test in response to at least one scale level being determined to be abnormal. ## Claims ### Claim 1 A method for verifying reliability of a test for products performed by test equipment, the method comprising: receiving result images generated from preprocessing of result data of the test, the result data of the test including labels for a plurality of scale levels and the received result images including first result images belonging to a first scale level of the plurality of scale levels and second result images belonging to a second scale level of the plurality of scale levels; making a first determination, from the first result images, whether the first scale level is normal or abnormal; making a second determination, from the second result images, whether the second scale level is normal or abnormal; and determining that no error occurred in the test in response to both the first scale level and the second scale level being determined to be normal; or determining that an error occurred in the test in response to at least one scale level being determined to be abnormal. ### Claim 2 (depends on 1) The method of claim 1, wherein each label includes an identifier of a scale level. ### Claim 3 (depends on 2) The method of claim 2, further comprising classifying the result images by their corresponding scale levels according to the labels. ### Claim 4 (depends on 1) The method of claim 1, wherein element areas of the result images comprise respective values, each value representing an individual test result of a corresponding product. ### Claim 5 (depends on 1) The method of claim 1, wherein: the making the first determination is based on a first feature extracted from the first result images by a neural network; and the making the second determination is based on a second feature extracted from the second result images by the neural network or by another neural network. ### Claim 6 (depends on 5) The method of claim 5, wherein the neural network comprises a convolution neural network (CNN) or a transformer network. ### Claim 7 (depends on 5) The method of claim 5, wherein the making the first determination comprises: calculating a statistic indicating a distance measure between a distribution of features of a ground truth image corresponding to the first scale level and a distribution of features of the first result images corresponding to the first scale level; and making the first determination by comparing the statistic with a predetermined reference value. ### Claim 8 (depends on 7) The method of claim 7, wherein the comparing the statistic with the predetermined reference value comprises determining that the first result images satisfies a first condition based on determining that the statistic is greater than the predetermined reference value. ### Claim 9 (depends on 7) The method of claim 7, wherein the ground truth image is derived from ground truth test result data of a prior test performed by the test equipment. ### Claim 10 An apparatus for verifying reliability of a test for products performed by test equipment, the apparatus comprising: one or more processors and a memory storing instructions configured to cause the one or more processors to perform a process comprising: receiving result images generated from preprocessing of result data of the test, the result data including labels for a plurality of scale levels and the received result images including first result images belonging to a first scale level of the plurality of scale levels and second result images belonging to a second scale level of the plurality of scale levels; making a first determination, from the first result images, whether the first scale level is normal or abnormal; making a second determination, from the second result images, whether the second scale level is normal or abnormal; and determining whether an error occurred in the test based on the first determination and the second determination. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein each label includes an identifier of a scale level. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the process further comprises classifying the result images by their corresponding scale levels according to the labels. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the result data comprises device-under-test (dut) maps respectively associated with the labels, wherein each of the dut maps comprises element areas each comprising a value determined by the test equipment during performance of the test. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein: the making the first determination is based on a first feature extracted from the first result images by a neural network; and the making the second determination is based on a second feature extracted from the second result images by the neural network or by another neural network. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the neural network comprises a convolution neural network (CNN) or a transformer network. ### Claim 16 (depends on 14) The apparatus of claim 14, wherein the making the first determination comprises: calculating a statistic indicating a distance measure between a distribution of features of a ground truth image corresponding to the first scale level and a distribution of features of the first result images corresponding to the first scale level; and making the first determination by comparing the statistic with a predetermined reference value. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the comparing the statistic with the predetermined reference value comprises: determining that the first result images satisfies a first condition based on determining that the statistic is greater than the predetermined reference value. ### Claim 18 (depends on 16) The apparatus of claim 16, wherein the ground truth image is derived from ground test result data of a prior test performed by the test equipment. ### Claim 19 A system for verifying reliability of a test of products, the system comprising: test computers each configured to perform the test on a respective set of the products and each test computer collecting a corresponding result map having element areas indicating individual test results respectively corresponding products; and a reliability verification computer configured to determine whether a reliability condition of the test is satisfied based on the result maps and based on scale levels of the result maps, the scale levels corresponding to hierarchical scales in testing the products. ### Claim 20 (depends on 19) The system of claim 19, wherein the hierarchal scales includes at least two of: a lot-level scale representing a production block of the products, a board-level scale of boards on which the products are installed for the test, a slot-level scale for the boards to interface with the test computers, and an equipment-level scale of the test computers performing the test on the products installed on the boards.