# US 12,008,007 B2 · Labeling system and method for time-series data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,007 B2 - Document ID: US12008007B2 - Status: Granted - Kind: B2 - Application number: 17/854,578 - Filed: 2022-06-30 - Published: 2023-01-12 - Granted: 2024-06-11 - Earliest claimed: 2021-07-07 - Assignees: Vingroup Joint Stock Company - Applicants: Vingroup Joint Stock Company - Inventors: Quang Ngoc Khuat; Minh Hieu Nguyen; Quy William Lee - Web page: https://patentgrep.com/patent/US12008007B2 ## CPC - G06F 16/2477: {Temporal data queries} - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A labeling method includes determining whether there is an intersection between the first annotated segment and the second annotated segment; if Yes, determining whether the first annotated segment and the second annotated segment have different labels from each other; if Yes, sending the label of the first annotated segment to the second agent unit and the label of the second annotated segment to the first agent unit; if No, determining whether a discrepancy-over-union ratio between the first annotated segment and the second annotated segment is below a first threshold; if Yes, determining the intersection of the first annotated segment and the second annotated segment as a final annotated segment for the piece of time-series data; if No, assigning a review tag to a union segment of the piece of time-series data which has a duration from a start of the first annotated segment to an end of the second annotated segment. ## Claims ### Claim 1 A labeling method comprising: S1: feeding a piece of time-series data to a first agent unit and a second agent unit; S2: annotating, by the first agent unit, the piece of time-series data to generate a first annotated piece of time-series data with at least one first annotated segment; S3: annotating, by the second agent unit, the piece of time-series data to generate a second annotated piece of time-series data with at least one second annotated segment; and S4: processing, by a labeling apparatus, the first annotated piece of time-series data and the second annotated piece of time-series data, wherein the processing comprises: S41: determining whether there is an intersection between the first annotated segment and the second annotated segment, in response to determining that there is an intersection between the first annotated segment and the second annotated segment, proceeding to S42, and in response to determining that there is not an intersection between the first annotated segment and the second annotated segment, ending the processing; S42: determining whether the first annotated segment and the second annotated segment have different labels from each other, in response to determining that the first annotated segment and the second annotated segment have different labels from each other, proceeding to S43, and in response to determining that the first annotated segment and the second annotated segment do not have different labels from each other, proceeding to S44; S43: sending the label of the first annotated segment to the second agent unit and the label of the second annotated segment to the first agent unit; S44: determining whether a discrepancy-over-union ratio between the first annotated segment and the second annotated segment is below a first threshold, in response to determining that the discrepancy-over-union ratio between the first annotated segment and the second annotated segment is below the first threshold, proceeding to S45, and in response to determining that the discrepancy-over-union ratio between the first annotated segment and the second annotated segment is not below the first threshold, proceeding to S46; S45: determining the intersection of the first annotated segment and the second annotated segment as a final annotated segment for the piece of time-series data; and S46: assigning a review tag to a union segment of the piece of time-series data which has a duration from a start of the first annotated segment to an end of the second annotated segment. ### Claim 2 (depends on 1) The labeling method of claim 1, wherein the method further comprises S48 of recommending, after determining whether there is an intersection between each first annotated segment and each second annotated segment, at least one stand-alone segment of one unit of the first and second agent units, which do not intersect with any segment of the other unit of the first and second agent units, to one or more units of the first and second agent units as recommend tags. ### Claim 3 (depends on 1) The labeling method of claim 1, wherein the steps S2 and S3 for annotating the piece of time-series data are performed by artificial intelligence (AI) models. ### Claim 4 (depends on 1) The labeling method of claim 1, wherein the steps S2 and S3 for annotating the piece of time-series data are performed by users. ### Claim 5 (depends on 1) The labeling method of claim 1, wherein the first agent unit and the second agent unit include artificial intelligence (AI) models for assisting the user on reviewing the disparity between user works on the piece of time-series data. ### Claim 6 (depends on 1) The labeling method of claim 1, wherein the step S43 further comprises: returning to S2 to annotate the piece of time-series data based on the label of the second annotated segment sent to the first agent unit; and returning to S3 to annotate the piece of time-series data based on the label of the first annotated segment sent to the second agent unit. ### Claim 7 (depends on 1) The labeling method of claim 1, wherein the step S46 further comprises: returning to S2 to annotate the piece of time-series data based on the assigned review tag; and returning to S3 to annotate the piece of time-series data based on the assigned review tag. ### Claim 8 (depends on 1) The labeling method of claim 1, wherein the discrepancy-over-union ratio between the first annotated segment and the second annotated segment is computed using the following equation: XoU = ( ❘ "\[LeftBracketingBar]" start 1 - start 2 ❘ "\[RightBracketingBar]" + ❘ "\[LeftBracketingBar]" end 1 - end 2 ❘ "\[RightBracketingBar]" ) max ⁢ ( max ⁢ ( end 1 , end 2 ) - min ⁢ ( start 1 , start 2 ) , 10 - 6 ) where, starts is the start of the first annotated segment, start2 is the start of the second annotated segment, ends is the end of the first annotated segment, and end2 is the end of the second annotated segment. ### Claim 9 A labeling system comprising: a first agent unit is configured to annotate a piece of time-series data to generate a first annotated piece of time-series data with at least one first annotated segment; a second agent unit is configured to annotate the piece of time-series data to generate a second annotated piece of time-series data with at least one second annotated segment; and a labeling apparatus is configured to process the first annotated piece of time-series data and the second annotated piece of time-series data, wherein the processing comprises: S41: determining whether there is an intersection between the first annotated segment and the second annotated segment, in response to determining that there is an intersection between the first annotated segment and the second annotated segment, proceeding to S42, and in response to determining that there is not an intersection between the first annotated segment and the second annotated segment, ending the processing; S42: determining whether the first annotated segment and the second annotated segment have different labels from each other, in response to determining that the first annotated segment and the second annotated segment have different labels from each other, proceeding to S43, and in response to determining that the first annotated segment and the second annotated segment do not have different labels from each other, proceeding to S44; S43: sending the label of the first annotated segment to the second agent unit and the label of the second annotated segment to the first agent unit; S44: determining whether a discrepancy-over-union ratio between the first annotated segment and the second annotated segment is below a first threshold, in response to determining that a discrepancy-over-union ratio between the first annotated segment and the second annotated segment is below a first threshold, proceeding to S45, and in response to determining that a discrepancy-over-union ratio between the first annotated segment and the second annotated segment is not below a first threshold, proceeding to S46; S45: determining the intersection of the first annotated segment and the second annotated segment as a final annotated segment for the piece of time-series data; and S46: assigning a review tag to a union segment of the piece of time-series data which has a duration from a start of the first annotated segment to an end of the second annotated segment. ### Claim 10 (depends on 9) The labeling system of claim 9, wherein the system is further configured to recommend, after determining whether there is an intersection between each first annotated segment and each second annotated segment, at least one stand-alone segment of one unit of the first and second agent units, which do not intersect with any segment of the other unit of the first and second agent units, to one or more units of the first and second agent units as recommend tags. ### Claim 11 (depends on 9) The labeling system of claim 9, wherein the first agent unit and second agent unit using artificial intelligence (AI) models for annotating the piece of time-series data. ### Claim 12 (depends on 9) The labeling system of claim 9, wherein the first agent unit and second agent unit are used by users for annotating the piece of time-series data. ### Claim 13 (depends on 9) The labeling system of claim 9, wherein the first agent unit and the second agent unit include artificial intelligence (AI) models for assisting the user on reviewing the disparity between user works on the piece of time-series data. ### Claim 14 (depends on 9) The labeling system of claim 9, wherein the first agent unit is configured to annotate the piece of time-series data based on the label of the second annotated segment sent to the first agent unit; and the second agent unit is configured to annotate the piece of time-series data based on the label of the first annotated segment sent to the second agent unit. ### Claim 15 (depends on 9) The labeling system of claim 9, wherein the first agent unit is configured to annotate the piece of time-series data based on the assigned review tag; and the second agent unit is configured to annotate the piece of time-series data based on the assigned review tag. ### Claim 16 (depends on 9) The labeling system of claim 9, wherein the discrepancy-over-union ratio between the first annotated segment and the second annotated segment is computed using the following equation: XoU = ( ❘ "\[LeftBracketingBar]" start 1 - start 2 ❘ "\[RightBracketingBar]" + ❘ "\[LeftBracketingBar]" end 1 - end 2 ❘ "\[RightBracketingBar]" ) max ⁢ ( max ⁢ ( end 1 , end 2 ) - min ⁢ ( start 1 , start 2 ) , 10 - 6 ) where, starts is the start of the first annotated segment, start2 is the start of the second annotated segment, ends is the end of the first annotated segment, and end2 is the end of the second annotated segment. ### Claim 17 (depends on 1) A non-transitory computer-readable storage medium comprising instructions which, when executed by at least one processor of a machine, cause the machine to perform the labeling method of claim 1. ### Claim 18 A labeling method comprising: S1: feeding a piece of time-series data to a first agent unit and a second agent unit; S2: annotating, by the first agent unit, the piece of time-series data to generate a first annotated piece of time-series data with at least one first annotated segment; S3: annotating, by the second agent unit, the piece of time-series data to generate a second annotated piece of time-series data with at least one second annotated segment; and S4: processing, by a labeling apparatus, the first annotated piece of time-series data and the second annotated piece of time-series data, wherein the processing comprises: S41: determining whether there is an intersection between the first annotated segment and the second annotated segment, in response to determining that there is an intersection between the first annotated segment and the second annotated segment, proceeding to S42, and in response to determining that there is not an intersection between the first annotated segment and the second annotated segment, ending the processing; S42: determining whether the first annotated segment and the second annotated segment have different labels from each other, in response to determining that the first annotated segment and the second annotated segment have different labels from each other, proceeding to S43, and in response to determining that the first annotated segment and the second annotated segment do not have different labels from each other, proceeding to S44; S43: sending the label of the first annotated segment to the second agent unit and the label of the second annotated segment to the first agent unit; S44: determining whether an intersection-over-union ratio between the first annotated segment and the second annotated segment exceeds a second threshold, in response to determining that the intersection-over-union ratio between the first annotated segment and the second annotated segment exceeds a second threshold, proceeding to S45, and in response to determining that the intersection-over-union ratio between the first annotated segment and the second annotated segment does not exceed a second threshold, proceeding to S46; S45: determining the intersection of the first annotated segment and the second annotated segment as a final annotated segment for the piece of time-series data; and S46: assigning a review tag to a union segment of the piece of time-series data which has a duration from a start of the first annotated segment to an end of the second annotated segment. ### Claim 19 (depends on 18) A non-transitory computer-readable storage medium comprising instructions which, when executed by at least one processor of a machine, cause the machine to perform the labeling method of claim 18.