# US 12,112,263 B2 · Reversal-point-based detection and ranking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,263 B2 - Document ID: US12112263B2 - Status: Granted - Kind: B2 - Application number: 17/116,184 - Filed: 2020-12-09 - Published: 2022-06-09 - Granted: 2024-10-08 - Earliest claimed: 2020-12-09 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Bo Yang; Chaofan Huang; Songtao Guo; Robert Perrin Reeves; Wan Qi Gao; Patrick Ryan Driscoll; Kristina Caroline Ryan; Michael Mario Jennings; Jeremy Lwanga; Manzarul Azad Kazi - Web page: https://patentgrep.com/patent/US12112263B2 ## CPC - G06N 3/08: Learning methods - G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation - G06F 18/2163: {Partitioning the feature space} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/25: Fusion techniques - G06F 18/29: {Graphical models, e.g. Bayesian networks} - G06N 3/09: Supervised learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In an example embodiment, a model is trained to specifically identify reversal points in data and then to rank these reversal points in order of importance. A reversal point shall be defined as a point in which a particular metric, specifically a first order derivative, crosses over from positive to negative or vice-versa. Users are more likely to be interested in abnormal and significant changes in data, and thus the machine-learned model is trained to evaluate a reversal point based on two dimensions: abnormality and significance. ## Claims ### Claim 1 A system for training and using a machine learned model, comprising: a non-transitory computer-readable medium having instructions stored thereon, which, when executed by a processor, cause the system to perform operations comprising: obtaining time series data, the time series data including a value for a first metric at each of a plurality of time points separated by time intervals; identifying, in the time series data, a plurality of reversal points, a reversal point being a time point at which the value for the first metric changed from positive to negative or vice-versa; obtaining training data from one or more databases; training a machine learned model, using the training data as input to a machine learning algorithm, to generate a ranking score for an input reversal point, the ranking score being based on: calculating an abnormality score by: determining a difference between a length of a trend, prior to the input reversal point, in a segment of the time series containing the input reversal point and a total number of reversal points in a plurality of reversal points in the segment, wherein a trend is a sequence of time points at which the value for the first metric maintained a same sign; dividing the difference by a value computed based on a total number of time points in the segment; calculating a significance score by: determining a maximum absolute value of differences between a magnitude change at the input reversal point and those magnitude changes in the trend in the segment prior to the input reversal point, applying a constant β to the maximum absolute value, β learned via the training of the machine learning model; applying a first weight to the abnormality score and a value related to the first weight to the significance score; and combining the weighted abnormality score and weighted significance score into the ranking score; evaluating one or more of the plurality of reversal points by passing each of the one or more reversal points to the machine learned model, thus outputting a ranking score for each of the one or more of the plurality of reversal points; ranking the one or more of the plurality of reversal points based on their corresponding ranking scores; and highlighting one or more of the reversal points in a graphical user interface based on the ranking. ### Claim 2 (depends on 1) The system of claim 1, wherein the operations further comprise retraining the machine learned model based on user feedback. ### Claim 3 (depends on 1) The system of claim 1, wherein the applying the first weight is performed using the following formula: Sranking=α*Sabnormality+(1−α)*Ssignificance wherein S is the ranking score, Sabnormality is the abnormality score, Ssignificance is the significance score, and α is the first weight. ### Claim 4 (depends on 3) The system of claim 3, wherein S abnormality = 1 2 + R - C 2 ⁢ N - 4 where R is the previous trend length, C is the total number of reversal points in the plurality of reversal points, and N is the total number of time points in the time series data. ### Claim 5 (depends on 3) The system of claim 3, wherein S significance = β * S magnitude - diff + ( 1 - β ) * S magnitude - diff - ratio ⁢ where ⁢ β ⁢ is ⁢ a ⁢ second ⁢ weight , S magnitude - diff = erf ( Δ frp - max 2 - 1 σ ⁢ 2 ) ⁢ and ⁢ S magnitude - diff - ratio = min ( Δ frp - max 2 H r , 1 ) where erf is an error function, σ is a standard deviation of Δ2frp-max, and Hr is a size of a company for which the time series data applies. ### Claim 6 (depends on 1) The system of claim 1, wherein the one or more highlighted reversal points are highlighted on a graph representing a second metric different than the first metric. ### Claim 7 (depends on 1) The system of claim 1, wherein the training data includes features extracted from user profiles and user interaction data corresponding to the user profiles and the training data is transformed prior to it being used to train the machine learned model. ### Claim 8 (depends on 1) The system of claim 1, wherein the machine learning algorithm is a neural network. ### Claim 9 A computerized method comprising: obtaining time series data, the time series data including a value for a first metric at each of a plurality of time points separated by time intervals; identifying, in the time series data, a plurality of reversal points, a reversal point being a time point at which the value for the first metric changed from positive to negative or vice-versa; obtaining training data from one or more databases; training a machine learned model, using the training data as input to a machine learning algorithm, to generate a ranking score for an input reversal point, the ranking score being based on: calculating an abnormality score by: determining a difference between a length of a trend, prior to the input reversal point, in a segment of the time series containing the input reversal point and a total number of reversal points in a plurality of reversal points in the segment, wherein a trend is a sequence of time points at which the value for the first metric maintained a same sign; dividing the difference by a value computed based on a total number of time points in the segment; calculating a significance score by: determining a maximum absolute value of differences between a magnitude change at the input reversal point and those magnitude changes in the trend in the segment prior to the input reversal point, applying a constant β to the maximum absolute value, β learned via the training of the machine learning model; applying a first weight to the abnormality score and a value related to the first weight to the significance score; and combining the weighted abnormality score and weighted significance score into the ranking score; evaluating one or more of the plurality of reversal points by passing each of the one or more reversal points to the machine learned model, thus outputting a ranking score for each of the one or more of the plurality of reversal points; ranking the one or more of the plurality of reversal points based on their corresponding ranking scores; and highlighting one or more of the reversal points in a graphical user interface based on the ranking. ### Claim 10 (depends on 9) The method of claim 9, further comprising retraining the machine learned model based on user feedback. ### Claim 11 (depends on 9) The method of claim 9, wherein the applying the first weight is performed using the following formula: Sranking=α*Sabnormality+(1−α)*Ssignificance wherein S is the ranking score, Sabnormality is the abnormality score, Ssignificance is the significance score, and α is the first weight. ### Claim 12 (depends on 11) The method of claim 11, wherein S abnormality = 1 2 + R - C 2 ⁢ N - 4 where R is the previous trend length, C is the total number of reversal points in the plurality of reversal points, and N is the total number of time points in the time series data. ### Claim 13 (depends on 12) The method of claim 12, wherein S significance = β * S magnitude - diff + ( 1 - β ) * S magnitude - diff - ratio ⁢ where ⁢ β ⁢ is ⁢ a ⁢ second ⁢ weight , S magnitude - diff = erf ( Δ frp - max 2 - 1 σ ⁢ 2 ) ⁢ and ⁢ S magnitude - diff - ratio = min ( Δ frp - max 2 H r , 1 ) where erf is an error function, σ is a standard deviation of Δ2frp-max, and Hr is a size of a company for which the time series data applies. ### Claim 14 (depends on 9) The method of claim 9, wherein the one or more highlighted reversal points are highlighted on a graph representing a second metric different than the first metric. ### Claim 15 (depends on 9) The method of claim 9, wherein the training data includes features extracted from user profiles and user interaction data corresponding to the user profiles and the training data is transformed prior to it being used to train the machine learned model. ### Claim 16 (depends on 9) The system of claim 9, wherein the machine learning algorithm is a neural network. ### Claim 17 A non-transitory machine-readable storage medium comprising instructions which, when implemented by one or more machines, cause the one or more machines to perform operations comprising: obtaining time series data, the time series data including a value for a first metric at each of a plurality of time points separated by time intervals; identifying, in the time series data, a plurality of reversal points, a reversal point being a time point at which the value for the first metric changed from positive to negative or vice-versa; obtaining training data from one or more databases; training a machine learned model, using the training data as input to a machine learning algorithm, to generate a ranking score for an input reversal point, the ranking score being based on: calculating an abnormality score by: determining a difference between a length of a trend, prior to the input reversal point, in a segment of the time series containing the input reversal point and a total number of reversal points in a plurality of reversal points in the segment, wherein a trend is a sequence of time points at which the value for the first metric maintained a same sign; dividing the difference by a value computed based on a total number of time points in the segment; calculating a significance score by: determining a maximum absolute value of differences between a magnitude change at the input reversal point and those magnitude changes in the trend in the segment prior to the input reversal point, applying a constant β to the maximum absolute value, β learned via the training of the machine learning model; applying a first weight to the abnormality score and a value related to the first weight to the significance score; and combining the weighted abnormality score and weighted significance score into the ranking score; evaluating one or more of the plurality of reversal points by passing each of the one or more reversal points to the machine learned model, thus outputting a ranking score for each of the one or more of the plurality of reversal points; ranking the one or more of the plurality of reversal points based on their corresponding ranking scores; and highlighting one or more of the reversal points in a graphical user interface based on the ranking. ### Claim 18 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein the operations further comprise retraining the machine learned model based on user feedback.