US 11,768,954 B2Granted

System, method and computer-accessible medium for capturing data changes

AssigneeCapital One Services, LLC

InventorsMayur Jagtap, Naga Venkata Sriram Vadakattu, Abhijit Chitnis, Janardhan Deepak Prabhakara, Anurag Jain, Parvesh Kumar, Rahul Surendra Nath, Behdad Forghani, Mark Assousa

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

The exemplary embodiments provide real-time data capture and processing which improves data processing performance and speed and facilitate passing of the processed data to various analytical sources, while maintaining superior data quality checks, particularly with respect to data elements associated with multiple data types. The proposed system and process can be used to continuously consume and listen to multiple events while mapping the events to appropriate schemas provided in a separate schema stream. The schema stream is provided once and cached to minimize bandwidth consumed by the transaction stream. The schema information is then further enriched with information from a metadata registry. The event data may then be compressed and aligned in memory tables based on the enriched schema. Once events are decoded and sorted into memory tables in accordance to the identified schema, each memory table can be processed in parallel.

Claims18

Dependent claims

A non-transitory computer-accessible medium having stored thereon computer-executable instructions for optimal batch processing of transaction streams in real-time, wherein, when a computer arrangement executes the instructions, the computer arrangement is configured to perform procedures comprising:

decomposing each transaction in a transaction stream into one or more discrete events, wherein the transaction stream is decomposed into a plurality discrete events;
determining a fingerprint for each of the plurality of discrete events, by mapping one or more event data items into shorter bit string that uniquely identifies a corresponding event data item;
receiving a plurality of schemas in a scheme stream transmitted via a distinct schema channel, wherein the plurality of schemas are transmitted once and stored in a cache memory;
determining at least one schema for each of the plurality of discrete events based on a corresponding fingerprint, wherein a change in one or more data values associated with a discrete event results in a new schema;
compressing, two or more discrete events, corresponding to a common row, into a single discrete event, the common row being identified based on the at least one schema, wherein the at least one schema is enhanced with one or more supplemental metadata provided by a metadata registry;
storing the plurality of discrete events across a parallel arrangement of one or more source tables, based on a schema hash value associated with the at least one schema;
scanning, in parallel, one or more source tables for one or more untokenized data elements corresponding to a sensitive information record, wherein each of the one or more source tables is associated with a distinct schema hash value one or more discrete events;
and
tokenizing the one or more untokenized data in parallel across the one or more source tables;
and
writing back the one or more tokenized data to one or more corresponding source tables in parallel.

8 more elements

A method for real-time capturing of data changes in a transaction stream, the method comprising:

decomposing each transaction in a transaction stream into one or more discrete events, wherein the transaction stream is decomposed into a plurality discrete events;
determining a fingerprint for each of the plurality of discrete events, by mapping one or more event data items into shorter bit string that uniquely identifies a corresponding event data item;
receiving a plurality of schemas in a scheme stream transmitted via a distinct schema channel, wherein the plurality of schemas are transmitted once and stored in a cache memory;
determining at least one schema for each of the plurality of discrete events based on a corresponding fingerprint, wherein a change in one or more data values associated with a discrete event results in a new schema;
compressing, two or more of discrete events, corresponding to a common row, into a single discrete event, the common row being identified based on the at least one schema, wherein the at least one schema is enhanced with one or more supplemental metadata provided by a metadata registry;
storing the plurality discrete events across a parallel arrangement of one or more source tables, based on a schema hash value associated with the at least one schema;
scanning, in parallel, one or more source tables for one or more untokenized data elements corresponding to a sensitive information record, wherein each of the one or more source tables is associated with a distinct schema hash value one or more discrete events;
and
tokenizing the one or more untokenized data in parallel across the one or more source tables;
and
writing back the one or more tokenized data to one or more corresponding source tables in parallel.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/902,535Filed 16 Jun 2020
Published: US 2021/0390204 A116 Dec 2021
Granted: 26 Sep 2023
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

G06F 21/6245
To a system of files or objects, e.g. local or distributed file system or database: protecting personal data, e.g. for financial or medical purposes
G06F 16/1734
Details of further file system functions: details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
G06F 16/1865
File system types: transactional file systems
G06F 16/211
Design, administration or maintenance of databases: schema design and management