Back to results
US 10,255,286 B2Granted
File metadata handler for storage and parallel processing of files in a distributed file system, and associated systems and methods
AssigneeSoftware AG
InventorsSatish Shankar, Sanju Krishnan Retnakumari
Life of this patent
Estimating the term…
Abstract
Certain example embodiments described herein relate to a File Metadata Handler that helps an integrated managed file transfer solution perform processing in connection with complex distributed file systems, while still cooperating with more straightforward and/or conventional file systems, servers, and/or the like. The File Metadata Handler of certain example embodiments may be able to work with, or take the place of, a more conventional file handler adapter, e.g., to provide possibly required pre-processing and/or additional file processing at runtime for more complex file systems, transport protocols, and/or the like.
Claims16
A method for enabling the accessing and/or processing of data stored in one or more computer systems in connection with a file metadata handler that has built-in support for a plurality of different file formats, the method comprising:
- receiving a request to create a file metadata definition for the file metadata handler, the file metadata definition being configurable for a first file format, and wherein files in the first file format have associated metadata for which processing logic is required to be defined in order to process such files;
- linking the received file metadata definition with an identifier that identifies files of the first file format, the first file format is a distributed file system file format used in an associated distributed file system;
- receiving format information that includes information about elements that are available to files having the first file format;
- updating the file metadata definition with at least one of received transaction boundary information including at least start and end elements for transaction boundaries, data validation information, data transformation requirements, including one or more conditions related to the elements, to be linked to the elements, or linkages between encrypted protocols and decryption tools for the first file format;
- storing the updated file metadata definition in a non-transitory computer readable storage medium, in enabling the file metadata handler to access and/or process data in the first file format;
- and
- deploying a plug-in to a map reduce framework associated with the distributed file system, the map reduce framework being used in accessing and/or manipulating files stored in the associated distributed file system.
4 more elements
A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor of a computer system, cause the computer system to:
- receive a request to create a file metadata definition for the file metadata handler, the file metadata definition being configurable for a first file format, and wherein files in the first file format have associated metadata for which processing logic is required to be defined in order to process such files;
- link the received file metadata definition with an identifier that identifies files of the first file format, the first file format is a distributed file system file format used in an associated distributed file system;
- receive format information that includes information about elements that are available to files having the first file format;
- update the file metadata definition with at least one of received transaction boundary information including at least start and end elements for transaction boundaries, data validation information, data transformation requirements, including one or more conditions related to the elements, to be linked to the elements, or linkages between encrypted protocols and decryption tools for the first file format;
- store the updated file metadata definition in a non-transitory computer readable storage medium, in enabling the file metadata handler to access and/or process data in the first file format;
- and
- deploy a plug-in to a map reduce framework associated with the distributed file system, the map reduce framework being used in accessing and/or manipulating files stored in the associated distributed file system.
4 more elements
A method of processing files in connection with a server, the method comprising:
- providing to a client computing device a listing of objects from a plurality of different computer systems that have different respective file systems and/or are accessible via different respective transport protocols, the listing organizing the objects in a common, integrated view of virtual folders;
- receiving, from the client computing device, a request to process an object that is stored remote from the client computer device;
- determining whether the server has built-in support for processing the object, based on the file system to be accessed, and the transport protocol to be used, in processing the request to process the object;
- in response to a determination that the server has built-in support for processing the object, processing the request to process the object using the built-in support of the server;
- and
- in response to a determination that the server does not have built-in support for processing the object:
- determining a file format for the object based on how the object is named,
- accessing from a data store a file metadata definition for the determined file format, the file metadata definition including format information about elements that are available in the determined file format and at least one of (a) transaction boundary information including at least start and end elements for transaction boundaries, (b) data validation information, (c) data transformation requirements, including one or more conditions related to the elements, to be linked to the elements, or (d) linkages between encrypted protocols and decryption tools, and
- processing the request to process the object using the accessed file metadata definition,
- wherein the server does not have built-in support for processing objects stored on one or more distributed file systems, and wherein the data store includes file metadata definitions for each said distributed file system,
- wherein the request to process the object relates to an object stored on a distributed file system for which the data store includes a file metadata definition, and
- wherein the method further comprises consulting a plug-in deployed to a map reduce framework associated with the distributed file system, the map reduce framework being used in processing files stored in the associated distributed file system.
9 more elements
A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor of a computer system, cause the computer system to:
- provide to a client computing device a listing of objects from a plurality of different computer systems that have different respective file systems and/or are accessible via different respective transport protocols, the listing organizing the objects in a common, integrated view of virtual folders;
- receive, from the client computing device, a request to process an object that is stored remote from the client computer device;
- determine whether the server has built-in support for processing the object, based on the file system to be accessed, and the transport protocol to be used, in processing the request to process the object;
- in response to a determination that the server has built-in support for processing the object, process the request to process the object using the built-in support of the server;
- and
- in response to a determination that the server does not have built-in support for processing the object:
- determine a file format for the object based on how the object is named,
- access from a data store a file metadata definition for the determined file format, the file metadata definition including format information about elements that are available in the determined file format and at least one of (a) transaction boundary information including at least start and end elements for transaction boundaries, (b) data validation information, (c) data transformation requirements, including one or more conditions related to the elements, to be linked to the elements, or (d) linkages between encrypted protocols and decryption tools, and
- process the request to process the object using the accessed file metadata definition,
- wherein the server does not have built-in support for processing objects stored on one or more distributed file systems, and wherein the data store includes file metadata definitions for each said distributed file system,
- wherein the request to process the object relates to an object stored on a distributed file system for which the data store includes a file metadata definition, and
- wherein the computer system being further caused to consult a plug-in deployed to a map reduce framework associated with the distributed file system, the map reduce framework being used in processing files stored in the associated distributed file system.
9 more elements
Dependencies are inferred from the claim text.