One of the leading suppliers for military and commercial airplane structures and components sought an alternative to their data architecture as the one they had in place came with a significant cost.
About the Client
Problem Statement
To get sensitive data from SAP to target systems, the client was using an extremely expensive ETL tool and wanted to replace it with a more affordable one.
So the problem can be summarized as:
So the problem can be summarized as:
- Expensive legacy tool
- Lack of expertise within the organization with this particular tool
- Data privacy and regulatory compliance was not fully met
The Challenge
Finding and implementing the right tool in a proposed new architecture became a challenge due to the following reasons:
- Complex process and stream of data
- Lack of resources
- Highly regulated and sensitive data information
Therefore the project went through an extensive process of planning, implementation and testing to create a solution to the initial problem.
The Solution
By the end of the project, three solutions were provided or recommended in order to replace the existing legacy tool:
- Implementation of Enterprise Service Bus using Kafka message queues in Confluent
- Use of a tool called Mapforce for generating the data transformations
- Proposed construction of an EDW using Data Vault 2.0 to leverage NoSQL and agile development
Tangible Results for the Client
The major takeaway from the proof of concept and project implementation was the feasibility of the use of Kafka and Confluent to handle message queues in the Enterprise Service Bus which would significantly reduce the costs attributed to the architecture.
Technologies used
- Kafka + Confluent
- MapForce
- SAP
