The client is a global healthcare provider that innovates medical technology. The client’s aim is to improve patient care and advance healthcare worldwide with their innovative products.
About the Client
Problem Statement
- Client objective: Enhancing data flow efficiency through implementation of a flexible and scalable data warehouse solution rooted in the Data Vault 2.0 framework, with the goals of boosting platform performance and reducing costs
- Compliance priority: Ensuring adherence to GDPR guidelines and specified data retention policies
- Model preference: Pursuing a hybrid operational model integrating a centralized core team with decentralized development teams to optimize efficiency and agility in project execution
The Challenge
The goal was to integrate a Data Vault 2.0 solution in an existing data platform facing these challenges:
- Transition existing domain-specific solutions to the updated Data Vault 2.0 platform
- Identify an automation tool to integrate diverse ad-hoc SQL and Python processes
- Develop a sophisticated access framework for object, row, and column-level permissions
- Address ETL challenges in the absence of a Persistent Staging Area (PSA) or data lake
The Solution
Scalefree supports the client by facilitating knowledge transfer and augmenting the team with additional manpower, focusing specifically on these solutions:
- Collaborating with the client to explore and select a suitable automation tool tailored to their requirements
- Introducing dbt cloud with Scalefree’s Data Vault 2.0 package, datavault4dbt, aided by Scalefree’s TurboVault4dbt for standardized development processes
- Crafting dbt macros to interpret an Access Control List (ACL) to implement the access framework
- Devising a consolidation strategy for the ETL process to establish a Persistent Staging Area (PSA)
Tangible Results for the Client
The result was a standardized data flow that ensured timely provision of accurate data to designated recipients, optimizing reporting systems by enhancing data quality and freshness. This framework ensured that the right data reaches the right consumers at the right time, bolstering decision-making processes and organizational efficiency.
Technologies involved
- Snowflake
- Dbt Cloud with datavault4dbt
- TurboVault4dbt
- Microsoft Azure DevOps
