On 15th November 2023 we launched our Fabric DWA Framework Accelerator. This is the result of over 3,500 hours of work migrating our Synapse DWA Framework to Fabric and also re-designing the template library system to make it extensible enough to handle any and all Fabric artefacts like Pipelines, Dataflows, Notebooks as well as traditional SQL Procedures and Views.
This framework not only saves months of development time by delivering an out of the box data warehouse accelerator but also is highly extensible and agile due to its re-use of templates to guarantee high supportability and flexibility.
Videos to explore concepts
Top Fabric DWA Features
1. Fabric Data Engineering Templates
We supply pre-built and tested python notebook and pipeline templates for the most common formats and end points such as:
- CSV
- Excel
- SharePoint
- Database
- Parquet
- SFTP
- Web API
We supply blank starter templates to extend for any other data source supported by Fabric or python. Additionally we have a long list of ERP and other solutions we have integrated with such as:
- SAP ERP
- Infor M3
- Oracle ERP
- Oracle Fusion PPM
- Workday
- Greenhouse
- Exchequer
- Agresso
- QuickBooks
2. Meta Data Driven Declarative Platform
Once you have a declarative template, all development is meta data driven. This allows data movement in the ingest and extract phases to be fully automated without any need for development.
Our table design allow for inheritance, so that common properties such as format and location can be defined for a group of pipelines, helping to eliminate any repetition (or DRY as the Architects would say)
3. Flexible Architecture
Microsoft Fabric offers a lot of choice in DW Architecture, especially around tools used to transform and query data. We support:
- Both Lakehouse and Datawarehouse engines. Our standard reference architecture uses a hybrid approach with lakehouse for staging data and datawarehouse for presenting and transforming a star schema.
- Full tool support for Notebooks, Pipelines, Dataflows, Azure functions and SQL. We recommend focusing on notebooks and SQL to minimise complexity and productivity.
4. World Class Extensibility
While the framework ships with many standard notebook templates, most projects have some non-standard formats which need a fresh template developed in any of the supported artefacts. These custom templates can be plugged into the Framework simply by registering meta data without any refactoring or code changes.
5. Star Schema Automation with Data Virtualisation
Moving from raw staged data to a clean star schema perfectly suitable for analytics is what we call the T and the L part of ETL. (Transform and Load).
While the framework supports using any template to make star schemas we also supply an out of the box solution for star schema automation from staging tables. This largely comprises using SQL Views to implement business mappings and calculations with automation for core function such as:
- Creating Physical Star Schema Tables
- Schema change and migration
- Business to Surrogate key handling
- Efficient loading using a wide range of approaches
A quick 10 minute demo of this process is shown below
6. Out of the Box Enterprise features
Use of a DWA Framework standardises and frees developers from writing common yet critical enterprise features such as:
- Logging on key activities. We log meta data activities to SQL tables, and also provide analytical templates for using Fabric logs to monitor pipeline performance.
- Password Management. All secrets are stored in Azure KeyVault. Allowing secure audited and policy based access.
- Data Lineage. Fabric itself support schema lineage to show how data flows from notebook to data warehouse to Power BI. We support full data lineage which tracks every row back to its source files for full auditability.
7. Dynamic Orchestration
Many DW solutions use hard coded orchestration to sequence activities such as what files to extract and load for a give business unit. Often this is developed in Pipelines.
The DWA Framework supports fully dynamic orchestration by grouping pipelines into units we call package groups. A package group can be dynamically edited to include/exclude pipelines without any code changes or code deployments.
8. Scalable and Parallel
All templates can be configured to control whether they run in sequence or parallel via their metadata. This allows for any amount of scale out that Fabric can supply to be harnassed.
9. Build 100% on Microsoft Fabric
The DWA Framework is simply an accelerator that is pre-built using Fabric components to install in your workspace, and brings your our 30 years experience as a head start in your project.
There is no license for any additional technology or or other licensing except for Fabric required.
10. Full EDW Lifecycle coverage
The DWA Framework doesnt just cover ETL it covers all the stages of a DW such as:
- Ingest
- Extract
- Transform
- Load
- Refresh (Power BI)
- Export
- Egress
11. Its Free !
We don’t charge for the Prodata DWA Framework. Merely for our consultancy services to help install, configure, and train you in its use, and if required, any build support.