The absolute key to getting agility and extensibility in a data warehouse project is a good meta data based automation framework and tools, regardless of whether you are using TSQL, Pool, Databricks and/or Python.
If you are coming to SqlBits 2022, then there are a few sessions touching on this subjects. Full Schedule on this link https://arcade.sqlbits.com/sessions/
Thursday 10th March 11:30am
Andre Kamman – Building your first Metadata Driven Azure Data Factory
This should be a good intro to some of the feature of ADF that help empower automation over earlier tools like SSIS.
Thurs 10th March, 12:00
Lessons in Lakehouse Automation
Simon Whitely is talking about his year journey in Lakehouse automation.
Id love to see that, but I’m talking at the same time on SQL Storage Fundamentals 😉
Thurs 10th March, 17:10
Creating a Metadata Driven Orchestration Framework Using Azure Data Integration Pipelines
Paul shows his open source framework for Azure Data Integration pipelines. This promises to be a great session and Pauls pipeline framework looks to be quite mature and well thought out.
Sat 12th March 14:40
EDW Automation in Synapse SQL Pools
I’ll be showing a meta data driven TSQL approach to automating the Star Schema lifecycle using dedicated Sql Pools, meta data and ADF.
All focus is on the EDW process less so much on ADF.
Sat 12th March 14:40
The “Meta-verse” – Using metadata to build your pipelines in less than 20 min
Ricardo Lopez will demonstrate how you can use metadata based frameworks to build a data pipeline that loads data from source to data lake to synapse in less than 20 min!
