Performance of API for GraphQL Web Activity vs Script Activity in Fabric Data Pipelines
This blog will show the comparison between executing a stored procedure using web activity for API for GraphQL and script activity. A stored procedure called usp_PipelineQueue which will return a list of pipelines for orchestration. The stored procedure is stored in Fabric SQL Database which serves as a metadata database. Prerequisite: Already configured item of…
What is Microsoft Fabric API for GraphQL?
As a data engineer, I constantly seek efficient ways to query data and improve accessibility, making it more valuable for data-driven businesses. If you’re interested in optimizing data workflows, this blog is for you. First, let’s explore what GraphQL is. What is GraphQL? GraphQL is an API query language that enables precise data retrieval, minimizing…
Connecting to Fabric SqlEndpoints using AAD/Entra Token in Notebooks
Fabric Notebooks give great connectivity options to the default Lakehouse for Spark and SparkSQL, but they are more limited when it comes to connecting securely and smoothly to a SqlEndpoint, be that LH or DW. We demand two principles Sample code below and also on GitHub link below. Paste this into a notebook which is…
Solution – Using Logical Names for DW in Fabric
Currently the Fabric TSQL end point for a workspace is seen like server and you can connect to it via SSMS or other. The big problem is the connection is made using the GUID of the workspace and Tennant, which makes it almost impossible to know if you are connecting to PRD or DEV. For…
Why SSMS is still king over ADS and Fabric Explorer?
I work with a lot of teams of folks with 10-20 years experience of TSQL development and one common factor is they all still use use SSMS over Azure Data Studio (ADS) and the newer web based Fabric Explorer and script editor. The web experience has put a lot of effort into features like intellisense…
Fabric Wish List for DWA
As you may have seen from some of my blog posts, videos or talks, we put a lot of thought and time into Data Warehouse Automation Frameworks (DWA) and recently completed out V1 of Fabric DWAhttps://prodata.ie/fabric-dwa/ During the thousands of hours R&D we came across a lot of things that frustrated us, or could have…
Should you use Datatypes on Staging Tables in Fabric ?
When you extract a file into Fabric (LH or DW) Table from a CSV, you can leave columns as the default varchar(8000), or with additional work apply the proper data types. Some questions we seek to answer In this post we are exploring the above using a python notebook to load a 7GB CSV file…
Integrating Sharepoint Online with OneLake in Fabric
Its common to have spreadsheets in SharePoint and want to ingest them into OneLake in Fabric. Typically this could be master data, mappings, budgets or any type of data that does not reside within a core enterprise system. We’ve developed a Python class that can be used in Fabric Notebooks that wraps the Graph API…
How to create a service principal to automate Fabric
Service Principals are essential to using the Fabric REST API for automation. In this blog well show you how to create a service principle that can be used to automate API calls to PowerBI such as refresh. Once you have created the service principle the next article shows you how to use the service principle to…
Fabric DW Automation Now Available!
We launched today our Fabric DWA Framework Accelerator. This is the result of over 3,500 hours of work migrating our Synapse DWA Framework to Fabric and also re-designing the template library system to make it extensible enough to handle any and all Fabric artefacts like Pipelines, Dataflows, Notebooks as well as traditional SQL Procedures and…
Sequential Identity in Fabric
Why do we need sequential identity in data warehousing? Currently Fabric doesnt support Identity columns. Even if/when it does, I assume that the numbers won’t be sequential as most MPPs such as SqlPools generate different ranges per “node” as per Synapse, resulting in numbers that may be unique but look somewhat random. https://azurede.com/2021/07/31/comparing-sql-server-and-synapse-analytics-identity-columns/ First up,…
Success with Azure Synapse Analytics: Mastering Data Warehousing
In today’s data-driven world, organizations must effectively manage and utilize vast amounts of information to gain insights and stay competitive. To achieve this, companies are increasingly turning to data warehouses, a centralized repository that stores and organizes structured and unstructured data from various sources.
Something went wrong. Please refresh the page and/or try again.
always on Azure Cloud consolidation Cube DW DWA EDW ERP Events Fabric Fabric DW Featured Finance Financial Reporting Licencing Load Test Maestro Many to Many MasterClass MCM mdx Modelling MOLAP OLAP Partitioning Performance Personal Power BI PowerPivot scom SQL SQLBits sqldw SQL IO SQL Saturday SQL Server SSAS SSIS storage Syanpse Synapse Tabular TSQL Virtualization
Subscribe to Prodata Blog
Get new content delivered directly to your inbox.