CI/CD for Azure Synapse Analytics – Part 1

Do you also wonder how to do continuous integration (CI) and continuous deployment (CD) for Synapse Analytics?  But first, lets talk about basic, what is CI/CD is simple terms. CI/CD is one of the best practices of agile methodology and it enables development teams to deploy stable and tested code changes more frequently.  This practice also knows as the CI/CD release pipelines.

Synapse Studio allow you to configure git repository to enable collaborations i.e it will associate your synapse workspace with a Git source control which will enable you to create release pipelines in Azure DevOps. Currently, Microsoft official documentation on release pipelines for Azure Synapse Analytics is very limited.

In this blog, we will talk about to how to setup source control on your Synapse workspace. Subsequently, how to build release pipelines to deploy changes from Development synapse workspace to TEST/PRD Synapse workspace.

We need only two things to get started:

1. At least two Synapse workspaces (i.e DEV and TEST/PRD)
2. Azure DevOps git repository

NOTE: Configure only your development Synapse workspace with Git integration. Changes to test and production workspaces are deployed via CI/CD and don’t need Git integration.

Step 1: Connect your Synapse workspace with your Git repository

Lets head over to https://web.azuresynapse.net/ and open your development Synapse Studio. After that click on Manage tab then find Git configuration under Source Control. As shown in screenshot below.

Step 2: Azure DevOps Git repository settings

Select repository type as Azure DevOps git from drop down list and select appropriate Azure Active Directory tenant and click continue.

Step 3: Configure a repository

Select the DevOps account from dropdown list that your orginaztion is using. We have created a new repository called SandboxSynapse and its same as the project name. You can have different names for project and repository as there is no specific standard. We prefer to keep Publish Branch name as default i.e workspace_publish. Once you click on Apply , your synapse workspace will be connected to Azure DevOps git repository.

Step 4: Publish Changes

Now, whenever you done any development work, make sure to click on Commit All and Publish, as this will sync all changes with Git rep.

It is important to sync changes only to your main collobration branch( master in this case).

Head over to your git repo, after you click publish button in Synapse Studio. You will see two branches one i.e master ( main collobration branch) it will contain all artificats created in your Synaspse studio and second will be workspace_publish branch. Azure Synapse Analytics will generate templates for synapse artifacts whenever you click Publish button and templates files will be synced in workspace_publish branch.

You can configure a custom publish branch, by adding publish_config.json file to the root folder in the collaboration branch. When publishing, Synapse Studio reads this file, looks for the field publishBranch, and saves workspace template files to the specified location. If the branch doesn’t exist, Synapse Studio will automatically create it. publish_config.json looks like this:

{
    "publishBranch": "workspace_publish"
}

In final part we will cover, how to create build and release pipelines for Azure Synapse Analytics. Stay Tuned!

If you have questions, feel free to drop us a line.