CI/CD For Azure Synapse Analytics – Part 2

In CI/CD Azure Synapse Analytics – part 1 we have covered:

  • Setting up git source control in Synapse Studio
  • The difference in main collaboration and workspace publish branch

In this blog we are going to cover:

  • How to create build pipelines
  • Deploy to Synapse production workspace using DevOps release pipelines

Build Pipeline in DevOps

Go to you https://dev.azure.com/<YourOrgName&gt; i.e Azure DevOps portal and select your Synapse project which you have initialized earlier by configuring git in Synapse Studio.

In DevOps, Under the Pipelines tab select Build Pipelines (#2 ) and create a new pipeline using the classic editor ( we prefer to do this way). Next is to select source, for this case it is Azure Repos Git and do appropriate selection for Team Project, Repository and git branch which will used to create build artifacts that will used to deploy to another Synapse environment ( eg QA, PRD etc)

If your default publish branch is workspace_publish then select workspace_publish instead of main/master branch. Refer to step 3 in the last blog CI/CD Azure Synapse Analytics – part 1. On the next page for template selection, we will select the Empty Job link.

Add a new Powershell script task and use inline scripting to copy synapse publish templates from your git source. Make sure to modify PowerShell as per your development workspace name, in our case it is called pro-sand-dev.

ROBOCOPY.exe $(System.DefaultWorkingDirectory)/pro-sand-dev/ $(build.artifactstagingdirectory)/ /S /R:0 /W:0 
exit 0

Add Publish build artifacts task under Powershell scrip and keep default values for the Display name, Path to publish, Artifcate name, etc. Finally, you can click on Save & Queue.

The build is completed and the artifact is published with success. Now, we can use published artifact to create release pipelines for deployment

Release Pipelines for Deployment

Go back to the Azure DevOps project, select Releases under the pipelines tab and create New Release Pipeline. Click on Empty Job during template selection.

We will use artifact which we have published earlier using Build pipelines. Click on Add an artifact and select Build. In the project setting, select your current project and the name of the latest build artifact. Do ensure to select the latest as Default version and click Add.

Now we will add deployment tasks to our release pipeline. Cick Stage 1 ( 1job, 0 task) to add new task.

Click on the + button to add a new task. Search for ‘Synapse workspace’, you can directly add Synapse workspace deployment task if it’s already installed otherwise you can click on Synapse workspace deployment under Marketplace to install it in your Azure DevOps project.

Select the file path for Workspace Template and Template parameters from the published artifact drop folder. Under your Azure subscription select apprpoirate connection type. Select the resource group of the target Synapse workspace. Enter the Synapse workspace name located in the resource group that you specified.

Select the managed service connection type that you created to connect to the Synapse workspace. Click Save

Note by Microsoft on service connection: To configure new a service connection, select the Azure subscription from the list and click ‘Authorize’. If your subscription is not listed or if you want to use an existing service principal, you can setup an Azure service connection using the ‘Add’ or ‘Manage’ button.

Managed Service Identity service connection scope is limited to access granted to the Azure virtual machine running the agent.

Ensure that the VM has access to specified resources. Service connection scoped at Management Group level are visible only in Azure PowerShell task.

The only drawback of using the synapse deployment task is that it copies your Development environment default linked services to QA/PROD environments. But we don’t want to use DEV linked services in QA/PROD. To clean up this, we will add a new task Azure CLI.

Select script type as PowerShell and Script location as Inline script. Use the below script as per your DEV environment.

az synapse linked-service delete --workspace-name <QA/PRD WorkspaceName> --name pro-sand-dev-WorkspaceDefaultSqlServer --yes
az synapse linked-service delete --workspace-name <QA/PRD WorkspaceName> --name pro-sand-dev-WorkspaceDefaultStorage --yes

Once you are happy with all changes, hit Create Release as it will start deployment to QA/PROD environment.

We will cover Override ARM Parameters i.e ConnectionString, Storage account URL etc in a separate blog as it require special attention.

5 thoughts on “CI/CD For Azure Synapse Analytics – Part 2

  1. Hi, Deepak!
    Thank you for the tutorial. Very informative. However, there’s one step I’m having troubles with: I can’t delete the default linked services, as DevOps tells me “The linked service xxx-WorkspaceDefaultStorage is managed by the workspace xxx and cannot be deleted”. How did you fix this problem?
    Thank you in advance!

    1. Hi Latte,

      In Azure CLI task, Azure Resource Manage Connection ( its a service should have access to target synapse workspace. You need to open synapse studio and grant access ADMIN to service prinicipal of Service connection used in Azure CLI task.

      1. Thank you for your reply! Trouble is, It’s already registered as Synapse Administrator. It’s the very SP that deployed it in the first place. My personal user is also Synapse Administrator, but I’m also unable to delete it. In the GUI, the delete icon is not even there on these particular linked services.

  2. Hi Latte,

    Sorry for late reply. If you can’t see delete icon, I would suggest you to raise technical support ticket with MSFT.

  3. Hi Deepak,

    Can you deploy these arm templates that are generated from azure cli? Or is there any other way that you can take the pipelines from one Synapse workspace and deploy them to another Synapse workspace from Azure CLI?

    Thank you in advance

Leave a Reply to Deepak Gupta Cancel reply