In CI/CD Azure Synapse Analytics – part 1 we have covered:
- Setting up git source control in Synapse Studio
- The difference in main collaboration and workspace publish branch
In this blog we are going to cover:
- How to create build pipelines
- Deploy to Synapse production workspace using DevOps release pipelines
Build Pipeline in DevOps
Go to you https://dev.azure.com/<YourOrgName> i.e Azure DevOps portal and select your Synapse project which you have initialized earlier by configuring git in Synapse Studio.
In DevOps, Under the Pipelines tab select Build Pipelines (#2 ) and create a new pipeline using the classic editor ( we prefer to do this way). Next is to select source, for this case it is Azure Repos Git and do appropriate selection for Team Project, Repository and git branch which will used to create build artifacts that will used to deploy to another Synapse environment ( eg QA, PRD etc)
If your default publish branch is workspace_publish then select workspace_publish instead of main/master branch. Refer to step 3 in the last blog CI/CD Azure Synapse Analytics – part 1. On the next page for template selection, we will select the Empty Job link.
Add a new Powershell script task and use inline scripting to copy synapse publish templates from your git source. Make sure to modify PowerShell as per your development workspace name, in our case it is called pro-sand-dev.
ROBOCOPY.exe $(System.DefaultWorkingDirectory)/pro-sand-dev/ $(build.artifactstagingdirectory)/ /S /R:0 /W:0 exit 0
Add Publish build artifacts task under Powershell scrip and keep default values for the Display name, Path to publish, Artifcate name, etc. Finally, you can click on Save & Queue.
The build is completed and the artifact is published with success. Now, we can use published artifact to create release pipelines for deployment
Release Pipelines for Deployment
Go back to the Azure DevOps project, select Releases under the pipelines tab and create New Release Pipeline. Click on Empty Job during template selection.
We will use artifact which we have published earlier using Build pipelines. Click on Add an artifact and select Build. In the project setting, select your current project and the name of the latest build artifact. Do ensure to select the latest as Default version and click Add.
Now we will add deployment tasks to our release pipeline. Cick Stage 1 ( 1job, 0 task) to add new task.
Click on the + button to add a new task. Search for ‘Synapse workspace’, you can directly add Synapse workspace deployment task if it’s already installed otherwise you can click on Synapse workspace deployment under Marketplace to install it in your Azure DevOps project.
Select the file path for Workspace Template and Template parameters from the published artifact drop folder. Under your Azure subscription select apprpoirate connection type. Select the resource group of the target Synapse workspace. Enter the Synapse workspace name located in the resource group that you specified.
Select the managed service connection type that you created to connect to the Synapse workspace. Click Save
Note by Microsoft on service connection: To configure new a service connection, select the Azure subscription from the list and click ‘Authorize’. If your subscription is not listed or if you want to use an existing service principal, you can setup an Azure service connection using the ‘Add’ or ‘Manage’ button.
Managed Service Identity service connection scope is limited to access granted to the Azure virtual machine running the agent.
Ensure that the VM has access to specified resources. Service connection scoped at Management Group level are visible only in Azure PowerShell task.
The only drawback of using the synapse deployment task is that it copies your Development environment default linked services to QA/PROD environments. But we don’t want to use DEV linked services in QA/PROD. To clean up this, we will add a new task Azure CLI.
Select script type as PowerShell and Script location as Inline script. Use the below script as per your DEV environment.
az synapse linked-service delete --workspace-name <QA/PRD WorkspaceName> --name pro-sand-dev-WorkspaceDefaultSqlServer --yes az synapse linked-service delete --workspace-name <QA/PRD WorkspaceName> --name pro-sand-dev-WorkspaceDefaultStorage --yes
Once you are happy with all changes, hit Create Release as it will start deployment to QA/PROD environment.
We will cover Override ARM Parameters i.e ConnectionString, Storage account URL etc in a separate blog as it require special attention.