Introduction to Azure DevOps for Developers

Case
I want to store, build and release my code after development. How does this and more all come together in Azure DevOps?

Solution
First it is important to agree on a way of working, regardless of the tools that can help. Especially when you are part of a team. Consider, for example, the use of version control (if so, which branching strategy), test and deployment moments, code reviews and registration of activities.

For this solution, we distinguish the following activities for supporting and implementing a way of working:

1. Configure Git
2. Define a branching strategy
3. Define user stories 
4. Create and approve code reviews
5. Automatically build the code
6. Automatically release the code
7. Support testing
8. Capture release to production

How can we translate this into the use of the services in Azure DevOps? 

Well, lets have a look at the picture below.  

Azure DevOps - Using Pipelines, Repos, Boards and Test Plans

In this blog post we will go deeper into each aspect and which service can be used.

Repos

This service enables you to manage, monitor, build, commit, verify and push your project code including all the other information about your project.

1) Configure Git
Configure Git for version control and code collaboration. This includes everything from creating a repository, determining the collaboration branch and configuring branch policies such as minimum number of reviewers or code comment(s) needs to be processed.

2) Branching strategy
Define a branching strategy including "collaboration" and "feature" branches. This is a more detailed step, also how to handle "hotfixes" for instance, then only determine the collaboration branch. More information about the strategy around Git branching can be found here.

4) Create and approve code reviews
Create and approve code reviews using Pull Requests. This always ensures a four eyes principle. More information about Pull Requests can be found here.

8) Capture release to production
Create a Tag inside a Git repository after a successful release to production. This helps you to check the current or past production version at any time. It can happen for example that one of your stakeholders has other work to do and even after testing, he/she has questions about some functionality that was released the previous sprint and a new release has already taken place.

Boards
This service will manage your software projects and makes use of known frameworks like "Scrum" and "Kanban" that are supported by default.

3) Define user stories
Define user stories to describe the activities and link those user stories to the code. You link user stories to features and epics to group work.

Pipelines
This service will bring together Continuous Integration (CI) and Continuous Delivery (CD) of your code and application using pipelines.

5) Automatically build the code
One of the most important things is working code. In a technical way, we say that your code should "compile" or "build" successfully. In addition to this, you can also write unit tests and add those to your build pipeline. With this, you do not have to worry about running it manually or to forget a test, before your code will be merged into the collaboration branch. 

In a separate blog post we will show you how to setup a basic build.

6) Automatically release the code
After a successfully build, you want to release your code to a next environment. With Releases you can setup a pipeline to automate this process. 

In a separate blog post we will show you how to setup a basic release.

7) Support testing
Besides unit testing, that can be part of both build and release pipelines, you can also add functional testing. For example the Business Acceptance Test (BAT) when you are releasing to the acceptance environment. You can run the test cases, that you have stored in Azure Test Plans, manually or it can be automated as part of the release. More information about Test Plans can be found here.

Conclusion

In this post you saw a high overview of how you can use Azure DevOps as part of your development process and how it can helps to deliver your software (in an automated way) including testing. With all those different services and options it is a lot and it will take some time and initial investment, but it pays off in the long run. That is why it is important, apart from Azure DevOps, to pay attention for which way of working suits your team the best.

In next blog posts we will go deeper into certain services in Azure DevOps, such as the pipelines that contains a build and release.

Analyze Azure Data Factory logs - part 1: setup

Case
Azure Data Factory has a complete monitor that logs all details, but besides a simple filter it has no customization options and we don't want to add old fashioned custom logging to each pipeline with stored procedures to create our own logging. Is there a better alternative?

Azure Data Factory & Azure Log Analytics

Solution
You can use Azure Log Analytics Workspaces to store and analyze the logs of ADF. Within the Azure Market place you can find some standard dashboards to monitor and manage your Data Factory, but you can also write custom queries with Kusto Query Language (KQL). And even better use those custom queries in an Power BI report to create a (semi) live dashboard on your ADF logs.

In the first part of this series of blog posts we will focus on setting up the Log Analytics Workspace and adding the standard dashboards from the Azure market place. Prerequisites: an Azure Data Factory with a couple of pipelines that can run to generate some log data.

1) Create Log Analytics Workspace

For this example we will use a free trail workspace where the retention is limited to 7 days and the a max of 500mb of daily data ingestion. Perfect for a first introduction to Log Analytics.

• Go to the Azure Portal and create a new Log Analytics workspace with a describing name and in the same region as your Data Factory.
• On the Pricing tier tab choose Free and then hit the Review + Create button.

Create Log Analytics Workspace for ADF

2) Add Azure Data Factory Analytics from Azure Market Place

Next step is to add Data Factory Analytics from Azure Market Place. Note that at the time of writing it is stil in preview.

• Go to the URL and click on the Get It Now button.
• Select your Log Analytics Workspace and click on Create

Add ADF Analytics from Azure Market Place

3) Connect ADF to Log Analytics Workspace

Now we need to tell your Data Factory to send its logs to the new Log Analytics Workspace.

• Go to the ADF Overview page in the Azure Portal and click on Diagnostic settings in the left menu
• Click on + Add diagnostic settings
• Give the diagnotsic setting a name because you could create multiple ones: ADFDiagnostics
• Now first choose what you want to log. The first three are for ADF Activities, Pipelines and Triggers. The second group of items that start with Sandbox are the debug runs and the third group of items are for SSIS. For this example we will only select the first three.
• Secondly choose which metrics you want to log: AllMetrics
• And thirdly choose the destinations: Send to Log Analytics Workspace
• Now select you workspace and click on Save

Connect ADF to the Analytics Workspace

4) See standard dashboards
Now lets see that standard ADF dashboard in Azure Log Analytics, but first make sure you run a couple of ADF pipelines so that there is actually something to see on the dashboard because old runs won't be available. (Note: It could take a couple of minutes before you see the first data appear in Log Analytics)

In Log Analytics the dashboard is called a workbook. You can either directly go to all the workbooks and then search for "AzureDataFactoryAnalytics" or first go Solutions and then to the ADF solution and after that to the workbooks. The second way is a few more clicks, but the list of workbooks will be smaller.

• Go to Azure Log Analytics and click on Workbooks in the left menu
• In the list of public templates scroll down and click in AzureDataFactoryAnalytics
• When opened scroll down to see all visuals.
• After the first look you could hit the edit button and start customizing.

AzureDataFactoryAnalytics

ADF dashboard in Log Analytics

Conclusion

In this post you learned how to set up Log Analytics for ADF. Now you have that default workbook  (template) which you can adjust to your own needs, but you can also see which KQL queries Microsoft wrote to create those visuals. In a next blogpost we will explain the basics of KQL and show you how you can use those in Power BI to create an ADF dashboard. This could save you building custom log mechanisms because most information is already available in the ADF logs.