Analyze Azure Data Factory logs - part 1: setup

Case
Azure Data Factory has a complete monitor that logs all details, but besides a simple filter it has no customization options and we don't want to add old fashioned custom logging to each pipeline with stored procedures to create our own logging. Is there a better alternative?

Azure Data Factory & Azure Log Analytics

Solution
You can use Azure Log Analytics Workspaces to store and analyze the logs of ADF. Within the Azure Market place you can find some standard dashboards to monitor and manage your Data Factory, but you can also write custom queries with Kusto Query Language (KQL). And even better use those custom queries in an Power BI report to create a (semi) live dashboard on your ADF logs.

In the first part of this series of blog posts we will focus on setting up the Log Analytics Workspace and adding the standard dashboards from the Azure market place. Prerequisites: an Azure Data Factory with a couple of pipelines that can run to generate some log data.

1) Create Log Analytics Workspace

For this example we will use a free trail workspace where the retention is limited to 7 days and the a max of 500mb of daily data ingestion. Perfect for a first introduction to Log Analytics.

• Go to the Azure Portal and create a new Log Analytics workspace with a describing name and in the same region as your Data Factory.
• On the Pricing tier tab choose Free and then hit the Review + Create button.

Create Log Analytics Workspace for ADF

2) Add Azure Data Factory Analytics from Azure Market Place

Next step is to add Data Factory Analytics from Azure Market Place. Note that at the time of writing it is stil in preview.

• Go to the URL and click on the Get It Now button.
• Select your Log Analytics Workspace and click on Create

Add ADF Analytics from Azure Market Place

3) Connect ADF to Log Analytics Workspace

Now we need to tell your Data Factory to send its logs to the new Log Analytics Workspace.

• Go to the ADF Overview page in the Azure Portal and click on Diagnostic settings in the left menu
• Click on + Add diagnostic settings
• Give the diagnotsic setting a name because you could create multiple ones: ADFDiagnostics
• Now first choose what you want to log. The first three are for ADF Activities, Pipelines and Triggers. The second group of items that start with Sandbox are the debug runs and the third group of items are for SSIS. For this example we will only select the first three.
• Secondly choose which metrics you want to log: AllMetrics
• And thirdly choose the destinations: Send to Log Analytics Workspace
• Now select you workspace and click on Save

Connect ADF to the Analytics Workspace

4) See standard dashboards
Now lets see that standard ADF dashboard in Azure Log Analytics, but first make sure you run a couple of ADF pipelines so that there is actually something to see on the dashboard because old runs won't be available. (Note: It could take a couple of minutes before you see the first data appear in Log Analytics)

In Log Analytics the dashboard is called a workbook. You can either directly go to all the workbooks and then search for "AzureDataFactoryAnalytics" or first go Solutions and then to the ADF solution and after that to the workbooks. The second way is a few more clicks, but the list of workbooks will be smaller.

• Go to Azure Log Analytics and click on Workbooks in the left menu
• In the list of public templates scroll down and click in AzureDataFactoryAnalytics
• When opened scroll down to see all visuals.
• After the first look you could hit the edit button and start customizing.

AzureDataFactoryAnalytics

ADF dashboard in Log Analytics

Conclusion

In this post you learned how to set up Log Analytics for ADF. Now you have that default workbook  (template) which you can adjust to your own needs, but you can also see which KQL queries Microsoft wrote to create those visuals. In a next blogpost we will explain the basics of KQL and show you how you can use those in Power BI to create an ADF dashboard. This could save you building custom log mechanisms because most information is already available in the ADF logs.