Introducing Azure Function Activity to Data Factory

Case
Last month Microsoft released the new Azure Function Activity for Azure Data Factory (ADF). This should make it easier to integrate Azure Function code in Data Factory. How does it work?

Azure Function activity in Azure Data Factory

Solution
Using Azure Functions (like other API's) was already possible via the Web Activity, but now ADF has its own activity which should make the integration be even better.

Azure Functions via the Web Activity

For a project at a client we where already using the Web Activity for a Azure Function with an HTTP trigger and to use this code with the new activity, we did need to slightly change the code. Note that I'm not a die hard .NET developer, but I got some help from my colleague Walter ter Maten.

1) HttpResponseMessage to IActionResult
The standard function did return an HttpResponseMessage, but for Azure Function Activity in the ADF pipeline it needs to return a json object. Below you find a very simple and partial code purely to show the differences in the function call and the return code. Also notice the extra reference/using.

Partial old code with the HttpResponseMessage (Web Activity only):

//C# Code
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Host;

namespace Calculate
{   
    public static class Calculate
    {
        [FunctionName("SumNumbers")]
        public static async Task<HttpResponseMessage> SumNumbersFunction([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
        {
            int number1 = 0;
            int number2 = 0;

            // Log start
            log.Info("C# HTTP trigger function processed a request.");

            // Get request body
            dynamic data = await req.Content.ReadAsAsync<object>();
            number1 = data?.number1 ?? 0;
            number2 = data?.number2 ?? 0;

            // Log parameters
            log.Info("Number1:" + number1.ToString());
            log.Info("Number2:" + number2.ToString());

            return req.CreateResponse(HttpStatusCode.OK, "The result is " + (number1 + number2).ToString());
        }
    }
}

Partial new code with the IActionResult (Azure Function Activity and Web Activity):

//C# Code
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.AspNetCore.Mvc;  // Added

namespace Calculate
{   
    public static class Calculate
    {
        [FunctionName("SumNumbers")]
        public static async Task<IActionResult> SumNumbersFunction([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
        {
            int number1 = 0;
            int number2 = 0;

            // Log start
            log.Info("C# HTTP trigger function processed a request.");

            // Get request body
            dynamic data = await req.Content.ReadAsAsync<object>();
            number1 = data?.number1 ?? 0;
            number2 = data?.number2 ?? 0;

            // Log parameters
            log.Info("Number1:" + number1.ToString());
            log.Info("Number2:" + number2.ToString());

            return new OkObjectResult(new { Result = "OK" });
        }
    }
}

We also needed to install the Microsoft.AspNetCore.Mvc NuGet Package. In the tools menu from Visual Studio you will find the NuGet Package Manager. Then search for this package and click the install button. After that you can add the new Using (Microsoft.AspNetCore.Mvc) in the code.

Bijschrift toevoegen

2) Get Function Key and App URL
In the next step we need to specify the Function URL and the Function Key. You can find them in the Azure Portal and then locate the specific function.

Function App URL

For the Function Key you need to go to Manage in you function and then copy the Function Key.

Function Key

3) Azure Function Activity
Now you can replace the Web Activity by an Azure Function Activity. Drag the new activity to the pipeline canvas and give it a suitable name. Next create a new Azure Function Linked Server. This is where you need the two strings from the previous step. After that enter the name of the function you want to execute, choose the Method and fill in the Body (copy it from the Web Activity).
Now delete the Web Activity and debug the pipeline to check the Function. Also check the Monitor (log) of your Azure Function to see whether it was successfully executed.

Replace Web Activity by Azure Function Activity

Summary
In this post you saw how to use the new Azure Function activity in the Azure Data Factory pipeline. It is fairly easy, but we did have to make a small change to the code. If you don't change the code you will get this error:
{
    "errorCode": "3600",
    "message": "Error calling the endpoint.",
    "failureType": "UserError",
    "target": "Sum this"
}
For now I did not see any improvements other than that you now can use the Function Key in Azure Functions which also allows you to revoke access for a specific key.