Sunday, 8 December 2019

Azure Data Factory Webhook activity calls Runbook

Case
The Azure Data Factory (ADF) Web activity calls my runbooks asynchronous (execute and don't wait for an answer). Is there a way to do this synchronous and even provide feedback to ADF in case it fails?
ADF Webhook activity























Solution
This year Microsoft added the Webhook activity to ADF. This works nearly the same as the existing Web activity, but it calls the webhook of your runbook synchronous and expects a callback to let ADF know it is ready. With some additional code you can adjust your runbook and make it work for both the Web and Webhook activity.

In a previous post we showed you how to retrieve parameters from ADF to an Azure Automation runbook. This code should be somewhere in the top of your code.
# Parameters
Param
(
    # Get webhook data
    [Parameter(Mandatory=$False,Position=1)]
    [object] $WebhookData
)
 
# Get all parameters from body 
# (passed from Data Factory Web Activity)
$Parameters = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)

We can reuse this code to also extract the parameter 'callBackUri'. This parameter is only provided by the Webhook activity and not by the Web activity. By asking whether you can retrieve this parameter you can distinguish a call between the Webhook and the Web activity.
# Read and store the callBackUri which
# is only provided by the Webhook activity
If ($Parameters.callBackUri)
{
    $callBackUri = $Parameters.callBackUri
}

The Webhook activity waits for a callback that the script is ready. Therefore we need to add some code at the bottom of your script. If you forget this step then the Webhook activity waits unit the timeout occurs and then fail your pipeline. So this step is mandatory for the Webhook, but cannot be used for the Web activity since it doesn't provide a call back uri.
# Let the Webhook activity know that the script is ready
# Otherwise its waits unit its timeout
If ($callBackUri)
{
    Invoke-WebRequest -Uri $callBackUri -Method Post
}
By default the Webhook activity has a timeout of 10 minutes. For scripts executing more than 10 minutes (like resuming the Integration Runtime in ADF) you should raise this number.
Timeout of 10 minutes























An other handy option is to provide some feedback to the Webhook activity. For example when something fails in your script or when you expect certain parameters that have not been provided. This saves you some time to search for what went wrong in your runbook activity log. You have to provide the error details via a JSON message:
# Let the Webhook activity know that the script is ready
# Otherwise it waits unit its timeout
If ($callBackUri)
{
    # Create an error message
    # Message and statuscode will show up in ADF
    $body = [ordered]@{
    error = @{
    ErrorCode = "ParameterError"
    Message = "Required parameters where not provided"
    }
    statusCode = "404"
    }

    # Convert the string into a real JSON-formatted string
    $bodyJson = $body | ConvertTo-Json

    # Call back with error message in body and a JSON contenttype
    Invoke-WebRequest -Uri $callBackUri -Method Post -Body $bodyJson -ContentType "application/json"
}

To enable this functionality you have to turn on the option 'Report status on callback' in the ADF Webhook activity. ADF will ignore your message when you do not enable it.
Report status on callback























The error message and statuscode will show up in ADF pipeline monitor.
Runbook error message shown in ADF









You can provide various HTTP status codes in the JSON message to give your error message a certain category. For example:
  • 401 - Unauthorized
  • 403 - Forbidden
  • 404 - Not Found
  • 429 - Too Many Requests
  • 500 - Internal Server Error
  • 503 - Service Unavailable

A complete list of HTTP status codes can be found here. All status codes from 400 and above are errors.

Summary
In this post we showed you how to use the ADF Webhook activity benefits in an Azure Automation runbook and also still make it work for calling via the ADF Web activity. The benefits are synchronously calling your runbook and providing error details in case of a failure.

6 comments:

  1. Great blog post. Something to add is when invoking the callBackURI we need to add -UseBasicParsing. This is discussed in this StackOverflow thread: https://stackoverflow.com/questions/38005341/the-response-content-cannot-be-parsed-because-the-internet-explorer-engine-is-no

    ReplyDelete
    Replies
    1. This is my experience as well, for some reason my callback worked fine for a while until it didn't. Adding -UseBasicParsing to the command worked for me!

      Delete
  2. if something fails in the runbook( for eg: table does not exist which i want to refresh) but still webhook gets suceeded.Ideally it should show the error message which is there in the runbook error logs. How to achieve this. I believe, some code needs to be added in the powershell which can pass the error message of runbook into ADF webhook activity

    ReplyDelete
    Replies
    1. Did you create an error message and use it in the callback (Invoke-WebRequest)?

      Delete
  3. I want the error message of the runbook to get paased to the webhook activity. Now, what's happening is that even if the runbook is getting failed(in my case, the table name is not correct) but still webhook activity is getting succeeded

    ReplyDelete
    Replies
    1. Did you check the 'Report status on callback'?

      Delete

Related Posts Plugin for WordPress, Blogger...