I'm retrieving sensitive secrets from Azure Key Vault in an Azure Data Factory pipeline and using those values to call an other service. Both the output and the input with the secret value are showing in the Data Factory log. Is there a way to prevent logging sensitive inputs or outputs in ADF?
 |
| Secret unintentionally showed in log |
Solution
The solution is very simpel. Almost all activities have a Secure input and Secure output option on the General tab. When checking the Secure input box none of the input will show up in the logs and when checking the Secure output box none of the output will be logged. This is very useful for not logging passwords and other secrets used in your pipeline.
 |
Now we are running the same pipeline again with the Secure output option turned on and now the secret won't be visible in the log.
 |
| No more output |
And you can of course still use the output in an subsequent activity with an expression. Just be sure to hide the input of that activity. Note that some activities like Set variable don't have the options to hide input and output.
 |
| You can still use hidden output |
Summary
In this post you learned how to hide sensitive input and output from activities so that they don't show up in the Azure Data Factory logging. The Key Vault example used in this post is explained here.
