Streaming data Azure & Power BI - Event Hubs

Case
I want to send streaming data to Power BI for reporting purposes. This post is the first part of a series on Streaming Data in Azure and Power BI. Here we will focus on the 'mailbox' service called Azure Event Hubs.

Azure Event Hubs

Solution

Azure Event Hubs is like a giant mailbox that can easily recieve loads of messages. One throughput unit can handle 1000 messages a second. However it can only recieve messages and not send messages it self. Other services like Azure Functions or Azure Stream Analytics can read the messages from this giant mailbox to process them. When the retention period expires the messages will automatically be deleted. 

Note: If want to recieve messages from IoT devices instead of events/messages from applications and services then you should have a look at Azure IoT hub. A similar mailbox service but specialized in IoT messages.

Posts in this series:

• Setting up Azure Event Hubs
• Sending test messages to the Event Hub
• Setting up hot path in Azure Stream Analytics
• Setting up cold pathin Azure Stream Analytics *soon
• Viewing streaming data in Power BI

1) Create an Azure Event Hubs Namespace

Lets create an Azure Event Hubs. Note that when you create an Azure Event Hubs via the Azure portal it will only create an Event Hubs Namespace. In the next step we need to add a Event Hub to that namespace.

• Besides the subscription and the resource group you first need to come up with a name for your namespace. This name is part of the URL and therefor only letters, numbers and hyphens (-) are allowed.
• Choose your Location wisely: pick a region close to rest of your platform (Power BI, the services/application sending messages, data lake, data warehouse or lake house). This helps to keep down the latency and costs for your real time platform.
• The Pricing tier depends on your needs. For testing purposes Basic is more than sufficient, unless you want to test Capture for which you need at least Standard.
  For production purposes you also need to take a look at the number of Consumer groups, the retention period and the number of events you need to handle.
• With the Throughput units you can scale up the capacity. One unit can recieve 1MB per second or a 1000 events per second. For this example we set it to 1.
• For testing purposes you can keep the rest of the settings to de default value.

Create Azure Event Hubs Namespace

New/empty Azure Event Hubs Namespace

2) Add Event Hub to namespace

Now we have an empty Event Hubs Namespace. We need to add an actual event hub to recieve messages. In the left menu go to Event Hubs located under Entities. Then click in the + Event Hub button.

+ Event Hub

Now think of a name for your Event Hub and set the retention time. In this case we set it to the max for Basic which is 24 hours. This means after 24 hours you will loose the messages. So make sure to process the messages within a day. If you didn't use the cheapest pricing tier then you also get an option to Capture the messages. This will be explained in a separate post, but it is basically a Stream Analytics Job saving all your messages in a data lake.

Adding an Event Hub to the Namespace

3) Add Shared access policies

To send messages to the Event Hub we need a shared access signature policy (SAS). Now go to the newly created Event Hub (callcenter in this example) and then to Shared access policies in the left menu under Setttings.

Shared Access Policies

Click on the +Add button to create a new one. In this case we want the callcenter application/service only to send messages. Therefore we also called it sendonly and checked only the Send checkbox.

Sendonly SAS policy

In the script we use the send test messages to the Event Hub we need the Primary or Secondary key of the newly created SAS Policy. You can click on it to retrieve them.

Retrieve primary key

4) Testing

To test the new Event Hub you need to start sending messages. The easiest way to test this is to write a little script in Python or PowerShell. We explained that in a separate blog post. When succesful you can see them arriving in your Event Hub.

Sending test messages

Conclusions

In this post we showed how easy it is to create an Azure Event Hub for a little test setup. The hardest part is to pick the right pricing tier. In most cases the Basic or Standard is probably enough. Within those pricing tiers you can scale up by adding extra Throughput units.

In the next post we will process the messages within the Event Hub via Azure Stream Analytics and sending them to Power BI.