Configuring Azure Arc VM’s Telemetry Collection with Terraform

As I mentioned in the previous post, the main reason I wanted to connect a custom VM to Azure Arc was that I could start telemetry collection. So today let’s do just that. This post relies on the resources that we created the last time, so if something looks like it came out of nowhere – check out the first post of this series.

VM Telemetry in Azure

Azure’s approach to VM’s telemetry collection is called VM Insights. It’s actually way more powerful than just a telemetry collector, but at this stage we just need a few CPU/Disk graphs to be happy.

Enabling VM Insights through UI is easy. We’d go to Insights blade of the VM, hit a friendly “Enable” button, and after 10 minutes or so we’d discover some dashboards with beautiful graphs on them. However, hitting buttons is not the Jedi way. Terraform is.

The button of weak. True Jedi uses Infrastructure as a Code.

Building blocks of VM Insights

It is still worth to hit that “Enable” button at least once, to see what’s going on under the hood. It is interesting, because at least four things are happening:

  • Azure creates Log Analytics workspace (LA) resource, which will act as a database for our telemetry.
  • It will also add a Data Collection Rule (DCR), which will specify what kind of data we are interested in (Microsoft Insights, Performance Counters), and where we want to send it to (Log Analytics workspace).
  • A virtual machine, that we enabled insights for, will receive an extension called Azure Monitor Agent (AMA), whose sole responsibility is to extract the metrics and send them to whatever Data Collection Rule we associated it with.
  • Finally, Azure will add an association between the collection rule and the agent, thus completing the path between metrics and their storage.

Configuring VM Insights with Terraform

Knowing the building blocks, it doesn’t require much of a brain power to automate them. However, there is a catch. We can create LA workspace and DCR upfront, no problem. Creating AMA and the association is a completely different story, though. Most likely there will be more than one Arc virtual machine to monitor, and we have no control of when exactly they will appear in Azure. All that makes “the right time and place” to create AMA and DCR association quite unknown.

But there is a solution – Azure Policy. The policy defines the desired configuration of resources, and Azure will constantly seek for violations to rectify (if told so). In other words, if we create policies that require every Arc machine to have AMA extension with DCR association, whenever a new VM comes along, Azure will detect missing properties and configures them in automatically. Profit!

In fact, we don’t even need to create those policies. Azure already has them:

  • “Configure Linux Arc-enabled machines to run Azure Monitor Agent” for AMA.
  • “Configure Linux Machines to be associated with a Data Collection Rule or a Data Collection Endpoint” for DCR.

We just need to assign them to the resource group with future Arc machines, and the science will happen.

Builtin policies for monitoring (one of the many)

Steps 1-2: Create Log Analytics Workspace and Data Collection Rule

Creating LA workspace is trivial:

We put LA into the same resource group as Arc machines, since they should share the same region.

Data Collection Rule is more complex. Mainly because it’s surprisingly hard to find a proper documentation about its settings. I ended up creating DCR manually, examining its properties in JSON, and then replicating the findings in Terraform.

Most of the configuration actually makes sense. destinations-log_analytics points to an LA workspace we declared above. That’s logical. data_flow block is also more or less straightforward – we just pass data from streams to destinations as is.

data_sources would’ve made the perfect sense, too, if it was backed by some documentation. Well, at least creating it manually and “reverse engineering” still works.

Either way, this configuration is valid, and we can go to the next steps – policies, that will connect these resources to Arc VMs.

Steps 3-4: Create Policies for AMA and DCR Association

It’s time for a little bit of magic! First, let’s add references to existing monitoring policies.

Then, assign those policies to our resource group with Arc machine. The first policy will install Azure Monitor Agent extension to machines that don’t have it:

And the other will create a Data Collection Rule Association between the VM and the DCR:

In addition to usual policy stuff, these two have “SystemAssigned” managed identities. Since the policies will actually change the state of resources, they must have some permissions. And permissions imply some identity to be granted to. Thus, the identity {} block.

To update Arc VM, the policies need “Azure Connected Machine Resource Administrator” role. DCR association policy also needs to be “Monitoring Contributor”:

A Test Drive

I put new resources to file, along with and from the last time, and run that terraform apply command:

Policy enforcement action kicked in 2 minutes after we gave it permissions to do so

Policy assignments appear right away, but it takes some time for Azure Policy engine to discover that our existing Arc machine is in violation. However, in 5–30 minutes you’ll notice that Azure Monitoring Agent extension appeared in Arc VM’the’s extensions list, and a few more minutes later there will be some data in VM’s built-in dashboard. Victory!

Azure Monitor Linux Agent extension, assigned to Arc Machine via Azure Policy
Builtin performance dashboard with CPU, Memory and many other metrics.


Hopefully by now you are convinced, that hybrid clouds are actually something useful. Of course, Digital Ocean, whose VM we connected to Azure Arc, has its own monitoring extensions, and monitoring it with Azure didn’t add much. But I have two comments to that.

First, Azure Monitoring is not the only thing we could’ve enabled. Just as easy we could’ve enabled logging, desired state configuration, update management and even real time protection to that VM. I don’t believe you could’ve done that with vanilla Digital Ocean.

And second, I chose Digital Ocean just for demonstration. The cloud provider that I actually used in real life doesn’t support built-in monitoring, so AMA extension alone was a huge win. If we were pursuing, for example, FedRAMP certification, I’d just point my approved collection of Azure extensions to that third-party Arc VM, and it would magically become compliant in minutes.

So yes, hybrid cloud and Azure Arc in particular is a pretty useful thing. I have spoken.

Leave a Reply

Your email address will not be published. Required fields are marked *