Mihai-Albert.com

You want to retrieve a list with all your Azure VMs, complete with all their private and public IPs. You’d also like to get this fast, without having to start some script and come the next day to find the results (or worse, to discover that it errored out, and you only have 5 minutes left to produce the report). There are bits and pieces around the web – like this query that retrieves just one public IP per each VM – regardless if they have multiple assigned – but no private IP whatsoever. There are also Powershell scripts around, but they take too long or provide incomplete information.

And since Azure has, at this time, resources deployed using two possible models (ASM and ARM), you need to be careful about what you use to get each set of VMs, as the tools used to retrieve the info for one are incompatible with the other.

You might also want to query across thousands of VMs spread out in hundreds of Azure subscriptions that make up your tenant. How about a solution that takes less than a second to get all this information:

TL;DR Jump here to see how to extract all the Azure VMs + all their private/public IPs in a matter of seconds.

If you’re not in a rush, then let’s delve deeper into the topic and explore the following:

• Possible approaches that can be employed, and which one will be chosen and discussed in the article
• Look at Azure Resource Graph (ARG): what it is, why do we want to use it, and a brief example of using it
• Describe Kusto
• What’s in the scope of our search, and briefly discuss the 2 Azure deployment models
• Azure Resource Manager (ARM) model: derive the final ARG query bit by bit, in parallel with going over the networking concepts
• Azure Service Manager (ASM) model: build the final ARG query
• Wrappers around ARG, with a focus on Powershell and how to implement pagination correctly
• How can REST clients be used with ARG
• Throttling limits when working with ARG
• Possible delays in obtaining networking data in ARG
• Handling more than 1,000 Azure subscriptions with ARG
• Permissions required
• Two explicit ways of getting the list of all Azure VMs with all their private and public IPs via Azure Resource Graph (ARG)
• Alternative: Non-ARG Powershell Cmdlets
• Alternative: Non-ARG Azure CLI
• Q&A

Possible Approaches

Azure Portal can show – in the “Virtual machines” blade – both classic (ASM) and the regular ARM VMs by filtering either on “Virtual Machines (classic)” or “Virtual Machines“. Currently editing the columns does allow seeing one public IP of the machine, but you won’t get to see the 3 public IPs a VM might have assigned on its various vmNics or within its multiple IP configurations. You also see only one private IP for each VM, but not all of them if the machine happens to have more. And the major problem is that the “Virtual machines” report can’t be downloaded – at least as of Sep 2020.

You might think of using the “All resources” blade, which has the option of exporting the results as CSV, after filtering for “virtual machine” and “virtual machine(classic)” types, but once you try to edit the columns, you’ll notice that there aren’t as many as in the “Virtual machines” blade, particularly there’s nothing about IPs that can be selected.

Powershell can be used to retrieve both ARM and ASM VMs as well. One of the problems is that the cmdlets acting on one type of VMs will not work on the other, and as such separate Powershell modules exist that contain them: Azure for ASM and Az (along with the soon-to-be-discontinued AzureRM) for ARM.

Azure CLI is another way to get to Azure VMs. Again, separate versions need to be used, depending on whether ARM or ASM VMs are targeted.
The problem with both the Powershell and the Azure CLI approach is that one can only collect information about a set of VMs only after switching to a specific Azure subscription, which burns quite a lot of time. We’ll explore both these legacy options in the non-ARG Powershell and non-ARG Azure CLI sections later.
Azure Resource Manager can be used as well, but it has its own limitations – which will be discussed in the next section – that doesn’t make it the best approach.
What we’ll be using, and discussing at length in this article, is Azure Resource Graph (ARG). Its major advantage, speed, is what will get us to our goal – of listing all Azure VMs with their full list of private and public IPs – in a matter of seconds. Bonus points, ARG also has Powershell and Azure CLI support.

Azure Resource Graph (ARG)

What is ARG? According to Microsoft’s documentation, ARG “is a service in Azure that is designed to extend Azure Resource Management by providing efficient and performant resource exploration with the ability to query at scale across a given set of subscriptions[…]”. Doesn’t sound bad, but the important question is: why use ARG?

The answer is included in the link above, and consists of a few points. 1. Because it has its own database, that aggregates data from the various providers. So unlike with Azure Resource Management, we won’t have to query different providers individually to get data about VMs and their network configuration. Another important aspect is that 2. ARG works across subscriptions.

To see these 2 limitations in action, take a look at the API call to retrieve resources in ARM here and at the API call for retrieving the network interfaces here. Notice that each call needs a specific subscription. Unlike ARM, ARG allows using complex filter and join operations based on different columns whose data comes from different providers, all across multiple subscriptions. ARG also takes care of its own DB, by relying on updates coming from ARM every time a resource’s config changes, and also by doing full “crawls”, in case one of these updates get missed. The net effect is that our final query will be fast, and it will benefit from up-to-date information.

But how sure can we be that ARG is any good in terms of performance? It must be, as ARG is the one used for the Azure portal’s search feature, as stated here.

Querying ARG can be done using your favorite REST client (eg Insomnia), directly from Microsoft’s documentation portal here or better still, Azure Resource Graph Explorer (ARGE) can be used. The latter’s advantage is that you get a query editor, Azure subscription filter, table schema and other useful features. The direct link for ARGE is here.

As we’re looking for a way to eventually display all VMs with specific details, let’s start small. Here’s a basic query ran against a test subscription with only one VM:

Let’s look next at the language used to write the ARG queries.

Kusto

The SQL-like language used within the Azure Resource Graph Explorer is called Kusto, with a capital K. We’re not going to delve into the details, but instead just focus on the concepts we’ll need for our goal. And our goal is to come up with a Kusto query that retrieves each VM’s name, its list of all private IPs, and its list of all public IPs.

But what’s a Kusto query, to begin with? According to Microsoft’s documentation, it “is a read-only request to process data and return results“. The same link goes on to say that from a hierarchical perspective there are 3 building blocks: databases, tables, and columns.

Our final query will be composed of a single tabular expression statement, a fancy term meaning a sequence of operations, such as reading from data sources, applying filters and projections, and rendering instructions, all linked together by the pipe (|) symbol.

To keep things consistent, a few naming conventions are in order:

• The columns can also be referred to as attributes, as specified here
• The properties column is usually a property bag. As per this each property bag contains zero or more slots. A slot is a mapping between a string value and a dynamic type value. A dynamic type value in turn is one of the following 4:
  • null
  • a primitive scalar data type value (such as string, int, etc)
  • an array of dynamic values (with zero or more values)
  • a property bag
• Arrays can also be defined, and are easily spotted by the use of [ ] in the query’s result within ARG. The same link above describes an example, which makes it clear that an array can contain different types of elements

From the above, it follows that a property bag can contain other property bags within, and so on, as described in this section.

Going back to the initial sample in figure 1, let’s look at that in more detail:

We can identify the entities based on what we discussed earlier:

• The table used in this query is “Resources”, indicated with green
• The columns that fit on the screen under the “Details” pane, belonging to the query’s single result are circled in red
• Of these columns, some of their types are primitive scalar data types, holding just one “piece” of information. This is the case for location, resourceGroup and subscriptionId. In contrast, properties is a property bag, which in turn contains multiple slots, of which visible in the screenshot are provisioningState, hardwareProfile and networkProfile. Of these 3 slots, the first contains a scalar data type (string) while the other 2 are also property bags
• Within the networkProfile slot – itself a property bag – the networkInterfaces slot’s value is an array. The tell-tale sign of the array – the square bracket – is highlighted in orange

How can one go about finding out the columns’ types? In ARGE, on the left side, the tables and their columns are shown:

Note in the previous picture something that doesn’t refer to an actual element: an “`indexer`” entry signals that the property above is an array (eg networkInterfaces).

As of now – Sep 2020 – Microsoft Support confirmed that the “common” columns, such as name, resource group, etc aren’t shown, but user voice here can be used to request it.

2 very important points to remember:

• Not all Kusto’s language features and functions are supported by Azure Resource Graph, as Microsoft states explicitly here. In fact, Microsoft states explicitly that the ARG’s own query language is not exactly Kusto, but based on it (here).
• Pretty much everything in Kusto is case-sensitive. This includes operators, functions and column names. Take care when writing the queries, otherwise you’ll end up with an error (the happy case) or unexplained blank values or side effects (eg if you misspell the name of a slot in a property bag).

Don’t worry if this theoretical part doesn’t make a lot of sense right now, because things will become clearer in one of the next sections, where we’ll be building our query from scratch, and see the outcome at each step.

Scope

There have been 2 models so far under which IaaS VMs could be deployed in Azure: ARM (Azure Resource Manager) and ASM (Azure Service Manager). The differences are expanded upon very nicely here. From the standpoint of what we’re trying to achieve, the 3 big differences between the models – which are in the table at the end of the linked article – are the following:

• A virtual network (VNet) is required in ARM for a VM to be hooked to. In ASM this is optional
• A network interface is an independent resource, with its own lifecycle within the ARM model. In ASM, “Primary and Secondary Network Interface and its properties were defined as network configuration of a Virtual machine“
• Public IP addresses are independent resources from the VMs under the ARM model. In ASM, they can be associated directly with the VM

Machines under the old ASM model can’t be created anymore, unless you’ve been using VMs through this model in Feb 2020, as per https://docs.microsoft.com/en-us/azure/virtual-machines/classic-vm-deprecation#how-does-this-affect-me.

Luckily, ARG can be used to query VMs provisioned using both models. This is convenient, as we’re after extracting both the modern, ARM-based VMs, as well as the ASM ones, known as “classic” VMs, in this article.

Coming back to the Kusto query language, we won’t concern ourselves with any database, as ARG uses an implicit one. As for the tables, we’ll be using a single one, called “Resources”, which contains all the data we’re interested in, for both the ARM and ASM models.

ARM (Azure Resource Manager) Model

The very first thing we’re going to look at is a generic model for how an ARM VM connects to the network infrastructure in Azure. There are 2 main things we’re interested in: the fact that a VM can have multiple vmNics, which can be connected to different subnets, and that each vmNic can have multiple IP Configurations, each with a private IP and optionally a public one. The private and public IPs can be either dynamic or static.

In this section, we’ll construct the final Kusto query bit by bit. We’ll start with a very simple VM, and keep adding network elements to it until it’s representative for a VM with an advanced network config, as the picture above showed. In parallel, we’ll develop the query incrementally. Once the query will work for this VM, we’ll be able to extrapolate it to all VMs.
Let’s start working towards our final query by creating a VM (name: “JustOneTestVM”) that has a very simple configuration: just one vmNic (name: “justonetestvm915”) connected to a virtual network’s (name: “JustOneVnet”) subnet (name= “JustOneSubnet”). This single vmNic has just one IP Configuration, consisting of a private IP and a public IP. Both IPs are dynamic.
We’ll run the Kusto query below, which simply filters for virtual machines whose names match the one we’re after. The =~ is simply the case-insensitive equality operator.

Resources
| where type =~ 'microsoft.compute/virtualmachines' and name =~ 'JustOneTestVM'

Listing 1

Notice below that in the details of the only result returned – corresponding to our VM – there’s only the id of the vmNic. There’s no IP – whether private or public – that can be found in any of the result’s columns, and that includes properties as well. As we’ve seen previously, the networkInterfaces slot is actually an array, which in our case contains a single entry, corresponding to the only vmNic.

But we need to get to the IPs, so let’s focus our query towards the network interface itself, by running the following Kusto query:

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'

Listing 2

The result of this query does contain the private IP explicitly. However, the public IP is only referenced by its id, as seen below, which makes sense if you think about it, as the public IP is a separate resource in the ARM model, just as the network interface resource is separate from the VM itself.

But we want the IPs shown in the result set itself, so let’s extract that information, using the following query. We’re simply indexing in the one and only vmNic IP configuration, then get to the right slot that contains the info we’re after. project simply returns only the columns we specify.

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | project privateIp = properties.ipConfigurations[0].properties.privateIPAddress,
        publicIpId = properties.ipConfigurations[0].properties.publicIPAddress.id

Listing 3

And here’s the result, as expected:

Yet we want our final query to be able to handle multiple IP configurations, not just one, as this feature was introduced back in 2017. Let’s modify our VM so that it has 2 IP configurations. We’ll only add a private IP, and skip associating a public IP:

So at this stage running the query in listing 1 will result in the properties.ipConfigurations array containing not one, but two elements. Each element will consist of a properties slot (not to be confused with the ipConfigurations‘s parent properties one) that in turn will contain the private IP for the respective IP configuration and optionally the public IP (if one is associated). Here’s just the top properties slot, as it’s returned by ARGE:

What we’d like next is to extract just the private IPs and the public ones. Although it may not feel like the step in the right direction, we’re going to split the 2 elements of the array, so that they’re placed on separate rows. So instead of just one row as the result of the query, we’ll have 2.

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | mv-expand ipconfig=properties.ipConfigurations

Listing 4

Note below the 2 output rows in the lower left. The columns and their values are identical for the 2 rows except for one extra column that was added, called ipconfig. On each row, subsequent elements of the properties.ipConfigurations array are “extracted” one by one. The “Details” pane in the picture shows the first element of the array, as extracted on the first row.

As we don’t need most of the columns, let’s just keep the IPs we’re interested in, along with the vmNic id. We’ll use project again to specify the columns we want to keep, and the query becomes:

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | mv-expand ipconfig=properties.ipConfigurations
    | project nicId = id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = ipconfig.properties.publicIPAddress.id

Listing 5

With the result below:

Notice one of the public IPs is missing, which is because we didn’t associate a public IP for the 2nd IP configuration when we added it.

Let’s do something about the public IPs, so the real addresses are shown, instead of just the id. We’ll start a separate query that simply lists all the public IP resources in my test subscription:

Resources
    | where type =~ 'microsoft.network/publicipaddresses'

Listing 6

The result is:

Looking at the details, we can see the public IP assigned (note that you might now see the IP right away due to delays):

The first entry belongs to a domain controller VM I’m using for a different purpose, while the second one corresponds to the public IP in the first IP configuration for our test VM’s only vmNic. There’s nothing to expand here as we’ve done previously, as each entry corresponds to a single public IP.

As we won’t care about most of the columns, let’s just keep the public IP id and address using the query below:

Resources
    | where type =~ 'microsoft.network/publicipaddresses'
    | project publicIpId = id, publicIp = properties.ipAddress

Listing 7

The result is below. The first entry is missing an actual IP address as the domain controller it belongs to is stopped and deallocated.

Coming back to the output in figure 10, let’s replace the ids for the public IPs with the real addresses. In essence, we’re looking to join the tables seen in figure 10 and figure 13. And that we can achieve using the join Kusto operator (described here) against the queries seen in Listing 5 and 7. The query we’ll attempt to run is below:

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | mv-expand ipconfig=properties.ipConfigurations
    | project nicId = id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = ipconfig.properties.publicIPAddress.id
    | join (Resources
        | where type =~ 'microsoft.network/publicipaddresses'
        | project publicIpId = id, publicIp = properties.ipAddress
    ) on publicIpId

Listing 8

The output however indicates there’s an error:

Fixing this is straightforward, as the error message tells explicitly what to do*. We’ll apply tostring against the public IP ids extracted from the vmNics objects:

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | mv-expand ipconfig=properties.ipConfigurations
    | project nicId = id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
    | join (Resources
        | where type =~ 'microsoft.network/publicipaddresses'
        | project publicIpId = id, publicIp = properties.ipAddress
    ) on publicIpId

Listing 9

Let’s think for a moment what the output should be, before seeing the actual results. What we’d hope to get is the table in figure 10, with the same 2 rows corresponding to the 2 IP configurations defined on that vmNic, but with one single change – have the real public IP address showing instead of the cryptic id. Let’s cross-check our expectations with the actual result:

We do get the public IP address resolved on the same row where initially we only got its id, but there are 2 issues: first, the id is still there but appears in 2 columns, and second, the 2nd row belonging to the vmNic’s 2nd IP configuration is now gone. What went wrong?

To understand, we need to take a closer look at the join operator and how it works. In the documentation there are a couple of key things worth knowing:

• The table on the left of the join is called the “outer” table, while the one on the right of the join is called the “inner” table. This convention will be useful in the context of the join flavor
• There are quite a few join flavors*, which represent the mechanism used by the join operator to output the resulting table

It turns out that if no join flavor is specified – and for our last query, this is just the case – Kusto will assume that we want a innerunique type of join. As per the documentation, this means that “Only one row from the left side is matched for each value of the on key. The output contains a row for each match of this row with rows from the right“. In our case, this simply means “take the unique values for publicIpId from the result in figure 10 (the left table) and match them to the values in the `publicIpId column in figure 13 (the right table). For every such match, output a row in the resulting table that consists of all the columns in the first table plus all the columns in the second one”.

How many such matches do we have? Well, there’s the public IP id of our test VM that corresponds to the private IP 10.0.1.4 which also shows up in table 13, next to the 104.40.204.240. And that’s it. The empty public IP id showing on the 2nd row in figure 10 can’t be matched to any id in figure 13, as there’s no empty string showing as id in this latter figure, so the join operator leaves it out altogether.

As for the id columns, and why we get to see 2 of them: the join operator will merge the rows of the 2 tables according to the specified join flavor, as discussed above. Since each of the 2 tables contains a column called publicIpId, Kusto has to somehow put both of them in the result table, so it resorts to renaming one of them to a different value, hence appending a ‘1’.

Coming back to the result we actually wanted, we don’t want only the rows whose public IP id in the left table matches one in the right table, instead, we want all the rows in the left table to be kept, and only add the rows in the right table when the ids for the public IPs match. Which describes quite well that the leftouter join flavor does. Let’s test with the modified query as follows:

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | mv-expand ipconfig=properties.ipConfigurations
    | project nicId = id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
    | join kind=leftouter (Resources
        | where type =~ 'microsoft.network/publicipaddresses'
        | project publicIpId = id, publicIp = properties.ipAddress
    ) on publicIpId

Listing 10

The result below, looking just as we expected:

We can easily remove the duplicated id columns, by using project-away as in the following query:

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | mv-expand ipconfig=properties.ipConfigurations
    | project nicId = id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
    | join kind=leftouter (Resources
        | where type =~ 'microsoft.network/publicipaddresses'
        | project publicIpId = id, publicIp = properties.ipAddress
    ) on publicIpId
    | project-away publicIpId, publicIpId1

Listing 11

The result without the redundant public IP ids:

At this point, we’d just want to “squash” the 2 rows, so that the vmNic id – the same for the 2 rows – is kept only once, and the 2 private IPs (10.0.1.4 and 10.0.1.5) will be turned to a single array containing both values, while for the single public IP (104.40.204.240) this should be kept as-is. And it turns out it’s quite simple to aggregate the data in this way, by using Kusto’s summarize operator together with the make_list() function. We’ll add one more row to our query, so it becomes:

Resources
    | where type =~ 'microsoft.network/networkinterfaces' and name =~ 'justonetestvm915'
    | mv-expand ipconfig=properties.ipConfigurations
    | project nicId = id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
    | join kind=leftouter (Resources
        | where type =~ 'microsoft.network/publicipaddresses'
        | project publicIpId = id, publicIp = properties.ipAddress
    ) on publicIpId
    | project-away publicIpId, publicIpId1
    | summarize privateIps = make_list(privateIp), publicIps = make_list(publicIp) by nicId

Listing 12

And the expected result below:

This is what we were after however let’s not forget that we’ve been working against a VM’s single vmNic all along. We need the final query to support multiple vmNics, so let’s go ahead and add a second one to our test VM. We’re going to have to stop the VM to do that, so the public IP currently assigned will most likely change after the VM is powered back on, as we’re not going to reserve it. The final state of the VM, with a second vmNic having a single IP configuration that has a private IP (10.0.2.4) and an associated public one:

This new vmNic (name= “justonetestvm916”) is connected to the same virtual network as the first vmNic (name: “JustOneVnet”) but to a different subnet within it (name= “JustAnotherSubnet”). Note that a vmNic cannot be connected to a different virtual network (VNet) than any vmNic that’s already connected to that VM, as per the note here. This single vmNic has just one IP Configuration, consisting of a private IP and a public IP. Both IPs are dynamic.
In the last query seen in listing 12, we’ll remove the filtering for the name of the first vmNic and the aggregation line, to get to the following query:

Resources
    | where type =~ 'microsoft.network/networkinterfaces'
    | mv-expand ipconfig=properties.ipConfigurations
    | project nicId = id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
    | join kind=leftouter (Resources
        | where type =~ 'microsoft.network/publicipaddresses'
        | project publicIpId = id, publicIp = properties.ipAddress
    ) on publicIpId
    | project-away publicIpId, publicIpId1

Listing 13

And the result, showing all the defined vmNics in the test Azure subscription used:

There’s no point in aggregating all the data now, as all we have are rows for every single IP configuration belonging to all the vmNics in turn. What we actually want is to aggregate all the IPs per each VM. And to get there we simply need to find another column – other than the vmNics’ id – to “link” our data, as follows: we know that each VM has an id (one is partially visible in figure 1), and we’d just need something to link all the vmNics to their parent VM (as a vmNic can only be hooked to a single VM). Luckily a vmNic has just one such attribute, as seen below:

Let’s remove the nicId column from the query in listing 13, and add the parent VM id instead:

Resources
    | where type =~ 'microsoft.network/networkinterfaces'
    | mv-expand ipconfig=properties.ipConfigurations
    | project vmId = properties.virtualMachine.id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
    | join kind=leftouter (Resources
        | where type =~ 'microsoft.network/publicipaddresses'
        | project publicIpId = id, publicIp = properties.ipAddress
    ) on publicIpId
    | project-away publicIpId, publicIpId1

Listing 14

And the result, showing an entry for each IP configuration and its vmNic’s parent VM id:

Let’s also extract a list of VMs, but keep only the VM id and the name of the VM, using this query:

Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project vmId = id, vmName = name

Listing 15

The result of the query, showing the 2 VMs currently present in the subscription, the second being the one we’ve been building at in this section:

At this point we can do the same thing we did when we resolved the public IP ids: we have 2 tables – the one in figure 21 and figure 22 – that contain a common column representing the VMs’ id. We’d simply have to join them to get to our goal. We’ll use the VM table (figure 22) as the left (outer) table, and the vmNic table (figure 21) as the right (inner) table. We know the rows for the left table are unique – as we don’t expect for a VM id to show up twice. For the right table, we do expect for at least some of the VM ids to show up twice, corresponding to VMs that have multiple IP configurations or multiple vmNics; we’d also expect to have cases where the some of the vmNics’ parent VM id is null. Why the latter, taking into account that – according to the ARM model – there cannot be a VM that doesn’t have at least one vmNic connected? Because a VM with multiple vmNics can have some of them disconnected, and once this happens, those vmNics can be left “orphaned”, with no parent VM id stamped (the value is null).

Therefore from the 3 join flavor that ARG supports, innerunique is not required as the VMs in the left table are already unique, leftouter is not suitable as we don’t expect to find VMs on the left table that don’t show up in the right table (there can’t be a vmNic that has a parent VM id not known in the full table of VMs, as the latter must contain all possible VMs that exist). Hence the inner kind will be the one we’ll use, and in the final result we’ll get a number of rows equal to that of the right table (we know the left table contains unique entries, so all combinations that join creates will essentially result in the right table that has the corresponding VM row “appended”). This leads us to the query below:

Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project vmId = id, vmName = name
    | join (Resources
        | where type =~ 'microsoft.network/networkinterfaces'
        | mv-expand ipconfig=properties.ipConfigurations
        | project vmId = properties.virtualMachine.id, privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
        | join kind=leftouter (Resources
            | where type =~ 'microsoft.network/publicipaddresses'
            | project publicIpId = id, publicIp = properties.ipAddress
        ) on publicIpId
        | project-away publicIpId, publicIpId1
        ) on vmId

Listing 16

f you remember our very first join, we’ve run into an error the first time we tried it. The same will occur for this query as well, if you try to run it as-is. The problem is the same one seen back in figure 14, and has to do with the fact that the the vmId column has the type dynamic, which join doesn’t support. The fix is the same, just use the tostring() function to convert it to a string primitive type. Since both the vmId columns are constructed – both in the left and right table – both expressions need to be converted, as so:

Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project vmId = tostring(id), vmName = name
    | join (Resources
        | where type =~ 'microsoft.network/networkinterfaces'
        | mv-expand ipconfig=properties.ipConfigurations
        | project vmId = tostring(properties.virtualMachine.id), privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
        | join kind=leftouter (Resources
            | where type =~ 'microsoft.network/publicipaddresses'
            | project publicIpId = id, publicIp = properties.ipAddress
        ) on publicIpId
        | project-away publicIpId, publicIpId1
        ) on vmId

Listing 17

Yet if you run this, there’s something really wrong about it – the rows for the IP configurations of our test VM are nowhere to be seen. All we get is a single row, belonging to the only IP configuration that the VM which already existed before we started has:

If you look closely at figures 21 and 22, you’ll notice something interesting – the resource group name in the VM’s id is in uppercase in the VM table (figure 22) while in the vmNic table all 3 rows corresponding to our test VM have the resource group in a different capitalization (figure 21). The net result is that the values are seen as completely different by the join operator since it acts in a case-sensitive way, and no rows are matched, which yields the result above.

How to fix this problem? We’ll just apply the tolower() function to both vmId columns, which will make the join key consistent between the 2 tables:

Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project vmId = tolower(tostring(id)), vmName = name
    | join (Resources
        | where type =~ 'microsoft.network/networkinterfaces'
        | mv-expand ipconfig=properties.ipConfigurations
        | project vmId = tolower(tostring(properties.virtualMachine.id)), privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
        | join kind=leftouter (Resources
            | where type =~ 'microsoft.network/publicipaddresses'
            | project publicIpId = id, publicIp = properties.ipAddress
        ) on publicIpId
        | project-away publicIpId, publicIpId1
        ) on vmId

Listing 18

The result is now as expected:

The only thing left to do is to aggregate the IPs, similar to how it was initially done, using the summarize operator and the make_list function we’ve introduced back in listing 12. The line will be placed in the exact same place, the only difference is that now we’ll aggregate by the vmId:

Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project vmId = tolower(tostring(id)), vmName = name
    | join (Resources
        | where type =~ 'microsoft.network/networkinterfaces'
        | mv-expand ipconfig=properties.ipConfigurations
        | project vmId = tolower(tostring(properties.virtualMachine.id)), privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
        | join kind=leftouter (Resources
            | where type =~ 'microsoft.network/publicipaddresses'
            | project publicIpId = id, publicIp = properties.ipAddress
        ) on publicIpId
        | project-away publicIpId, publicIpId1
        | summarize privateIps = make_list(privateIp), publicIps = make_list(publicIp) by vmId
        ) on vmId

Listing 19

Now we can safely get rid of the doubled vmId1 column, which now has no purpose anymore. We’ll keep the vmId as a tie-breaker when 2 or more VMs have the same name across subscriptions, and we’ll also sort by the VM name, with the final query becoming:

Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project vmId = tolower(tostring(id)), vmName = name
    | join (Resources
        | where type =~ 'microsoft.network/networkinterfaces'
        | mv-expand ipconfig=properties.ipConfigurations
        | project vmId = tolower(tostring(properties.virtualMachine.id)), privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
        | join kind=leftouter (Resources
            | where type =~ 'microsoft.network/publicipaddresses'
            | project publicIpId = id, publicIp = properties.ipAddress
        ) on publicIpId
        | project-away publicIpId, publicIpId1
        | summarize privateIps = make_list(privateIp), publicIps = make_list(publicIp) by vmId
    ) on vmId
    | project-away vmId1
    | sort by vmName asc

Listing 20

And the result below:

As we’ll see later, when going over pagination, sorting the result set has important implications, aside the “cosmetical” alphabetical order by VM name.

ASM (Azure Service Manager) Model

We’re not going to go over the ASM model in detail, as things are very well explained here. What we do want to know is the differences at the networking layer between the 2 models, in order to build the ASM ARG query appropriately.

Let’s take a look at the details of one such VM:

The first thing that you can notice is that the IPs are within a property bag called instanceView. Although the documentation around the notion of “instance view” is rather scarce, funny enough we can get some info from the Powershell cmdlet used in the ARM model, as Get-AzVM‘s description here currently states that “The model view is the user specified properties of the virtual machine. The instance view is the instance level status of the virtual machine“. So getting the actually assigned values for the various parameters (such as IP addresses) should come from the instance view.

Let’s look at the private IP addresses, and understand whether a classic VM can have multiple ones, as was the case with ARM, or not. properties‘ instanceView property bag contains a slot called privateIpAddress, whose value is a string, not an array. Also the documentation here states that “Multiple IP addresses cannot be assigned to resources created through the classic deployment model“. The guide for classic VMs here also doesn’t show a way to create additional IP addresses, be it private or public.

So we can only have a single private IP address for the classic VMs. Let’s move on to the public IPs.

The public IPs, as defined in properties‘ instanceView property bag, is an array (note the information is enclosed within []). So we know that there can be multiple public IPs per one classic VM.

For the query itself:

• We need a join because doing mv-expand on a VM row that doesn’t have even a single public IP will return 0 rows in the result set. Keeping those VMs that don’t have a single public IP assigned is thus possible by doing a leftouter type of join
• The tables against which the join is performed have the same source, therefore the same id can be used, with no tolower() needed. Contrast this to the ARM query, where the network interfaces themselves, including the VM id specified against them, were pulled from a different set of objects

We’ll keep the VMs’ id, to be able to differentiate between identically named VMs across different subscriptions, and also sort the result set. The final ASM query thus becomes:

Resources
    | where type =~ 'microsoft.classiccompute/virtualmachines'
    | project id, name, privateIp = properties.instanceView.privateIpAddress
    | join kind=leftouter (Resources
        | where type =~ 'microsoft.classiccompute/virtualmachines'
        | mv-expand publicIp=properties.instanceView.publicIpAddresses
        | summarize publicIps = make_list(publicIp) by id
    ) on id
    | project-away id1
    | sort by name asc

Listing 21

If you run the query, you might see some of your classic VMs returned with multiple public IPs reported, despite their status being “Stopped (deallocated)“. You could rightly wonder how this is so, and particularly how can multiple public IPs be assigned to the same VM, particularly since a single private IP is allowed. The answer here sheds light on both questions, as follows:

• A classic VM can have both a Cloud Service Public IP and an Instance Level Public IP. A VM showing with 2 public IP addresses most likely has one of them belonging to a Cloud Service that includes it
• A Cloud Service Public IP is reserved for the duration of the VM’s lifetime, as explained here. This would probably explain why even though the VM is “Stopped (deallocated)” it can still hang on to that IP

Wrappers (Powershell)

With both the ARM and ASM ARG queries ready, let’s see what we can use aside ARGE to interact with them programmatically. Azure CLI and Powershell can be used to run and obtain the result sets for ARG queries. Both have a brief intro here. .NET/C# access is possible as well, but we’ll leave that for a future post, as the current one has grown to a considerable size as it is.

Of the 3 methods above, we’ll only look thoroughly at how to use Powershell to interact with ARG. Using Azure CLI to query ARG will be touched upon at the end of this article, but only briefly.

In order to use Powershell to run our ARG queries, we’ll need the Search-AzGraph cmdlet, which resides in the Az.ResourceGraph module. Make sure you have this one installed (as of Sep 2020, this is not present by default in Cloud Shell, and needs to be installed; the current version is 0.7.7). Then you need to connect to your tenant, using Connect-AzAccount (if you’re using Cloud Shell this step is done automatically for you). At this point, we can run the Search-AzGraph -Query <ARG_query>, and get all the rows back as objects, which can then be indexed into and manipulated as usual.

If no -Subscription value is specified, then Search-AzGraph will perform the query against the whole tenant, across subscriptions, which is what we’re after actually*.

The concern is what happens when our queries return a significant number of results, as in a big number of VMs in the result set. In this context, Search-AzGraph doesn’t handle pagination itself transparently, but offers parameters to implement it easily ourselves.

Our pagination code will simply run the same exact Kusto query in a loop, and use a “rolling” window against the same result set. This window will be obtained by using the Search-AzGraph‘s -First and -Skip parameters. The -Skip will tell where the result window starts from, and the -First parameter will tell how many rows will be retrieved from that starting point. Using the numeric example here, the “rolling” window starts at index 3000 and spans for 1000 rows.

To get the best speed, we’ll use the maximum page size currently available, which is 5000 entries*.

Our code will consist of a loop that makes sure that the “rolling” window is moved across the whole result set. You’ll notice the Search-AzGraph shows twice in the code below, and that is because it doesn’t support 0 as the value for -Skip (if you attempt it, you get “The 0 argument is less than the minimum allowed range of 1“), so the very first batch of results needs to be treated on a separate if branch. When the number of results is no longer equal to the page size, it means our “rolling” window is right above the last set of entries (or is “looking” at a completely null set, if the very last row fitted neatly into the previous filled page).

We’ll run the pagination code twice – first for the ARG query handling ARM VMs, and second for the ARG query handling the ASM ones. The output is then written to disk as CSV files whose filenames are timestamped. We’ll use separate CSV files to keep the ARM VMs separate from the ASM (classic) ones.

Here’s our loop below, which adds each subsequent Search-AzGraph output to an array that will eventually contain the final result set.

$fullResultSet = @()
    $pageSize = 5000
    $pagesProcessedSoFar = 0
    do {
        $results = @()
        if($pagesProcessedSoFar -eq 0) {
            $results = Search-AzGraph -Query $ARG_query -First $pageSize
        }
        else {
            $results = Search-AzGraph -Query $ARG_query -First $pageSize -Skip ($pagesProcessedSoFar * $pageSize)
        }
        $pagesProcessedSoFar++
        Write-Host "Processed $pagesProcessedSoFar pages so far. A number of $(($results | Measure-Object).count) results returned in the last page"
        $fullResultSet += $results
    } while(($results | Measure-Object).count -eq $pageSize)

Listing 22

3 very important issues need to be kept in mind, and we’ll discuss each next.

First, the ARG queries need to be sorted, otherwise the paging mechanism will not work. If no sorting is performed, the outcome will be that the results might be wrong, and in certain cases the loop will never end*.

Secondly, a page size of 5000 is not possible for our queries in their current state (listing 20 for ARM and listing 21 for ASM). The maximum number of rows obtained per query – if you attempt to use Search-AzGraph against a large enough VM inventory – will be 1000. Although I don’t have a firm answer right now I’m assuming it’s because neither of the original id columns are kept, particularly given the last important note here. We do have the vmId column, but ARG doesn’t consider the result set as including a primary key, so it downgrades to 1000 of maximum results returned, instead of the 5000*. If however we keep the id of the VM (make the 3rd line of either ARM/ASM query to project the id as the first field), then ARG will honor a -First value between 1000 and 5000, and return an equally sized result set. One quirk to be aware of is that aside from the id (recognized as the primary key by ARG), Search-AzGraph includes a column in the result set, called ResourceId, which contains the same values as the id itself (if you run the query in ARGE you’ll notice that this isn’t the case, and this column doesn’t show up). The ResourceId always gets included if the primary key (the id) is also present, regardless of how many rows are asked for via -First (it can even be 1 and the column is there). In the final Powershell code we’ll eliminate this column from the output.

As such, let’s rewrite the ARM ARG query so that it’s large-page-friendly, by including the “default” id column for the VMs. We’ll get rid of the vmId one we’ve used when building the query, since it’s no longer required.

Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project id, vmId = tolower(tostring(id)), vmName = name
    | join (Resources
        | where type =~ 'microsoft.network/networkinterfaces'
        | mv-expand ipconfig=properties.ipConfigurations
        | project vmId = tolower(tostring(properties.virtualMachine.id)), privateIp = ipconfig.properties.privateIPAddress, publicIpId = tostring(ipconfig.properties.publicIPAddress.id)
        | join kind=leftouter (Resources
            | where type =~ 'microsoft.network/publicipaddresses'
            | project publicIpId = id, publicIp = properties.ipAddress
        ) on publicIpId
        | project-away publicIpId, publicIpId1
        | summarize privateIps = make_list(privateIp), publicIps = make_list(publicIp) by vmId
    ) on vmId
    | project-away vmId, vmId1
    | sort by vmName asc

Listing 23

Here’s the output in ARGE, and notice the original id field that’s now included:

Thirdly, looking at the Powershell object returned by Search-AzGraph will not show anything for the arrays containing the IPs. For example, for a VM with 3 private IPs, the only thing shown is a cryptic {, , } instead of the array containing those 3 IPs. Even more, trying to display the array won’t return anything:

Why this is so is explained here. In short, ToString() needs to be called. As we’re doing Export-Csv at the end of our code, this will actually result in the string for the array to be written, simply because under the hood Export-Csv calls ToString(). The downside is that the file is written to using the JSON format, which looks a bit cumbersome when opened in Excel:

The quick fix is to parse the private and public IP arrays and convert them, as such:

$results | Select-Object id, vmName, @{label="privateIps";expression={$_.privateIps.ToString() | ConvertFrom-Json}}, @{label="publicIps";expression={$_.publicIps.ToString() | ConvertFrom-Json}}

Listing 24

And this is how the output now looks in Powershell:

The final Powershell code further into the article takes into account all the issues.

One last thing: in theory, it’s possible – although unlikely – to have a “tear” in the results. Consider if one or multiple VMs get deleted when the set of queries is running, in the middle of pagination. Similarly, it’s theoretically possible to have doubled results, eg if a VM gets created inside a page “bin” that’s past that which the current query feeds.

REST Clients

A REST client can be used against Azure Resource Graph. Simply query this endpoint https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2019-04-01, and submit a “Bearer” token obtained using the Powershell lines here, as follows:

$azContext = Get-AzContext
$azProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile
$profileClient = New-Object -TypeName Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient -ArgumentList ($azProfile)
$token = $profileClient.AcquireAccessToken($azContext.Subscription.TenantId)
$token.AccessToken

Listing 25

Copy the access token (don’t worry that it’s multiline) and paste it in your REST client’s authentication tab. Here’s how this looks like for Insomnia:

Next, provide the payload as described here and use the Kusto query in listing 23. The square brackets around the “subscriptions” attribute indicate that an array can be supplied, and as such, multiple subscriptions can be targeted by the query; simply separate the quoted Azure subscriptions’ ids by commas.

Here’s the payload and the response, when querying against my test subscription:

Note that the tokens obtained via Cloud Shell, as described previously, are valid for 1h, and are valid with 5 minutes ahead of the issuance time, and up until exactly 1h after they’re issued; this can be easily seen with https://jwt.io (hover over the numbers representing Linux timestamps, and it’ll be converted to human-readable format).

If you want to get inspiration about the headers and payload itself, use Search-AzGraph with your desired ARG query and provide the -Debug switch parameter. You’ll get to see the request and the reply’s respective header and payload. Here’s the partial output when supplying the ARM query in listing 23:

Throttling

4 attributes appear to control how many requests can be made. From my experiments (using both Search-AzGraph and Insomnia) I’ve consistently obtained the values below in the reply to the query seen in Listing 23 across some 4k VMs stored in 150+ Azure subscriptions. Since they’re obtained after one call, it’s safe to assume that 15 is the number of requests that can be made in 5 seconds by default, which this article confirms. As it can be seen, I’ve barely made a dent in my quota, although the workload wasn’t negligible at all.

• x-ms-ratelimit-remaining-tenant-reads: 11995
• x-ms-ratelimit-remaining-tenant-resource-requests: 14
• x-ms-user-quota-remaining: 14
• x-ms-user-quota-resets-after: 00:00:05

You can actually see these headers back in picture 34.

As described in the Azure throttling docs here, Microsoft can be contacted to increase that limit for a specific tenant.

A Word About Delays

One issue I’ve run into was the fact that getting the most recent IPs was inconsistent – sometimes I would change an IP (be it either private or public) against a VM and ARG would show the result immediately, other times it would take hours for the new IP to show in the result of the ARG query. Microsoft Support again provided the answer, which I paste here verbatim:

Resource updates in ARG depend on the Resource Provider mostly. In this case, as you have issues with IPs updating, that’s the Network resource provider that is actually not tracked by ARM directly. This means that right now the Network Resource provider sends notifications that resources were created in ARM. The timeframe for getting this notification can be anywhere from 10 seconds up to 30 hours unfortunately. We are aware of this issue and it should be solved starting October, lowering this timeframe to less than 1 minute.

The thing is that ARG depends on the various providers to get their data. This Microsoft article explains further: “When an Azure resource is updated, Resource Graph is notified by Resource Manager of the change. Resource Graph then updates its database. Resource Graph also does a regular full scan. This scan ensures that Resource Graph data is current if there are missed notifications or when a resource is updated outside of Resource Manager.“

More Than 1,000 Azure Subscriptions?

If you have more than 1,000 Azure subscriptions, there’s a problem, since an ARG query – sent via either Powershell or Azure CLI – will only run against 1,000 of them. This is described here, along with a very elegant solution, that’s grouping the Azure subscriptions into small enough batches so that the limitation is bypassed.

For our final Powershell code, this means we’re going to have an additional layer of pagination, at the level of subscription batches. We’ll end up not with just one loop, but with 2. The outer one will iterate through the subscription batches, while the inner one handles the pagination of Search-AzGraph‘s result set. The cmdlet will be scoped to the current subscription batch, using the -Subscription parameter, which takes as input an array. The array will contain the Azure subscription ids that happen to be inside the current subscription batch.

Each aggregated result from the inner loop that’s calling Search-AzGraph repeatedly gets added to the final result set, as the subscription batches are iterated through.

One small problem is that since the ARM/ASM ARG query runs against a specific subscription batch, the guarantee that the results are ordered is only per batch, as it’s the ARG query that’s doing the sorting within. The final “stitched” results most likely won’t be sorted overall, so we’ll have to handle that manually, by calling Sort-Object right before exporting the CSV files.

If you don’t have more than 1,000 subscriptions, you can gain a few seconds per runtime by removing this extra batching code from the final script. Useful if you’ll be automating and know that you’re under the limit.

Permissions

As per https://docs.microsoft.com/en-us/azure/governance/resource-graph/overview#permissions-in-azure-resource-graph: “To use Resource Graph, you must have appropriate rights in Role-based access control (RBAC) with at least read access to the resources you want to query. Without at least read permissions to the Azure object or object group, results won’t be returned.“

If you happen to be a global admin for your tenant, then you can grant yourself access to all subscriptions within via a simple setting. From the Azure Active Directory blade, toggle the option below to “Yes”:

Important: if the global administrator account doesn’t have access to at least one Azure subscription, nothing will be visible, despite the self-elevation. To fix this, grant yourself access (“Owner” permission will do) to at least one Azure subscription. Although not effective immediately, eventually all the subscriptions will become available.

As for the minimum permissions required, the “Reader” Azure RBAC role will do. Simply grant this either at the tenant root management group level to get rights against all subscriptions, or assign it to different management groups or subscriptions so ARG can operate only on those.

Get the List of All Azure VMs With All Their Private and Public IPs (Azure Resource Graph)

Before you begin, make sure the account you use to login to Azure has the required permissions, described above.

Two approaches are listed below, with both of them resulting in a set of 2 separate CSV files – one file for ARM VMs and another file for ASM VMs.

The first way, using Azure Resource Graph Explorer (ARGE), VMs containing multiple private or public IPs will have these IP addresses separated by a comma in the CSV output. The second way, using Powershell, will output any multiple IPs separated by a space.

In terms of runtime, running each query as part of option 1 should take seconds at most, ideally below 1s if you’re targeting only a few thousand VMs. For option 2, the time is slightly larger as the subscriptions must be enumerated to workaround a current ARG limitation, but still the time is around 10s for a few thousand VMs.

Option 1: Azure Resource Graph Explorer (ARGE)

1. Use this direct link to open the ARM query (listing 23) directly in Azure Resource Graph Explorer (ARGE)
2. Check that you have access to all the Azure subscriptions from the drop-down in the top right. There’s currently a bug in ARGE that requires you to repeatedly click the drop-down, and scroll through the list of subscriptions, before the full list of subscriptions that you have access to shows up
3. Run the query
4. Download the result as CSV
5. Use this direct link to open the ASM query (listing 21) directly in Azure Resource Graph Explorer (ARGE)
6. Run the query
7. Download the result as CSV
8. Remove the following 3 characters from both CSV files: [ ] “. Don’t worry about messing up any of the other columns, as the virtual machine’s name is not allowed to contain these 3 special characters (ARM table is here). The resource group’s name is also not allowed to contain those special characters (doc here), which translates to the VM’s id – itself made up of multiple particles including the resource group name and the VM name – to also not contain them.

Option 2: Powershell

1. Either start Azure Cloud Shell as described here or use a local Powershell console on your machine
2. Install the Az.ResourceGraph Powershell module if not already present as described here, by using Install-Module Az.ResourceGraph in an admin Powershell command (or directly in Cloud Shell, if you’re using that)
3. If you’re running from a local Powershell console, you need to connect to your tenant first using Connect-AzAccount (important: make sure that under the SubscriptionName column of this command’s output you see a name; if that’s blank, run Clear-AzContext followed by Connect-AzAccount, and make sure you get a subscription name this time). If you’re using Cloud Shell, you’re automatically authenticated to your tenant
4. Save the script below, then run it in the Powershell console. If using Cloud Shell, upload the script first, run it, then issue the ls command to obtain the name of the output CSV files, followed by downloading them (transfer to/from Cloud Shell is described here)

Alternative: Non-ARG Powershell Cmdlets

Without Azure Resource Graph (ARG), there’s the Get-AzVM cmdlet. Microsoft already provides some code to extract all the VM data – including their private and public IPs – per one subscription, here. The downside is that for VMs having more than 1 vmNic there will be multiple rows with the same VM name, which makes things less clear. Aside from this, the code has already been adapted by others to work against all subscriptions, by enclosing it in a loop, as seen here.

Exporting the data to a CSV file needs however to take into account VMs that might have multiple IP configurations per vmNic. When this is the case, simply piping the output to Export-Csv directly will result in a System.Object[] entry in the private IP address column. One way of solving this is to explicitly specify the property, which will result in a string containing all the IP addresses separated by the chosen separator, which by default is space.

The final Powershell code follows:

$report = @()
Get-AzSubscription | % {
Select-AzSubscription $_
    $vms = Get-AzVM
    $publicIps = Get-AzPublicIpAddress
    $nics = Get-AzNetworkInterface | ?{ $_.VirtualMachine -NE $null}
    foreach ($nic in $nics) {
        $info = "" | Select-Object VmName, ResourceGroupName, Region, VirturalNetwork, Subnet, PrivateIpAddress, OsType, PublicIPAddress, SubscriptionName
        $vm = $vms | ? -Property Id -eq $nic.VirtualMachine.id
        foreach($publicIp in $publicIps) {
            if($nic.IpConfigurations.id -eq $publicIp.ipconfiguration.Id) {
                $info.PublicIPAddress = $publicIp.ipaddress
            }
        }
        $info.OsType = $vm.StorageProfile.OsDisk.OsType
        $info.VMName = $vm.Name
        $info.ResourceGroupName = $vm.ResourceGroupName
        $info.Region = $vm.Location
        $info.VirturalNetwork = $nic.IpConfigurations.subnet.Id.Split("/")[-3]
        $info.Subnet = $nic.IpConfigurations.subnet.Id.Split("/")[-1]
        $info.PrivateIpAddress = $nic.IpConfigurations.PrivateIpAddress
        $info.SubscriptionName=$_.Name
        $report+=$info
    }
}
$report | Select-Object VmName, ResourceGroupName, Region, VirtualNetwork, Subnet,`
    @{label="PrivateIpAddress";expression={$_.PrivateIpAddress}}, OsType, PublicIPAddress,`
    SubscriptionName | Export-Csv -NoTypeInformation "VMs_$(Get-Date -Uformat "%Y%m%d-%H%M%S").csv"

Listing 27 – Retrieving all private and public IPs for all ARM VMs within an Azure tenant using non-ARG cmdlets

The output CSV file will contain multiple IP addresses separated by space, just as the ARG Powershell code we’ve seen before.

But there’s a problem, as Get-AzVM will only operate against machines deployed using the ARM model, as explicitly stated here: “However, the Resource Manager cmdlet Get-AzVM only returns virtual machines deployed through Resource Manager“. For the ASM, or Azure classic VMs, you’ll have to install the respective Powershell module, as described here, and use different code to get the list of classic VMs, based most likely on Select-AzureSubscription and Get-AzureVM.

As for the ARM code above, speed is not it’s main quality, as there’s no parallelism whatsoever (eg Powershell background jobs). Subscriptions are selected in turn, and VM data is obtained for each. Even more, if using Azure Cloud Shell, the session will timeout after 20 minutes by default. This means that the export will most likely never finish for a large VM inventory unless you’re interacting with the respective browser window in some way for the duration the code runs. Even if you keep yourself active in that session, Cloud Shell still issues tokens valid for 1h, so the cmdlets running will start erroring out after that time, with the dreaded “The access token expiry UTC time <time> is earlier than current UTC time <current time>“. To get an idea about the time the code above in listing 27 takes, running it across 4k VMs homed in 150+ subscriptions took about 20 minutes.

You might also get errors reported when running, such as “The current subscription type is not permitted to perform operations on any provider namespace. Please use a different subscription“. This will evidently result in a lower number of VMs in the final report as opposed to what actually exists.

One word of warning: consider using the Az module, as that’s the only one going forward, as detailed here. AzureRM is being discontinued, and also doesn’t work with Powershell 7, as discussed on this StackOverflow thread.

Alternative: Non-ARG Azure CLI

The problem with Azure CLI and the “classic”, non-ARG commands, is that you have to work against one subscription at a time, same as with its Powershell counterpart, as explained here. Not that it doesn’t mean you’re not allowed to run things in parallel (as we’ll see a bit later), but the jobs you invoke have to act against a certain subscription.

The fact that the subscription context needs to be switched often has come up in the past, unfortunately, it appears that at least as of now, changing the underlying code to make this less tedious is not that easy, as described at length here.

One important question is whether Azure CLI can retrieve classic VMs? CLI 2+ doesn’t have support for ASM. As described here in the note, for the classic deployment model, the Azure classic CLI must be installed. The CLIs are invoked differently, with v1 using azure, and v2 using az. Cloud Shell only appears to support version 2 of the CLI.

The current version of Azure CLI at the time of this writing is 2.12. Let’s use it to work towards our goal, of showing all private and public IPs for all VMs. If you’re using it from a local machine, use az login first; if you’re using Cloud Shell bash, you’ll get authenticated directly.

The nice thing about the CLI is that you can quickly get all the private and public IPs, without having to resort to anything extra. So the simple command az vm list -d --query "[].{Name:name, PublicIPs:publicIps, PrivateIPs:privateIps}" -o table will return the VMs in the current context (current subscription) and parse the IPs nicely:

As for the command itself: the -d switch retrieves all the details for the VMs (without it you’ll get neither the private nor the public IPs). The [] simply flattens the current array, as described here, while the following part just rewrites the names of the columns in the final output. The table is just one of the the various outputs that Azure CLI supports.

But this was running against a single subscription, and we want to get the output for all the Azure subscriptions in the tenant.

What we’ll do is get a list of all subscriptions first, then iterate through them, point the current context to each in turn, followed by exporting the data for that particular subscription. The command becomes: for i in `az account list --query "[].{id:id}" --output tsv`; do az account set --subscription $i; az vm list -d --query "[].{Name:name, PublicIPs:publicIps, PrivateIPs:privateIps}" --output tsv; done

It might look like magic at first, but not quite: for simply iterates through the list of Azure subscription ids, which is obtained with the az account list command that only returns the id of the subscriptions using the --query parameter. Inside the loop itself, 2 operations are performed: switching to a new subscription (az account set…) followed by extracting the VM information from that subscription as we’ve seen previously.

The problem with this command is that it’s running synchronously, thus retrieving results per one subscription at a time only. We can easily make this run asynchronously, by having just a single operator added. & schedules the jobs in the for loop to run in parallel in the background, as seen here. Let’s also write the output to a file, and make sure this file is removed in the beginning, if it exists. The bash command for Cloud Shell, using background jobs, becomes:

rm VMs.csv -f; for i in `az account list --query "[].{id:id}" --output tsv`; do az account set --subscription $i; az vm list -d --query "[].{Name:name, PublicIPs:publicIps, PrivateIPs:privateIps}" --output tsv >> VMs.csv & done; wait

Listing 28 – Retrieving all private and public IPs for all ARM VMs within an Azure tenant, from a bash shell, using background jobs

Writing works in parallel, as each background job that happens to finish will append its data to the CSV file. The >> is the append operator in bash (> writes to the file, but overwrites). One thing to be aware of is that there’s no ordering whatsoever, as background jobs write as soon as they finish, and there’s also no guarantee that there’s ordering in each az vm list command (as explained here).

One important thing to notice is that if wait is not used, you’ll most likely miss data: background jobs will keep writing to the output file even after control is returned to the console, so copying the output file after the command wrongly appears to have finished will result in partial output only. With wait, the shell will wait for all the background jobs to complete.

As for the numbers, the time it took to go through roughly 4,000 ARM VMs homed in more of 150 subscriptions – with the parallel background jobs – was a bit under 10 minutes. Compare this to the synchronous version before, which takes in excess of 40 minutes. Not bad at all. Yet the question is, as Tim Roughgarden would put it: “Can we do better?”. And as we’ve seen, we certainly can – in about 10 seconds – by using ARG.

If using Excel to work with the output file, make sure you’re importing the file by using tab as the delimiter, otherwise it will split columns by default using a comma, which is not what we want, given that only multiple IPs are separated by a comma. After all, tsv in the output type stands for “tab-separated values”. Also, note that no column header is added to the file.

Same as for the non-ARG Powershell approach, you might run into “The current subscription type is not permitted to perform operations on any provider namespace. Please use a different subscription“. Although this will occur less than in Powershell, I don’t know what exactly causes this, but I’ll update the article when I find out.

From an Azure CLI session running on a Windows box, the command is slightly different. Unlike the bash version, we’ll opt to get the name column – instead of the id – explicitly in the command that returns the subscription names, and use delimiters with FOR /F to handle whitespace within the subscriptions’ names, by specifying the separator to be something else than space, as described here. Inside the for loop, the same 2 actions are performed: switching the context to the current subscription and retrieving the corresponding list of VMs together with the name and IP details.

FOR /F "DELIMS=|" %S IN ('az account list --query "[].{Name:name}" --output tsv') DO (az account set --subscription "%S" & az vm list -d --query "[].{Name:name, PublicIPs:publicIps, PrivateIPs:privateIps}" --output table)

Listing 29 – Retrieving all private and public IPs for all ARM VMs within an Azure tenant, from a Windows command prompt

In this context, & makes sure that the commands linked by it run one after another, as described here.

Important: please note that this section looked specifically into non-ARG Azure CLI commands for retrieving the private and public IPs for Azure VMs. Azure CLI itself supports Azure Resource Graph (ARG) just fine through the az graph command. The extension “resource-graph” – currently in preview as of Sep 2020 – is needed (Cloud Shell will prompt you to install this automatically), and then you can easily run the ARM query (in listing 20) using az graph query -q "<ARG_query>", with the same lightning speed.

Q&A

Azure Resource Graph (ARG)

Q: This Kusto language looks complicated. Where can I begin with some really basic stuff?
A: You can start from this Kusto tutorial here https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/tutorial?pivots=azuredataexplorer. Once you master the basics, you can move over to Azure Resource Graph queries, here https://docs.microsoft.com/en-us/azure/governance/resource-graph/samples/starter?tabs=azure-cli and here https://docs.microsoft.com/en-us/azure/governance/resource-graph/samples/advanced?tabs=azure-cli.

Q: Is this Kusto language brand new?
A: According to the history of Kusto here, the language first showed up in 2014.

Q: Aren’t there multiple Kusto query statements within some of the samples in this article?
A: According to the article here https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/, “the query consists of a sequence of query statements, delimited by a semicolon (;)“. Semicolons aren’t used in any of the queries in this article, therefore each one is a single query statement.

Q: When running a query in ARG Explorer, I get “Query result set has exceeded the limit. Showing first 1000 of…“. How can I get to the second page of the result set (rows 1001-2000)?
A: As of end of Sep 2020 you shouldn’t be hitting that problem anymore, as the ARG Explorer now has pagination. Before this got introduced however, one needed to serialize the data, then add the row number, followed by filtering for a specific “rolling window” in order to get to the right page in the results.

For our ARM query for example, we already have the data sorted (therefore serialized), so the only remaining thing left to do was adding the following 2 lines at the end of listing 20 in order to retrieve the rows 3000-3999 of that query. Note that the row_number function (described here) is 1-based.
| extend rn=row_number()
| where rn>3000

Q: Back in figure 2, are sku and plan dynamic types or primitive types (eg string)?
A: They’re dynamic types. You can spot this by their null values in the respective figure, which is one of the 4 incarnations of a dynamic type, as seen above.

Q: I’m trying to run the simple join samples here https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/joinoperator?pivots=azuredataexplorer, but for some reason this can’t be done in the Azure Resource Graph Explorer.
A: Use instead the UI here https://dataexplorer.azure.com/clusters/help/databases/Samples to run samples.

Q: I’m trying to run a Kusto query in ARG that’s using the join operator. Specifically I want to get all the matches for values on the right table that aren’t present in the left table. From the join operator’s documentation I’ve picked up the rightanti join flavor. But every time I run it I get “(Code: InvalidQuery) The join kind “RightAntiSemi” is not supported or not allowed. (Code: UnsupportedJoinFlavor)“
A: Remember that ARG only supports a subset of the Kusto query language. The actual functionalities that are either allowed or not are presented here. Note that for the join operator it’s specifically listed that “Join flavors supported: innerunique, inner, leftouter. Limit of 3 join in a single query. Custom join strategies, such as broadcast join, aren’t allowed. May be used within a single table or between the Resources and ResourceContainers tables.“

Q: I’m trying to solve the problem back in listing 17, by using on $left.vmId =~ $right.vmId instead of using tolower(), so that this rule is applied by the join operator. The =~ will do the match case-insensitive. But running the modified query doesn’t work, and instead the following error is thrown: (Code: InvalidQuery) join: Only equality is allowed in this context. (Code: Default). What’s wrong?
A: If you cross-check join‘s documentation you’ll find that the equality-by-value rule is only allowed with the explicit == operator.

Q: I’m trying to find the GitHub repositories for Azure Resource Graph (ARG) and Azure Resource Graph Explorer (ARGE) so I can contribute / look at current issues, but I can’t seem to be able to find them.
A: ARG and ARGE are developed completely within Microsoft, as opposed to an open source model, as Microsoft Graph Explorer is for example.

Q: In this article it’s stated that “First currently has a maximum allowed value of 5000, which it achieves by paging results 1000 records at a time.” Wouldn’t it be more efficient to repeated queries and retrieving only the first 1000 results, as opposed to relying on the Search-AzGraph to perform the pagination itself against the 5000 maximum value for the -First parameter?
A: No, as you’re paying the overhead for sending/receiving the smaller requests. Here’s a look against 3000 results – the first runtime is computed against the query ran a single time, while the second running the query 3 times on 1000-capped rows per query:

Q: Is sorting required for pagination to work with Search-AzGraph?
A: From my experiments with v0.7.7 of the Az.ResourceGraph module that contains this cmdlet, the outcome of an unsorted query is wildly different whether you have an id column in your query’s output or not. If you don’t have the id in the query (such as the one in listing 20), then Search-AzGraph‘s pagination mechanism (-First and -Skip) is guaranteed not to work correctly (and as such, the pagination code in listing 22 will be broken as well). There are 2 concerns: consistency and “skip” functionality, and neither works as expected when the id is missing. For the first issue, consistency, take the query and its result below:

This shows how running the very same command returns different results, although the Azure infrastructure wasn’t changed in any way. The results were captured by running the command in succession in under 20 seconds.

For the “skip” functionality, this fails consistently. Even more, if the value for -Skip is large enough (larger even than the number of entries in the result set), then you’ll still get results back, in a sort of wrap-around bug, as seen below for the same query:

If you keep the original column containing an id, pagination appears to work even without sorting. Let’s discuss the 2 concerns above for this case: consistency looks to work as expected, at least from my tests, as I could not reproduce the issue seen in first photo of this answer. As for the “skip” functionality, again based on my own testing, appears to work ok, and also the wrap-around bug doesn’t seem to occur.

Yet even if you have the id in your query, it still doesn’t mean that it’ll always work, and using it as such will expose you to the mercy of the internal cmdlet’s implementation – as it may or may not use the original id column as the primary key – leaving you with different outcomes if you run the same cmdlet multiple times, or potentially buggy results.

Sorting is recommended – although strangely not made a requirement – by Microsoft in its own documentation here. Most likely this is tied to the notion of serializing the row sets, as described here, as sorting is one way to achieve it. Bottom line: sort the result if doing pagination with Search-AzGraph.

Q: I’ve come across an important note in this article https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/work-with-data: “When First is configured to be greater than 1000 records, the query must project the id field in order for pagination to work. If it’s missing from the query, the response won’t get paged and the results are limited to 1000 records.” Is this real?
A: Yes. Below you can see the result of running Search-AzGraph by specifying it should return the first 2000 network interfaces. The first query only projects the name of the vmNics, and discards the rest of the columns, including the id. When the query runs, only 1000 results are returned, just like the article states. The second query keeps all the columns, including the id for the vmNics. When this query runs, all 2000 results are returned:

Q: I’m trying to do pagination using the Search-AzGraph cmdlet against a query that contains the limit operator, and I’m seeing a strange outcome when trying to use the -Skip and -First parameters as described here https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/work-with-data#paging-results. Specifically, consider the query below, which retrieves all the vmNics in a test Azure tenant:

Limiting the number of results to 2, using the limit operator within the query itself, works as expected as seen in the first output below. Using the Search-AzGraph‘s -First parameter to obtain only the first row also works as expected, as the 2nd output shows. But trying to display the first row after skipping the very first element – which in essence should yield the 2nd row – doesn’t work as expected. Note in the 3rd output below that the vmNic returned is still the first one, as opposed to the second one.

Why is this happening?

A: It’s a known limitation with Search-AzGraph and the limit Kusto operator. At the time of this writing – Sep 2020 – the referenced article doesn’t explicitly tell about this known limitation. I have discussed with Microsoft Support, and the Product Team is due to update the article. Hopefully by the time you read this, it’s already done. Note that the problem can’t be fixed by serializing (eg via sorting) the results, neither by keeping the id in the result set. Update 10/6/2020: On Oct 1st, Microsoft has updated their documentation here https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/query-language#supported-tabulartop-level-operators to state that limit doesn’t work with -Skip.

As it turns out, Microsoft Graph behaves in a similar way when doing pagination against it, couple with top, as it was discussed in an earlier article here.

Q: In the output of Search-AzGraph, I can’t see some of the VMs I know I have access to. What’s going on?
A: If for any reason you don’t see VMs returned that you know you have access to (eg they’re in subscriptions where you already have access) see the last note here https://docs.microsoft.com/en-us/azure/governance/resource-graph/first-query-powershell#run-your-first-resource-graph-query about the default context.

Q: I would like to see what Search-AzGraph is actually doing behind the covers. Sure, I can use Fiddler locally to look inside the request, but what to do when working from Cloud Shell?
A: Use -Debug with the cmdlet. You’ll see the query itself, pagination settings, http headers, etc

Q: How can I see the list of providers that ARG is using, along with their version?
A: Use the Kusto query here https://docs.microsoft.com/en-us/azure/governance/resource-graph/samples/advanced?tabs=azure-cli#apiversion

Q: Back in listing 22, why not loop while the number of results returned is greater than 0, instead of verifying whether the last result set had a size equal to that of the page length?
A: Doing that will trigger another query to be sent, which will be guaranteed to return 0 results. And Search-AzGraph will generate the following warning “WARNING: Unable to paginate the results of the query. Some resources may be missing from the results. To rewrite the query and enable paging, see the docs for an example:https://aka.ms/arg-results-truncated“. The warning will still be generated in the script as it’s written in the article, if the number of the last result set is equal to that of the size of the page, since the next query will again return 0 results.

Q: What’s the parent VM id for a disconnected vmNic? Is it null?
A: Once a vmNic is disconnected from the VM it’s attached to, its parent VM id becomes null. So for example the value highlighted in figure A+15 would become null if that respective vmNic is removed from its parent VM.

Q: Is there an official legend of the icons within ARGE on the left side?
A: There’s a grid icon for the “resources” table, which makes sense. It would appear further that things are simple, with horizontal-lines-icon indicating primitive types, while the grid-icon represents a dynamic type. But double-checking with Microsoft Support turned out that this isn’t the case. I’ve created a user voice entry here https://feedback.azure.com/users/1609311493.

Q: Aside from the “resources” table, what do the rest of the tables seen in ARGE on the left side do?
A: The tables seen in ARGE on the left side are all described here https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/query-language#resource-graph-tables.

Q: I’m using a projected column whose values are copied from one that’s in the Resources table, and whose type appears to be string. Why am I getting an error that the type is dynamic? This was the case in this article’s figure 14, where the properties.IPConfigurations[indexer].properties.publicIPAddress.id slot had to be converted to string first. But if one looks at the schema, it would appear that that is already the case:

A: I’ve gotten in touch with Microsoft Support, and the verbatim answer was that “any value extracted from a dynamic column has a type of dynamic. Since properties is a dynamic column, properties.IPConfigurations[indexer].properties.publicIPAddress.id is a dynamic value as well. This is by design. To use the join operator on publicIpAddress you’ll need to call tostring() first to transform them into strings“.

Q: Can I be sure of the type seen in the Azure Resource Graph Explorer (ARGE) in Schema explorer on the left? Eg can I be sure that properties.IPConfigurations[indexer].properties.publicIPAddress.id is a string?
A: As per the previous question, that particular slot is not a string. As for the types seen in the Schema explorer, what you see is not the full story. As per Microsoft Support: “Regarding to types in the schema explorer, we show the type of publicIpAddress.id as string since we evaluated periodically the type of inner fields inside properties. However we know those types as a aftermath and there is no guarantee that, for example, starting from tomorrow the ip will have a different type, or it may not be there at all. This means when executing queries, the type info is not there in the context.“

Q: Can both dynamic and static IPs be retrieved using ARG?
A: Both dynamic and static IPs can be retrieved using ARG for VMs deployed using the ARG model. There was an article here written about a year ago, stating that dynamic IP addresses couldn’t be retrieved using ARG. However checking with Microsoft Support, which in turn got in touch with the Product Group, confirmed that currently both static and dynamic IP addresses can be retrieved.

Q: I’m getting “No tenant found in the context.  Please ensure that the credentials you provided are authorized to access an Azure subscription, then run Connect-AzAccount to login” while running Search-AzGraph. What can I do to solve this?
A: Run Clear-AzContext followed by Connect-AzAccount, then retry the query. This is very nicely described here https://johan.driessen.se/posts/Fixing-the-missing-Azure-Context-in-Azure-Powershell/

Q: I tried using the command in listing 29 on a Windows machine, by saving it as a .cmd file, then running that inside a command prompt. I do have Azure CLI correctly installed, but there seems to be a problem with that file. What’s wrong?
A: If you’re using a batch file, you need to use %% for variables instead of %, as described here https://ss64.com/nt/for.html.

Q: Is there a way to supply the Kusto queries in an embedded direct link, like some of MS’s own documentation does?
A: Yes, simply encode the Kusto query using an online URL encoder (such as this), then append this to https://portal.azure.com/?feature.customportal=false#blade/HubsExtension/ArgQueryBlade/query/.

Q: Can I use Kusto.Explorer to connect directly to the Azure Resource Graph database for my Azure tenant?
A: No. I did talk to Microsoft Support, and they explicitly stated that “ARG database is fully managed by Microsoft and you will not be connecting to it directly in Kusto.Explorer“.

Q: A feature in Azure Resource Graph Explorer (ARGE) is not working as expected, and Microsoft Support is telling me that it will take a while to be fixed. What can I do in the meantime? Eg here’s a current bug whereby the Details tab doesn’t show anything:

A: Try using the preview version of the Azure portal, where the bug might have been already fixed, or not present at all: https://preview.portal.azure.com/.

ARM Model

Q: Where can I read about the networking model under ARM, and how the vmNics, VNets, subnets, public IP addresses and all the other types of objects come together?
A: A very good description of the networking concepts is here https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-multiple-ip-addresses-portal, in the very first section. A discussion around public/private IP addresses, with some very interesting notes, is here https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface-addresses. Using multiple vmNics is also described in this older post here https://azure.microsoft.com/en-us/blog/multiple-vm-nics-and-network-virtual-appliances-in-azure/.

Q: Why is the Azure resource group name sometimes showing up with different casing, prompting the use of tolower() for consistency? Is this a bug?
A: According to this GitHub comment, it’s by design.

Q: Can an additional IP configuration be added to an existing vmNic while the parent VM is running?
A: Yes. Unlike adding a new vmNic, which requires stopping the VM, a new IP configuration can be added to a vmNic while the VM is running.

Q: Can a VM be left without any vmNic after it has been created?
A: The last vmNic hooked to a VM cannot be detached, as described here https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface-vm#remove-a-network-interface-from-a-vm in the note: “If only one network interface is listed, you can’t detach it, because a virtual machine must always have at least one network interface attached to it.”

Q: For one vmNic attached to a VM, can one of its IP configurations be pointed to one subnet, while a different IP configuration made to point to a different subnet?
A: No. The association to a VNet’s subnet is done at the vmNic level, therefore all its IP configurations will be hooked to the same subnet.

Q: Can there be a vmNic without a private IP? Eg just a vmNic that only has a public IP?
A: For IPv4 at least, a private IP is required for a vmNic, as clearly stated here https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface-addresses#ipv4. It follows that the answer to the 2nd question is also ‘no’. It’s the public IPs that are optional.

Q: I always get prompted to enter a Context when using Select-AzSubscription -Name <name>. What’s wrong?
A: Select-AzSubscription is an alias of Set-AzContext (you can quickly check using Get-Alias Select-AzSubscription | fl). An Azure Context consists of more than just a reference to a subscription, as it’s detailed here https://docs.microsoft.com/en-us/powershell/azure/context-persistence?view=azps-4.7.0#overview-of-azure-context-objects. To work around it, for an uniquely named subscription, just use Get-AzSubscription | ? { $_.Name -like "<name>" } | Select-AzSubscription.

Q: I’m trying to add a vmNic to an Azure VM, but the “Attach network interface” option on the Networking blade is greyed out. I’ve checked the Azure VM Size spreadsheet and my VM supports the number of vmNics I have in mind. What’s wrong?
A: Most likely your VM is running. You need to shut it down and bring it in a “Stopped (deallocated)” state before adding the new vmNic, as described here https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface-vm#add-a-network-interface-to-an-existing-vm.

Q: I have a ARM VM with one vmNic that’s connected to a virtual network (VNet). Can I attach another vmNic and connect it to a different VNet?
A: No. All the vmNics that you add to a VM must be connected to the same virtual network, as described here https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface-vm#add-a-network-interface-to-an-existing-vm. Of course, nothing prevents you from connecting each vmNic to a different subnet within that VNet.

General Azure

Q: How can Cloud Shell export CSV files, and most importantly how can one download them?
A: See https://docs.microsoft.com/en-us/azure/cloud-shell/persisting-shell-storage#transfer-local-files-to-cloud-shell

Q: Where can I get more info about “model view” and “instance view”?
A: That’s a good question, and unfortunately I currently don’t have an answer. But I did mentioned the problem here.

Q: How did you get to the cryptic one liner back in listing 28?”
A: Honestly, by reading a lot of Stack Overflow posts, trial-and-error and even running into almost what I was after (like this https://www.reddit.com/r/AZURE/comments/6fdt5k/azurecli_command_to_get_all_public_ips_of_all/ or this https://lnx.azurewebsites.net/bash-script-to-start-or-deallocate-all-vms-in-resource-group/ or this https://azsec.azurewebsites.net/2019/01/29/query-private-ip-address-using-azure-cli/), given that bash is not really my thing. The fact that I had to look up how to clear the current command gives a hint about my general ability with it.

Q: How did you measure the time it took for the Azure CLI bash command in listing 28 to run?
A: To find out the time required to run the bash command, simply hook date +"%T" at the beginning and at the end, like so: date +"%T"; for i in az account list --query "[].{id:id}" --output tsv;do az account set --subscription $i; az vm list -d --query "[].{Name:name, PublicIPs:publicIps, PrivateIPs:privateIps}" --output tsv >> VMs.csv & done; wait; date +"%T". What date does is pretty obvious, what’s not so obvious is the %T format, which simply outputs the time (minus the date).

Q: My Cloud Shell bash session is running a command but I can’t stop it in any way. Ctrl+C doesn’t work. What can I do?
A: Press Ctrl+Z.

Q: My Cloud Shell bash session is running a command that had invoked background jobs of which some are still running. If I press Ctrl+Z the background jobs still seem to be running. How can I terminate all of them?
A: Get the cursor back – eg by pressing Ctrl+Z, followed by Ctrl+C – then issue pkill -f <pattern>. For example, to cancel all the background jobs invoked by the commands in listing 28, we’ll use the fact that all the jobs get spawned by the az command, thus we can run pkill -f az