Troubleshooting Azure Subnet Issues Causing Ephemeral Runner Creation Failures

Hey guys! Having trouble with those ephemeral runners failing to create in your Azure subnets? It's a frustrating issue, especially when things were working smoothly before. Let's dive into a common scenario and how to tackle it.

Understanding the Problem

So, you're using Azure CLI and GitHub Actions to spin up VMs on demand, right? Suddenly, you're seeing errors like "Subnet does not exist," even though you know the subnet is there. It's like your Azure resources are playing hide-and-seek! This issue often pops up randomly, with some jobs succeeding while others fail. You might be using an Azure CLI command similar to this:

az vm create --resource-group cstrm-p-eaus-sast-rg --name appsec-ephemeral-$(date '+%Y%m%d%H%M%S')-$GITHUB_RUN_NUMBER --image ${{ env.image }} --public-ip-sku Standard --nic-delete-option delete --os-disk-delete-option delete --admin-username azureuser --admin-password ${{ secrets.AZURE_PASSWORD_AZUSR }} --size  ${{ env.size }} --public-ip-address "" --subnet  ${{ env.subnet }} --nsg ${{ env.nsg }} --tags ${{ env.tags }} --user-data ${{ env.user-data }} --security-type Standard

And you're staring at an error message that screams:

ERROR: Subnet '/subscriptions/04b33af8-29f6-436d-9639-d07a1300084c/resourceGroups/aa-cstrm-network-eastus-rg/providers/Microsoft.Network/virtualNetworks/prod-eastus-10-227-32-0_19-vnet/subnets/iaas-10-227-32-0_22-snet' does not exist.

Sounds familiar? Let's break down the potential causes and solutions.

Why is This Happening?

This "Subnet does not exist" error, even when the subnet does exist, can be a real head-scratcher. Here's a breakdown of the most likely culprits:

• Timing Issues and Resource Propagation: Azure is a powerful cloud platform, but sometimes things don't happen instantly. When you're creating resources in rapid succession, there can be delays in resource propagation. This means that a resource, like your subnet, might not be fully available across all Azure services when your VM creation script tries to use it. It's like trying to use a freshly baked cake – it looks delicious, but it needs time to cool and set properly. In the context of Azure, this means that while the subnet has been created in the control plane, it might not be fully synchronized across all the data plane nodes that handle VM creation requests. This is a common cause of intermittent failures, where a command works sometimes and fails other times, seemingly at random. The key here is understanding that cloud environments have inherent latency, and your automation needs to account for this.

• Azure CLI and API Version Incompatibilities: The Azure CLI relies on the Azure Resource Manager APIs to perform actions. Sometimes, using an older version of the CLI or specifying an incompatible API version in your scripts can lead to unexpected behavior. Think of it like using an old key for a new lock – it just won't work. If your script is explicitly setting an API version (like 2025-01-01 in the example), make sure it's compatible with your Azure subscription and the resources you're trying to create. Older API versions might not fully support newer features or have different validation rules, leading to errors even if your configuration seems correct. Always strive to use the latest stable versions of the Azure CLI and avoid pinning to specific API versions unless absolutely necessary, as this reduces the risk of compatibility issues.

• Intermittent Network Glitches: Cloud networks are complex beasts. Occasional network hiccups can occur, causing temporary connectivity problems. These glitches can disrupt the communication between your VM creation process and the Azure networking services, leading to the "Subnet does not exist" error. It's like a brief phone line disconnection – the call drops, even though both phones are working perfectly. These glitches are often transient and self-correcting, but they can still trigger failures in automated processes that don't have proper error handling. While you can't completely eliminate network glitches, you can build resilience into your scripts by adding retry logic and error handling, which we'll discuss later.

• Resource Locking Issues: Azure Resource Manager uses locking mechanisms to prevent conflicting operations from modifying the same resource simultaneously. If a lock is held on the subnet (perhaps due to a previous failed operation or another process modifying the virtual network), your VM creation might be blocked, resulting in the error. It's like a traffic jam – everyone's stuck because one car is blocking the road. Resource locks are designed to ensure consistency and prevent data corruption, but they can sometimes be the source of unexpected errors, especially in highly automated environments. You can use the Azure portal or CLI to check for existing locks on your resources and release them if necessary.

How to Fix It: Your Troubleshooting Toolkit

Alright, enough with the doom and gloom. Let's arm ourselves with some practical solutions to conquer this subnet issue.

1. Implement Retries with Exponential Backoff: This is your best friend when dealing with intermittent cloud issues. Instead of giving up after the first failure, add logic to your script to retry the VM creation a few times. Exponential backoff means increasing the delay between retries, giving Azure's services time to catch up. Imagine it like gently nudging a door that's stuck – you don't want to force it, but you keep trying with increasing persistence. Here's a basic example of how you might implement retries in a Bash script:

   MAX_RETRIES=5
   RETRY_DELAY=5 # seconds
   for i in $(seq 1 $MAX_RETRIES); do
     az vm create ... # Your VM creation command
     if [ $? -eq 0 ]; then # Check if the command succeeded
       echo