James E. Blair 91804a5e16 Azure: switch to Azul

The Azure SDK for Python uses threads to manage async operations.
Every time a virtual machine is created, a new thread is spawned
to wait for it to finish (whether we actually end up polling it or
not).  This will cause the Azure driver to have significant
scalability limits compared to other drivers, possibly limiting
the number of simultaneous nodes to 50% compared to others.

To address this, switch to using a very simple requests-based
REST client I'm calling Azul.  The consistency of the Azure API
makes this simple.  As a bonus, we can use the excellent Azure
REST API documentation directly, rather that mapping attribute
names through the Python SDK (which has subtle differences).

A new fake Azure test fixture is also created in order to make
the current unit test a more thorough exercise of the code.

Finally, the "zuul-private-key" attribute is misnamed since we
have a policy of a one-way dependency from Zuul -> Nodepool.  It's
name is updated to match the GCE driver ("key") and moved to the
cloud-image section so that different images may be given different
keys.

Change-Id: I87bfa65733b2a71b294ebe2cf0d3404d0e4333c5

2021-03-08 14:58:31 -08:00

6.2 KiB

Raw Blame History

zuul

Azure Compute Driver

Selecting the azure driver adds the following options to the providers section of the configuration.

providers.[azure]

An Azure provider's resources are partitioned into groups called pool, and within a pool, the node types which are to be made available are listed

Note

For documentation purposes the option names are prefixed providers.[azure] to disambiguate from other drivers, but [azure] is not required in the configuration (e.g. below providers.[azure].pools refers to the pools key in the providers section when the azure driver is selected).

Example:

providers:
   - name: azure-central-us
     driver: azure
     resource-group-location: centralus
     location: centralus
     resource-group: nodepool
     auth-path: /Users/grhayes/.azure/nodepoolCreds.json
     subnet-id: /subscriptions/<subscription-id>/resourceGroups/nodepool/providers/Microsoft.Network/virtualNetworks/NodePool/subnets/default
     cloud-images:
       - name: bionic
         username: zuul
         key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAA...
         image-reference:
           sku: 18.04-LTS
           publisher: Canonical
           version: latest
           offer: UbuntuServer
     pools:
       - name: main
         max-servers: 10
         labels:
           - name: bionic
             cloud-image: bionic
             hardware-profile:
               vm-size: Standard_D1_v2
             tags:
               department: R&D
               purpose: CI/CD

name

A unique name for this provider configuration.

location

Name of the Azure region to interact with.

resource-group

Name of the Resource Group in which to place the Nodepool nodes.

resource-group-location

Name of the Azure region where the home Resource Group is or should be created.

auth-path

Path to the JSON file containing the service principal credentials. Create with the Azure CLI and the --sdk-auth flag

subnet-id

Subnet to create VMs on

cloud-images

Each entry in this section must refer to an entry in the labels section.

cloud-images:
  - name: bionic
    username: zuul
    image-reference:
      sku: 18.04-LTS
      publisher: Canonical
      version: latest
      offer: UbuntuServer
  - name: windows-server-2016
    username: zuul
    image-reference:
       sku: 2016-Datacenter
       publisher: MicrosoftWindowsServer
       version: latest
       offer: WindowsServer

Each entry is a dictionary with the following keys

name

Identifier to refer this cloud-image from labels section. Since this name appears elsewhere in the nodepool configuration file, you may want to use your own descriptive name here.

username

The username that a consumer should use when connecting to the node.

key

The SSH public key that should be installed on the node.

image-reference

sku

Image SKU

publisher

Image Publisher

offer

Image offers

version

Image version

pools

A pool defines a group of resources from an Azure provider. Each pool has a maximum number of nodes which can be launched from it, along with a number of cloud-related attributes used when launching nodes.

name

A unique name within the provider for this pool of resources.

labels

Each entry in a pool's labels section indicates that the corresponding label is available for use in this pool. When creating nodes for a label, the flavor-related attributes in that label's section will be used.

labels:
  - name: bionic
    cloud-image: bionic
    hardware-profile:
      vm-size: Standard_D1_v2

Each entry is a dictionary with the following keys

name

Identifier to refer this label.

cloud-image

Refers to the name of an externally managed image in the cloud that already exists on the provider. The value of cloud-image should match the name of a previously configured entry from the cloud-images section of the provider.

hardware-profile

vm-size

VM Size of the VMs to use in Azure. See the VM size list on azure.microsoft.com for the list of sizes availabile in each region.

6.2 KiB Raw Blame History

Azure Compute Driver

6.2 KiB

Raw Blame History