Zuul b0fa778ded Merge "OpenShift/k8s Provider: Allow passing env vars to Pods"

2020-07-30 20:17:47 +00:00

75 KiB

Raw Blame History

zuul

Configuration

Nodepool reads its configuration from /etc/nodepool/nodepool.yaml by default. The configuration file follows the standard YAML syntax with a number of sections defined with top level keys. For example, a full configuration file may have the diskimages, labels, and providers sections:

diskimages:
  ...
labels:
  ...
providers:
  ...

The following sections are available. All are required unless otherwise indicated.

Options

webapp

Define the webapp endpoint port and listen address

port

The port to provide basic status information

listen_address

Listen address for web app

elements-dir

If an image is configured to use diskimage-builder and glance to locally create and upload images, then a collection of diskimage-builder elements must be present. The elements-dir parameter indicates a directory that holds one or more elements.

images-dir

When we generate images using diskimage-builder they need to be written to somewhere. The images-dir parameter is the place to write them.

Note

The builder daemon creates a UUID to uniquely identify itself and to mark image builds in ZooKeeper that it owns. This file will be named builder_id.txt and will live in the directory named by the images-dir option. If this file does not exist, it will be created on builder startup and a UUID will be created automatically.

build-log-dir

The builder will store build logs in this directory. It will create one file for each build, named <image>-<build-id>.log; for example, fedora-0000000004.log. It defaults to /var/log/nodepool/builds.

build-log-retention

At the start of each build, the builder will remove old build logs if they exceed this value. This option specifies how many will be kept (usually you will see one more, as deletion happens before starting a new build). By default, the last 7 old build logs are kept. Set this to -1 to disable removal of logs.

zookeeper-servers

Lists the ZooKeeper servers uses for coordinating information between nodepool workers.

zookeeper-servers:
  - host: zk1.example.com
    port: 2181
    chroot: /nodepool

Each entry is a dictionary with the following keys

host

A zookeeper host

port

Port to talk to zookeeper

chroot

The chroot key, used for interpreting ZooKeeper paths relative to the supplied root path, is also optional and has no default.

zookeeper-tls

To use TLS connections with Zookeeper, provide this dictionary with the following keys:

cert

The path to the PEM encoded certificate.

key

The path to the PEM encoded key.

The path to the PEM encoded CA certificate.

labels

Defines the types of nodes that should be created. Jobs should be written to run on nodes of a certain label. Example

labels:
  - name: my-precise
    max-ready-age: 3600
    min-ready: 2
  - name: multi-precise
    min-ready: 2

Each entry is a dictionary with the following keys

name

Unique name used to tie jobs to those instances.

max-ready-age

Maximum number of seconds the node shall be in ready state. If this is exceeded the node will be deleted. A value of 0 disables this.

min-ready

Minimum number of instances that should be in a ready state. Nodepool always creates more nodes as necessary in response to demand, but setting min-ready can speed processing by attempting to keep nodes on-hand and ready for immedate use. min-ready is best-effort based on available capacity and is not a guaranteed allocation. The default of 0 means that nodepool will only create nodes of this label when there is demand. Set to -1 to have the label considered disabled, so that no nodes will be created at all.

max-hold-age

Maximum number of seconds a node shall be in "hold" state. If this is exceeded the node will be deleted. A value of 0 disables this.

This setting is applied to all nodes, regardless of label or provider.

diskimages

This section lists the images to be built using diskimage-builder. The name of the diskimage is mapped to the providers.[openstack].diskimages section of the provider, to determine which providers should received uploads of each image. The diskimage will be built in every format required by the providers with which it is associated. Because Nodepool needs to know which formats to build, if the diskimage will only be built if it appears in at least one provider.

To remove a diskimage from the system entirely, remove all associated entries in providers.[openstack].diskimages and remove its entry from diskimages. All uploads will be deleted as well as the files on disk.

A sample configuration section is illustrated below.

diskimages:
  - name: base
    abstract: True
    elements:
      - vm
      - simple-init
      - openstack-repos
      - nodepool-base
      - cache-devstack
      - cache-bindep
      - growroot
      - infra-package-needs
    env-vars:
      TMPDIR: /opt/dib_tmp
      DIB_CHECKSUM: '1'
      DIB_IMAGE_CACHE: /opt/dib_cache

  - name: ubuntu-bionic
    parent: base
    pause: False
    rebuild-age: 86400
    elements:
      - ubuntu-minimal
    release: bionic
    username: zuul
    env-vars:
      DIB_APT_LOCAL_CACHE: '0'
      DIB_DISABLE_APT_CLEANUP: '1'
      FS_TYPE: ext3

  - name: ubuntu-focal
    base: ubuntu-bionic
    release: focal
    env-vars:
      DIB_DISABLE_APT_CLEANUP: '0'

  - name: centos-8
    parent: base
    pause: True
    rebuild-age: 86400
    formats:
      - raw
      - tar
    elements:
      - centos-minimal
      - epel
    release: '8'
    username: centos
    env-vars:
      FS_TYPE: xfs

Each entry is a dictionary with the following keys

name

Identifier to reference the disk image in providers.[openstack].diskimages and labels.

abstract

An abstract entry is used to group common configuration together, but will not create any actual image. A diskimage marked as abstract should be inherited from in another diskimage via its diskimages.parent attribute.

An abstract entry can have a diskimages.parent attribute as well; values will merge down.

parent

A parent diskimage entry to inherit from. Any values from the parent will be populated into this image. Setting any fields in the current image will override the parent values execept for the following:

diskimages.env-vars: new keys are additive, any existing keys from the parent will be overwritten by values in the current diskimage (i.e. Python update() semantics for a dictionary).
diskimages.elements: values are additive; the list of elements from the parent will be extended with any values in the current diskimage. Note that the element list passed to diskimage-builder is not ordered; elements specify their own dependencies and diskimage-builder builds a graph from that, not the command-line order.

Note that a parent diskimage may also have it's own parent, creating a chain of inheritance. See also diskimages.abstract for defining common configuration that does not create a diskimage.

formats

The list of formats to build is normally automatically created based on the needs of the providers to which the image is uploaded. To build images even when no providers are configured or to build additional formats which you know you may need in the future, list those formats here.

In case the diskimage is not used by any provider and no formats are configured, the image won't be built.

rebuild-age

If the current diskimage is older than this value (in seconds), then nodepool will attempt to rebuild it. Defaults to 86400 (24 hours).

release

Specifies the distro to be used as a base image to build the image using diskimage-builder.

build-timeout

How long (in seconds) to wait for the diskimage build before giving up. The default is 8 hours.

elements

Enumerates all the elements that will be included when building the image, and will point to the elements-dir path referenced in the same config file.

env-vars

Arbitrary environment variables that will be available in the spawned diskimage-builder child process.

pause

When set to True, nodepool-builder will not build the diskimage.

username

The username that a consumer should use when connecting to the node.

python-path

The path of the default python interpreter. Used by Zuul to set ansible_python_interpreter. The special value auto will direct Zuul to use inbuilt Ansible logic to select the interpreter on Ansible >=2.8, and default to /usr/bin/python2 for earlier versions.

dib-cmd

Configure the command called to create this disk image. By default this just disk-image-create; i.e. it will use the first match in $PATH. For example, you may want to override this with a fully qualified path to an alternative executable if a custom diskimage-builder is installed in another virutalenv.

Note

Any wrapping scripts or similar should consider that the command-line or environment arguments to disk-image-create are not considered an API and may change.

providers

Lists the providers Nodepool should use. Each provider is associated to a driver listed below.

Each entry is a dictionary with the following keys

name

Name of the provider

max-concurrency

Maximum number of node requests that this provider is allowed to handle concurrently. The default, if not specified, is to have no maximum. Since each node request is handled by a separate thread, this can be useful for limiting the number of threads used by the nodepool-launcher daemon.

driver

The driver type.

aws

For details on the extra options required and provided by the AWS driver, see the separate section providers.[aws]

gce

For details on the extra options required and provided by the GCE driver, see the separate section providers.[gce]

kubernetes

For details on the extra options required and provided by the kubernetes driver, see the separate section providers.[kubernetes]

openshift

For details on the extra options required and provided by the openshift driver, see the separate section providers.[openshift]

openshiftpods

For details on the extra options required and provided by the openshiftpods driver, see the separate section providers.[openshiftpods]

openstack

For details on the extra options required and provided by the OpenStack driver, see the separate section providers.[openstack]

static

For details on the extra options required and provided by the static driver, see the separate section providers.[static]

azure

For details on the extra options required and provided by the Azure driver, see the separate section providers.[azure]

OpenStack Driver

Selecting the OpenStack driver adds the following options to the providers section of the configuration.

providers.[openstack]

Specifying the openstack driver for a provider adds the following keys to the providers configuration.

Note

For documentation purposes the option names are prefixed providers.[openstack] to disambiguate from other drivers, but [openstack] is not required in the configuration (e.g. below providers.[openstack].cloud refers to the cloud key in the providers section when the openstack driver is selected).

An OpenStack provider's resources are partitioned into groups called "pools" (see providers.[openstack].pools for details), and within a pool, the node types which are to be made available are listed (see providers.[openstack].pools.labels for details).

Within each OpenStack provider the available Nodepool image types are defined (see providers.[openstack].diskimages).

providers:
  - name: provider1
    driver: openstack
    cloud: example
    region-name: 'region1'
    rate: 1.0
    boot-timeout: 120
    launch-timeout: 900
    launch-retries: 3
    image-name-format: '{image_name}-{timestamp}'
    hostname-format: '{label.name}-{provider.name}-{node.id}'
    post-upload-hook: /usr/bin/custom-hook
    diskimages:
      - name: trusty
        meta:
            key: value
            key2: value
      - name: precise
      - name: devstack-trusty
    pools:
      - name: main
        max-servers: 96
        availability-zones:
          - az1
        networks:
          - some-network-name
        security-groups:
          - zuul-security-group
        labels:
          - name: trusty
            min-ram: 8192
            diskimage: trusty
            console-log: True
          - name: precise
            min-ram: 8192
            diskimage: precise
          - name: devstack-trusty
            min-ram: 8192
            diskimage: devstack-trusty
  - name: provider2
    driver: openstack
    cloud: example2
    region-name: 'region1'
    rate: 1.0
    image-name-format: '{image_name}-{timestamp}'
    hostname-format: '{label.name}-{provider.name}-{node.id}'
    diskimages:
      - name: precise
        meta:
            key: value
            key2: value
    pools:
      - name: main
        max-servers: 96
        labels:
          - name: trusty
            min-ram: 8192
            diskimage: trusty
          - name: precise
            min-ram: 8192
            diskimage: precise
          - name: devstack-trusty
            min-ram: 8192
            diskimage: devstack-trusty

cloud

Name of a cloud configured in clouds.yaml.

The instances spawned by nodepool will inherit the default security group of the project specified in the cloud definition in clouds.yaml (if other values not specified). This means that when working with Zuul, for example, SSH traffic (TCP/22) must be allowed in the project's default security group for Zuul to be able to reach instances.

More information about the contents of clouds.yaml can be found in the openstacksdk documentation.

boot-timeout

Once an instance is active, how long to try connecting to the image via SSH. If the timeout is exceeded, the node launch is aborted and the instance deleted.

launch-timeout

The time to wait from issuing the command to create a new instance until that instance is reported as "active". If the timeout is exceeded, the node launch is aborted and the instance deleted.

nodepool-id

Deprecated

A unique string to identify which nodepool instances is using a provider. This is useful if you want to configure production and development instances of nodepool but share the same provider.

launch-retries

The number of times to retry launching a server before considering the job failed.

region-name

The region name if the provider cloud has multiple regions.

hostname-format

Hostname template to use for the spawned instance.

image-name-format

Format for image names that are uploaded to providers.

post-upload-hook

Filename of an optional script that can be called after an image has been uploaded to a provider but before it is taken into use. This is useful to perform last minute validation tests before an image is really used for build nodes. The script will be called as follows:

<SCRIPT> <PROVIDER> <EXTERNAL_IMAGE_ID> <LOCAL_IMAGE_FILENAME>

If the script returns with result code 0 it is treated as successful otherwise it is treated as failed and the image gets deleted.

rate

In seconds, amount to wait between operations on the provider.

clean-floating-ips

If it is set to True, nodepool will assume it is the only user of the OpenStack project and will attempt to clean unattached floating ips that may have leaked around restarts.

port-cleanup-interval

If greater than 0, nodepool will assume it is the only user of the OpenStack project and will attempt to clean ports in DOWN state after the cleanup interval has elapsed. This value can be reduced if the instance spawn time on the provider is reliably quicker.

diskimages

Each entry in a provider's diskimages section must correspond to an entry in diskimages. Such an entry indicates that the corresponding diskimage should be uploaded for use in this provider. Additionally, any nodes that are created using the uploaded image will have the associated attributes (such as flavor or metadata).

If an image is removed from this section, any previously uploaded images will be deleted from the provider.

diskimages:
  - name: precise
    pause: False
    meta:
        key: value
        key2: value
  - name: windows
    connection-type: winrm
    connection-port: 5986

Each entry is a dictionary with the following keys

name

Identifier to refer this image from providers.[openstack].pools.labels and diskimages sections.

pause

When set to True, nodepool-builder will not upload the image to the provider.

config-drive

Whether config drive should be used for the image. Defaults to unset which will use the cloud's default behavior.

Static Driver

Selecting the static driver adds the following options to the providers section of the configuration.

providers.[static]

The static provider driver is used to define static nodes.

Note

For documentation purposes the option names are prefixed providers.[static] to disambiguate from other drivers, but [static] is not required in the configuration (e.g. below providers.[static].pools refers to the pools key in the providers section when the static driver is selected).

Example:

providers:
  - name: static-rack
    driver: static
    pools:
      - name: main
        nodes:
          - name: trusty.example.com
            labels: trusty-static
            timeout: 13
            connection-port: 22022
            host-key: fake-key
            username: zuul
            max-parallel-jobs: 1

pools

A pool defines a group of statically declared nodes.

Note

When providing different labels, it is better to have one pool per label to avoid requests being queued when one label is at capacity.

Each entry is a dictionary with entries as follows

name

Pool name

node-attributes

A dictionary of key-value pairs that will be stored with the node data in ZooKeeper. The keys and values can be any arbitrary string.

nodes

Each entry indicates a static node and it's attributes.

name

The hostname or ip address of the static node. This must be unique across all nodes defined within the configuration file.

labels

The list of labels associated with the node.

host-key-checking

Specify custom behavior of validation of host connection. When set to False, nodepool-launcher will not scan the nodes before they are registered. This might be needed if nodepool-launcher and the static nodes are on isolated networks. The default value is True.

timeout

The timeout in second before the ssh ping is considered failed.

connection-type

The connection type that a consumer should use when connecting to the node.

winrm

ssh

connection-port

The port that a consumer should use when connecting to the node. For most nodes this is not necessary. This defaults to 22 when connection-type is 'ssh' and 5986 when it is 'winrm'.

host-key

The ssh host key of the node.

username

The username nodepool will use to validate it can connect to the node.

python-path

max-parallel-jobs

The number of jobs that can run in parallel on this node.

Kubernetes Driver

Selecting the kubernetes driver adds the following options to the providers section of the configuration.

providers.[kubernetes]

A Kubernetes provider's resources are partitioned into groups called pools (see providers.[kubernetes].pools for details), and within a pool, the node types which are to be made available are listed (see providers.[kubernetes].pools.labels for details).

Note

For documentation purposes the option names are prefixed providers.[kubernetes] to disambiguate from other drivers, but [kubernetes] is not required in the configuration (e.g. below providers.[kubernetes].pools refers to the pools key in the providers section when the kubernetes driver is selected).

Example:

providers:
  - name: kubespray
    driver: kubernetes
    context: admin-cluster.local
    pools:
      - name: main
        labels:
          - name: kubernetes-namespace
            type: namespace
          - name: pod-fedora
            type: pod
            image: docker.io/fedora:28

context

Name of the context configured in kube/config.

Before using the driver, Nodepool either needs a kube/config file installed with a cluster admin context, in which case this setting is required, or if Nodepool is running inside Kubernetes, this setting and the kube/config file may be omitted and Nodepool will use a service account loaded from the in-cluster configuration path.

launch-retries

The number of times to retry launching a node before considering the job failed.

pools

A pool defines a group of resources from a Kubernetes provider.

name

Namespaces are prefixed with the pool's name.

node-attributes

A dictionary of key-value pairs that will be stored with the node data in ZooKeeper. The keys and values can be any arbitrary string.

labels

Each entry in a pool`s labels section indicates that the corresponding label is available for use in this pool.

Each entry is a dictionary with the following keys

name

Identifier for this label; references an entry in the labels section.

type

The Kubernetes provider supports two types of labels:

namespace

Namespace labels provide an empty namespace configured with a service account that can create pods, services, configmaps, etc.

pod

Pod labels provide a dedicated namespace with a single pod created using the providers.[kubernetes].pools.labels.image parameter and it is configured with a service account that can exec and get the logs of the pod.

image

Only used by the providers.[kubernetes].pools.labels.type.pod label type; specifies the image name used by the pod.

python-path

cpu

Only used by the providers.[kubernetes].pools.labels.type.pod label type; specifies the number of cpu to request for the pod.

memory

Only used by the providers.[kubernetes].pools.labels.type.pod label type; specifies the amount of memory in MB to request for the pod.

env

Only used by the providers.[kubernetes].pools.labels.type.pod label type; A list of environment variables to pass to the Pod.

name

The name of the environment variable passed to the Pod.

value

The value of the environment variable passed to the Pod.

Openshift Driver

Selecting the openshift driver adds the following options to the providers section of the configuration.

providers.[openshift]

An Openshift provider's resources are partitioned into groups called pool (see providers.[openshift].pools for details), and within a pool, the node types which are to be made available are listed (see providers.[openshift].labels for details).

Note

For documentation purposes the option names are prefixed providers.[openshift] to disambiguate from other drivers, but [openshift] is not required in the configuration (e.g. below providers.[openshift].pools refers to the pools key in the providers section when the openshift driver is selected).

Example:

providers:
  - name: cluster
    driver: openshift
    context: context-name
    pools:
      - name: main
        labels:
          - name: openshift-project
            type: project
          - name: openshift-pod
            type: pod
            image: docker.io/fedora:28

context

Name of the context configured in kube/config.

Before using the driver, Nodepool services need a kube/config file manually installed with self-provisioner (the service account needs to be able to create projects) context. Make sure the context is present in oc config get-contexts command output.

launch-retries

The number of times to retry launching a node before considering the job failed.

max-projects

Maximum number of projects that can be used.

pools

A pool defines a group of resources from an Openshift provider.

name

Project's name are prefixed with the pool's name.

node-attributes

A dictionary of key-value pairs that will be stored with the node data in ZooKeeper. The keys and values can be any arbitrary string.

labels

Each entry in a pool`s labels section indicates that the corresponding label is available for use in this pool.

Each entry is a dictionary with the following keys

name

Identifier for this label; references an entry in the labels section.

type

The Openshift provider supports two types of labels:

project

Project labels provide an empty project configured with a service account that can create pods, services, configmaps, etc.

pod

Pod labels provide a new dedicated project with a single pod created using the providers.[openshift].labels.image parameter and it is configured with a service account that can exec and get the logs of the pod.

image

Only used by the providers.[openshift].labels.type.pod label type; specifies the image name used by the pod.

image-pull

The ImagePullPolicy, can be IfNotPresent, Always or Never.

python-path

cpu

Only used by the providers.[openshift].labels.type.pod label type; specifies the number of cpu to request for the pod.

memory

Only used by the providers.[openshift].labels.type.pod label type; specifies the amount of memory in MB to request for the pod.

env

Only used by the providers.[openshift].labels.type.pod label type; A list of environment variables to pass to the Pod.

name

The name of the environment variable passed to the Pod.

value

The value of the environment variable passed to the Pod.

Openshift Pods Driver

Selecting the openshift pods driver adds the following options to the providers section of the configuration.

providers.[openshiftpods]

The Openshift Pods driver is similar to the Openshift driver, but it only support pod label to be created in a single project. This enable using an unprivileged service account that doesn't requires the self-provisioner role.

Example:

providers:
  - name: cluster
    driver: openshiftpods
    context: unprivileged-context-name
    pools:
      - name: main
        labels:
          - name: openshift-pod
            image: docker.io/fedora:28

context

Name of the context configured in kube/config.

launch-retries

The number of times to retry launching a pod before considering the job failed.

max-pods

Maximum number of pods that can be used.

pools

A pool defines a group of resources from an Openshift provider.

name

The project's name that will be used to create the pods.

node-attributes

A dictionary of key-value pairs that will be stored with the node data in ZooKeeper. The keys and values can be any arbitrary string.

labels

Each entry in a pool`s labels section indicates that the corresponding label is available for use in this pool.

Each entry is a dictionary with the following keys

name

Identifier for this label; references an entry in the labels section.

image

The image name.

image-pull

The ImagePullPolicy, can be IfNotPresent, Always or Never.

cpu

The number of cpu to request for the pod.

memory

The amount of memory in MB to request for the pod.

python-path

env

A list of environment variables to pass to the Pod.

name

The name of the environment variable passed to the Pod.

value

The value of the environment variable passed to the Pod.

AWS EC2 Driver

Selecting the aws driver adds the following options to the providers section of the configuration.

providers.[aws]

An AWS provider's resources are partitioned into groups called pool (see providers.[aws].pools for details), and within a pool, the node types which are to be made available are listed (see providers.[aws].pools.labels for details).

See Boto Configuration for information on how to configure credentials and other settings for AWS access in Nodepool's runtime environment.

Note

For documentation purposes the option names are prefixed providers.[aws] to disambiguate from other drivers, but [aws] is not required in the configuration (e.g. below providers.[aws].pools refers to the pools key in the providers section when the aws driver is selected).

Example:

providers:
  - name: ec2-us-west-2
    driver: aws
    region-name: us-west-2
    cloud-images:
      - name: debian9
        image-id: ami-09c308526d9534717
        username: admin
    pools:
      - name: main
        max-servers: 5
        subnet-id: subnet-0123456789abcdef0
        security-group-id: sg-01234567890abcdef
        labels:
          - name: debian9
            cloud-image: debian9
            instance-type: t3.medium
            iam-instance-profile:
              arn: arn:aws:iam::123456789012:instance-profile/s3-read-only
            key-name: zuul
            tags:
              key1: value1
          - name: debian9-large
            cloud-image: debian9
            instance-type: t3.large
            key-name: zuul
            tags:
              key1: value1
              key2: value2

name

A unique name for this provider configuration.

region-name

Name of the AWS region to interact with.

profile-name

The AWS credentials profile to load for this provider. If unspecified the boto3 library will select a profile.

See Boto Configuration for more information.

boot-timeout

Once an instance is active, how long to try connecting to the image via SSH. If the timeout is exceeded, the node launch is aborted and the instance deleted.

launch-retries

The number of times to retry launching a node before considering the job failed.

cloud-images

Each entry in this section must refer to an entry in the labels section.

cloud-images:
  - name: ubuntu1804
    image-id: ami-082fd9a18128c9e8c
    username: ubuntu
  - name: ubuntu1804-by-filters
    image-filters:
      - name: name
        values:
         - named-ami
    username: ubuntu
  - name: my-custom-win2k3
    connection-type: winrm
    username: admin

Each entry is a dictionary with the following keys

name

Identifier to refer this cloud-image from providers.[aws].pools.labels section. Since this name appears elsewhere in the nodepool configuration file, you may want to use your own descriptive name here and use image-id to specify the cloud image so that if the image id changes on the cloud, the impact to your Nodepool configuration will be minimal. However, if image-id is not provided, this is assumed to be the image id in the cloud.

image-id

If this is provided, it is used to select the image from the cloud provider by ID.

image-filters

If provided, this is used to select an AMI by filters. If the filters provided match more than one image, the most recent will be returned. image-filters are not valid if image-id is also specified.

Each entry is a dictionary with the following keys

name

The filter name. See Boto describe images for a list of valid filters.

values

A list of str values to filter on

username

The username that a consumer should use when connecting to the node.

python-path

connection-type

The connection type that a consumer should use when connecting to the node. For most images this is not necessary. However when creating Windows images this could be 'winrm' to enable access via ansible.

connection-port

The port that a consumer should use when connecting to the node. For most diskimages this is not necessary. This defaults to 22 for ssh and 5986 for winrm.

pools

A pool defines a group of resources from an AWS provider. Each pool has a maximum number of nodes which can be launched from it, along with a number of cloud-related attributes used when launching nodes.

name

A unique name within the provider for this pool of resources.

node-attributes

A dictionary of key-value pairs that will be stored with the node data in ZooKeeper. The keys and values can be any arbitrary string.

subnet-id

If provided, specifies the subnet to assign to the primary network interface of nodes.

security-group-id

If provided, specifies the security group ID to assign to the primary network interface of nodes.

public-ip-address

Specify if a public ip address shall be attached to nodes.

host-key-checking

labels

Each entry in a pool's labels section indicates that the corresponding label is available for use in this pool. When creating nodes for a label, the flavor-related attributes in that label's section will be used.

labels:
  - name: bionic
    instance-type: m5a.large

Each entry is a dictionary with the following keys

name

Identifier to refer this label. Nodepool will use this to set the name of the instance unless the name is specified as a tag.

cloud-image

Refers to the name of an externally managed image in the cloud that already exists on the provider. The value of cloud-image should match the name of a previously configured entry from the cloud-images section of the provider. See providers.[aws].cloud-images.

ebs-optimized

Indicates whether EBS optimization (additional, dedicated throughput between Amazon EC2 and Amazon EBS,) has been enabled for the instance.

instance-type

Name of the flavor to use.

iam-instance-profile

Used to attach an iam instance profile. Useful for giving access to services without needing any secrets.

name

Name of the instance profile. Mutually exclusive with providers.[aws].pools.labels.iam-instance-profile.arn

arn

ARN identifier of the profile. Mutually exclusive with providers.[aws].pools.labels.iam-instance-profile.name

key-name

The name of a keypair that will be used when booting each server.

volume-type

If given, the root EBS volume type

volume-size

If given, the size of the root EBS volume, in GiB.

userdata

A string of userdata for a node. Example usage is to install cloud-init package on image which will apply the userdata. Additional info about options in cloud-config: https://cloudinit.readthedocs.io/en/latest/topics/examples.html

tags

A dictionary of tags to add to the EC2 instances

Google Cloud Compute Engine (GCE) Driver

Selecting the gce driver adds the following options to the providers section of the configuration.

providers.[gce]

An GCE provider's resources are partitioned into groups called pool (see providers.[gce].pools for details), and within a pool, the node types which are to be made available are listed (see providers.[gce].pools.labels for details).

See Application Default Credentials for information on how to configure credentials and other settings for GCE access in Nodepool's runtime environment.

Note

For documentation purposes the option names are prefixed providers.[gce] to disambiguate from other drivers, but [gce] is not required in the configuration (e.g. below providers.[gce].pools refers to the pools key in the providers section when the gce driver is selected).

Example:

- name: gce-uscentral1
  driver: gce
  project: nodepool-123456
  region: us-central1
  zone: us-central1-a
  cloud-images:
    - name: debian-stretch
      image-project: debian-cloud
      image-family: debian-9
      username: zuul
      key: ssh-rsa ...
  pools:
    - name: main
      max-servers: 8
      labels:
        - name: debian-stretch
          instance-type: f1-micro
          cloud-image: debian-stretch
          volume-type: standard
          volume-size: 10

name

A unique name for this provider configuration.

region

Name of the region to use; see GCE regions and zones.

zone

Name of the zone to use; see GCE regions and zones.

boot-timeout

Once an instance is active, how long to try connecting to the image via SSH. If the timeout is exceeded, the node launch is aborted and the instance deleted.

launch-retries

The number of times to retry launching a node before considering the job failed.

cloud-images

Each entry in this section must refer to an entry in the labels section.

cloud-images:
  - name: debian-stretch
    image-project: debian-cloud
    image-family: debian-9
    username: zuul
    key: ssh-rsa ...

Each entry is a dictionary with the following keys:

name

Identifier to refer this cloud-image from providers.[gce].pools.labels section.

image-id

If this is provided, it is used to select the image from the cloud provider by ID.

image-project

If providers.[gce].cloud-images.image-id is not provided, this is used along with providers.[gce].cloud-images.image-family to find an image.

image-family

If providers.[gce].cloud-images.image-id is not provided, this is used along with providers.[gce].cloud-images.image-project to find an image.

username

The username that a consumer should use when connecting to the node.

key

An SSH public key to add to the instance (project global keys are added automatically).

python-path

connection-type

connection-port

The port that a consumer should use when connecting to the node. For most diskimages this is not necessary. This defaults to 22 for ssh and 5986 for winrm.

pools

A pool defines a group of resources from an GCE provider. Each pool has a maximum number of nodes which can be launched from it, along with a number of cloud-related attributes used when launching nodes.

name

A unique name within the provider for this pool of resources.

node-attributes

A dictionary of key-value pairs that will be stored with the node data in ZooKeeper. The keys and values can be any arbitrary string.

host-key-checking

use-internal-ip

Whether to access the instance with the internal or external IP address.

labels

labels:
  - name: debian
    instance-type: f1-micro
    cloud-image: debian-stretch

Each entry is a dictionary with the following keys

name

Identifier to refer this label.

cloud-image

Refers to the name of an externally managed image in the cloud that already exists on the provider. The value of cloud-image should match the name of a previously configured entry from the cloud-images section of the provider. See providers.[gce].cloud-images.

instance-type

Name of the flavor to use. See GCE machine types.

volume-type

If given, the root volume type (pd-standard or pd-ssd).

volume-size

If given, the size of the root volume, in GiB.

Azure Compute Driver

Selecting the azure driver adds the following options to the providers section of the configuration.

providers.[azure]

An Azure provider's resources are partitioned into groups called pool, and within a pool, the node types which are to be made available are listed

Note

For documentation purposes the option names are prefixed providers.[azure] to disambiguate from other drivers, but [azure] is not required in the configuration (e.g. below providers.[azure].pools refers to the pools key in the providers section when the azure driver is selected).

Example:

providers:
   - name: azure-central-us
     driver: azure
     zuul-public-key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAA...
     resource-group-location: centralus
     location: centralus
     resource-group: ZuulCIDev
     auth-path: /Users/grhayes/.azure/nodepoolCreds.json
     subnet-id: /subscriptions/<subscription-id>/resourceGroups/ZuulCI/providers/Microsoft.Network/virtualNetworks/NodePool/subnets/default
     cloud-images:
       - name: bionic
         username: zuul
         image-reference:
           sku: 18.04-LTS
           publisher: Canonical
           version: latest
           offer: UbuntuServer
     pools:
       - name: main
         max-servers: 10
         labels:
           - name: bionic
             cloud-image: bionic
             hardware-profile:
               vm-size: Standard_D1_v2
             tags:
               department: R&D
               purpose: CI/CD

name

A unique name for this provider configuration.

location

Name of the Azure region to interact with.

resource-group-location

Name of the Azure region to where the home Resource Group is or should be created.

auth-path

Path to the JSON file containing the service principal credentials. Create with the Azure CLI and the --sdk-auth flag

subnet-id

Subnet to create VMs on

cloud-images

Each entry in this section must refer to an entry in the labels section.

cloud-images:
  - name: bionic
    username: zuul
    image-reference:
      sku: 18.04-LTS
      publisher: Canonical
      version: latest
      offer: UbuntuServer
  - name: windows-server-2016
    username: zuul
    image-reference:
       sku: 2016-Datacenter
       publisher: MicrosoftWindowsServer
       version: latest
       offer: WindowsServer

Each entry is a dictionary with the following keys

name

Identifier to refer this cloud-image from labels section. Since this name appears elsewhere in the nodepool configuration file, you may want to use your own descriptive name here.

username

The username that a consumer should use when connecting to the node.

image-reference

sku

Image SKU

publisher

Image Publisher

offer

Image offers

version

Image version

pools

A pool defines a group of resources from an Azure provider. Each pool has a maximum number of nodes which can be launched from it, along with a number of cloud-related attributes used when launching nodes.

name

A unique name within the provider for this pool of resources.

labels

labels:
  - name: bionic
    cloud-image: bionic
    hardware-profile:
      vm-size: Standard_D1_v2

Each entry is a dictionary with the following keys

name

Identifier to refer this label.

cloud-image

Refers to the name of an externally managed image in the cloud that already exists on the provider. The value of cloud-image should match the name of a previously configured entry from the cloud-images section of the provider.

hardware-profile

vm-size

VM Size of the VMs to use in Azure. See the VM size list on azure.microsoft.com for the list of sizes availabile in each region.

75 KiB Raw Blame History

Configuration

Options

OpenStack Driver

Static Driver

Kubernetes Driver

Openshift Driver

Openshift Pods Driver

AWS EC2 Driver

Google Cloud Compute Engine (GCE) Driver

Azure Compute Driver

75 KiB

Raw Blame History