Set default python-path to "auto"

The "python-path" configuration option makes its way through to Zuul
where it sets the "ansible_interpreter_path" in the inventory.
Currently this defaults to "/usr/bin/python2" which is wrong for
Python 3-only distributions.

Ansible >=2.8 provides for automated discovery of the interpreter to
avoid runtime errors choosing an invalid interpreter [1].  Using this
should mean that "python-path" doesn't need to be explicitly for any
common case.  As more distributions become Python 3 only, this should
"do the right thing" without further configuration.

This switches the default python-path to "auto".  The dependent change
updates Zuul to accept this and use it when running with Ansible
>=2.8, or default back to "/usr/bin/python2" for earlier Ansible
versions.

Testing and documentation is updated, and a release note added.

[1] https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html

Depends-On: https://review.opendev.org/682275
Change-Id: I02a1a618c8806b150049e91b644ec3c0cb826ba4

doc/source/configuration.rst

@@ -311,9 +311,13 @@ Options
 
    .. attr:: python-path
       :type: string
-      :default: /usr/bin/python2
+      :default: auto
 
-      The path of the default python interpreter.
+      The path of the default python interpreter.  Used by Zuul to set
+      ``ansible_python_interpreter``.  The special value ``auto`` will
+      direct Zuul to use inbuilt Ansible logic to select the
+      interpreter on Ansible >=2.8, and default to
+      ``/usr/bin/python2`` for earlier versions.
 
    .. attr:: dib-cmd
       :type: string
@@ -711,9 +715,13 @@ Selecting the OpenStack driver adds the following options to the
 
      .. attr:: python-path
         :type: str
-        :default: /usr/bin/python2
+        :default: auto
 
-        The path of the default python interpreter.
+        The path of the default python interpreter.  Used by Zuul to set
+        ``ansible_python_interpreter``.  The special value ``auto`` will
+        direct Zuul to use inbuilt Ansible logic to select the
+        interpreter on Ansible >=2.8, and default to
+        ``/usr/bin/python2`` for earlier versions.
 
      .. attr:: connection-type
         :type: str
@@ -1134,7 +1142,11 @@ Selecting the static driver adds the following options to the
             :type: str
             :default: /usr/bin/python2
 
-            The path of the default python interpreter.
+            The path of the default python interpreter.  Used by Zuul to set
+            ``ansible_python_interpreter``.  The special value ``auto`` will
+            direct Zuul to use inbuilt Ansible logic to select the
+            interpreter on Ansible >=2.8, and default to
+            ``/usr/bin/python2`` for earlier versions.
 
          .. attr:: max-parallel-jobs
             :type: int
@@ -1257,9 +1269,14 @@ Selecting the kubernetes driver adds the following options to the
 
          .. attr:: python-path
             :type: str
-            :default: /usr/bin/python2
+            :default: auto
+
+           The path of the default python interpreter.  Used by Zuul to set
+           ``ansible_python_interpreter``.  The special value ``auto`` will
+           direct Zuul to use inbuilt Ansible logic to select the
+           interpreter on Ansible >=2.8, and default to
+           ``/usr/bin/python2`` for earlier versions.
 
-            The path of the default python interpreter.
 
 
 Openshift Driver
@@ -1384,9 +1401,13 @@ Selecting the openshift driver adds the following options to the
 
       .. attr:: python-path
          :type: str
-         :default: /usr/bin/python2
+         :default: auto
 
-         The path of the default python interpreter.
+          The path of the default python interpreter.  Used by Zuul to set
+          ``ansible_python_interpreter``.  The special value ``auto`` will
+          direct Zuul to use inbuilt Ansible logic to select the
+          interpreter on Ansible >=2.8, and default to
+          ``/usr/bin/python2`` for earlier versions.
 
       .. attr:: cpu
          :type: int
@@ -1629,9 +1650,13 @@ section of the configuration.
 
       .. attr:: python-path
          :type: str
-         :default: /usr/bin/python2
+         :default: auto
 
-         The path of the default python interpreter.
+         The path of the default python interpreter.  Used by Zuul to set
+         ``ansible_python_interpreter``.  The special value ``auto`` will
+         direct Zuul to use inbuilt Ansible logic to select the
+         interpreter on Ansible >=2.8, and default to
+         ``/usr/bin/python2`` for earlier versions.
 
       .. attr:: connection-type
          :type: str

nodepool/config.py

@@ -119,7 +119,7 @@ class Config(ConfigValue):
             d.image_types = set(diskimage.get('formats', []))
             d.pause = bool(diskimage.get('pause', False))
             d.username = diskimage.get('username', 'zuul')
-            d.python_path = diskimage.get('python-path', '/usr/bin/python2')
+            d.python_path = diskimage.get('python-path', 'auto')
             d.build_timeout = diskimage.get('build-timeout', (8 * 60 * 60))
             self.diskimages[d.name] = d
 

nodepool/driver/aws/config.py

@@ -192,7 +192,7 @@ class AwsProviderConfig(ProviderConfig):
             i.name = image['name']
             i.image_id = image.get('image-id', None)
             i.username = image.get('username', None)
-            i.python_path = image.get('python-path', '/usr/bin/python2')
+            i.python_path = image.get('python-path', 'auto')
             i.connection_type = image.get('connection-type', 'ssh')
             i.connection_port = image.get(
                 'connection-port',

nodepool/driver/kubernetes/config.py

@@ -56,7 +56,7 @@ class KubernetesPool(ConfigPool):
             pl.type = label['type']
             pl.image = label.get('image')
             pl.image_pull = label.get('image-pull', 'IfNotPresent')
-            pl.python_path = label.get('python-path', '/usr/bin/python2')
+            pl.python_path = label.get('python-path', 'auto')
             pl.pool = self
             self.labels[pl.name] = pl
             full_config.labels[label['name']].pools.append(self)

nodepool/driver/openshift/config.py

@@ -61,7 +61,7 @@ class OpenshiftPool(ConfigPool):
             pl.image_pull = label.get('image-pull', 'IfNotPresent')
             pl.cpu = label.get('cpu')
             pl.memory = label.get('memory')
-            pl.python_path = label.get('python-path', '/usr/bin/python2')
+            pl.python_path = label.get('python-path', 'auto')
             pl.pool = self
             self.labels[pl.name] = pl
             full_config.labels[label['name']].pools.append(self)

nodepool/driver/openstack/config.py

@@ -325,7 +325,7 @@ class OpenStackProviderConfig(ProviderConfig):
             i.image_id = image.get('image-id', None)
             i.image_name = image.get('image-name', None)
             i.username = image.get('username', None)
-            i.python_path = image.get('python-path', '/usr/bin/python2')
+            i.python_path = image.get('python-path', 'auto')
             i.connection_type = image.get('connection-type', 'ssh')
             i.connection_port = image.get(
                 'connection-port',

nodepool/driver/static/config.py

@@ -59,7 +59,7 @@ class StaticPool(ConfigPool):
                 'connection-type': node.get('connection-type', 'ssh'),
                 'username': node.get('username', 'zuul'),
                 'max-parallel-jobs': int(node.get('max-parallel-jobs', 1)),
-                'python-path': node.get('python-path', '/usr/bin/python2'),
+                'python-path': node.get('python-path', 'auto'),
             })
             if isinstance(node['labels'], str):
                 for label in node['labels'].split():

nodepool/tests/unit/test_launcher.py

@@ -147,7 +147,7 @@ class TestLauncher(tests.DBTestCase):
         self.assertEqual(nodes[1].type, ['fake-label1'])
         self.assertEqual(nodes[2].type, ['fake-label4'])
         self.assertEqual(nodes[3].type, ['fake-label2'])
-        self.assertEqual(nodes[0].python_path, '/usr/bin/python2')
+        self.assertEqual(nodes[0].python_path, 'auto')
 
     def _test_node_assignment_at_quota(self,
                                        config,
@@ -1331,7 +1331,7 @@ class TestLauncher(tests.DBTestCase):
         self.assertEqual('winrm', nodes[0].connection_type)
         self.assertEqual(5986, nodes[0].connection_port)
         self.assertEqual(nodes[0].host_keys, [])
-        self.assertEqual(nodes[0].python_path, '/usr/bin/python2')
+        self.assertEqual(nodes[0].python_path, 'auto')
 
         nodes = self.waitForNodes('fake-label-arbitrary-port')
         self.assertEqual(len(nodes), 1)

releasenotes/notes/python-path-auto-83881e94d835ff83.yaml

@@ -0,0 +1,23 @@
+---
+features:
+  - |
+    The default ``python-path`` is now ``auto`` (instead of
+    ``/usr/bin/python2``).  With this, Zuul 3.11.1 and greater will set the
+    ``ansible_python_interpreter`` to ``auto`` when using Ansible
+    >=2.8 to use `automated interpreter discovery
+    <https://docs.ansible.com/ansible/latest/reference_appendices/interpreter_discovery.html>`__.
+    When using earlier Ansible, it will remain the old default of
+    ``/usr/bin/python2``.
+
+    This will remove the need to override python-path explicitly for
+    Python 3-only distributions, which should be detected correctly
+    automatically.
+
+upgrade:
+  - |
+    This release should only be run against Zuul 3.11.1 or greater.
+    Earlier Zuul releases will not convert the new default
+    ``python-path`` of ``auto`` to ``/usr/bin/python2`` for Ansible
+    <2.8, leading to a configuration error.  It may be possible to use
+    earlier Zuul releases if you you are only using Ansible >= 2.8, or
+    explicitly set ``python-path`` for every image.