First ensure ssh connection is valid before scanning keys

We have a network appliance we test via nested virt. While the outer
node is live and the port we nodescan is open, the nested node is still
booting up SSHd.  Which causes nodescan to return:

  paramiko.ssh_exception.SSHException: Error reading SSH protocol banner

until SSHd is properly running.

Perviously we set out boot-timeout to 5 mins, to allow for the nested
SSHd to come online properly. This should restore that functionality.

Change-Id: I7f43530ee77a81f7c969d548190a71bfb9b03455
Signed-off-by: Paul Belanger <pabelanger@redhat.com>

nodepool/nodeutils.py

@@ -78,10 +78,17 @@ def nodescan(ip, port=22, timeout=60, gather_hostkeys=True):
             timeout, exceptions.ConnectionTimeoutException,
             "connection to %s on port %s" % (ip, port)):
         sock = None
+        t = None
         try:
             sock = socket.socket(family, socket.SOCK_STREAM)
             sock.settimeout(10)
             sock.connect(sockaddr)
+            # NOTE(pabelanger): Try to connect to SSH first, before breaking
+            # our loop. This is to ensure the SSHd on the remote node is
+            # properly running before we scan keys below.
+            if gather_hostkeys:
+                t = paramiko.transport.Transport(sock)
+                t.start_client(timeout=timeout)
             break
         except socket.error as e:
             if e.errno not in [errno.ECONNREFUSED, errno.EHOSTUNREACH, None]:
@@ -90,6 +97,11 @@ def nodescan(ip, port=22, timeout=60, gather_hostkeys=True):
         except Exception:
             log.exception("ssh socket connection failure")
         finally:
+            try:
+                if t:
+                    t.close()
+            except Exception as e:
+                log.exception('Exception closing paramiko: %s', e)
             try:
                 if sock:
                     sock.close()