tatu/TRY_IT.rst

Notes on using Tatu for the first time

In this example, I'm the "demo" user and I need to connect to VMs in projects named "demo" and "invisible_to_admin".

Generate SSH keys and certificates

Since you'll need separate SSH user certificates for each of your projects, generate separate ssh keys for each of your projects:

ssh-keygen -f ~/.ssh/demo_key
ssh-keygen -f ~/.ssh/inv_key

Now generate the certificate for each of your projects (this can also be done in Horizon). First set your environment variables to select your user and project. Note that ssh client expects the certificate's name to be the private key name followed by "-cert.pub":

source /opt/stack/devstack/openrc demo demo
openstack ssh usercert create -f value -c Certificate "`cat ~/.ssh/demo_key.pub`" > ~/.ssh/demo_key-cert.pub
openstack ssh usercert create --os-project-name invisible_to_admin -f value -c Certificate "`cat ~/.ssh/inv_key.pub`" > ~/.ssh/inv_key-cert.pub

You can examine a certificate as follows:

ssh-keygen -Lf ~/.ssh/inv_key-cert.pub

And the output will look like this:

/root/.ssh/inv_key-cert.pub:
    Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:4h+zwW8L+E1OLyOz4uHh4ffcqJFS/p5rETlf15Q04x8
    Signing CA: RSA SHA256:s8FpsDHkhly3ePtKDihO/x7UVj3sw3fSILLPLQJz2n0
    Key ID: "demo_5"
    Serial: 5
    Valid: from 2018-03-09T13:05:23 to 2019-03-10T13:05:23
    Principals:
            Member
    Critical Options: (none)
    Extensions:
            permit-X11-forwarding
            permit-agent-forwarding
            permit-port-forwarding
            permit-pty
            permit-user-rc

Note that the Signing CA is different for each certificate. You'll have to use the corresponding key/certificate to ssh to a project's VM.

Configure your client to trust the Certificate Authority for hosts

Now configure your ssh client to trust SSH host certificats signed by the Host CAs of your projects. Given how Tatu currently generates Host certificates, you must trust each project's Host CA for hostnames in any domain (hence the "*" in the command):

demo_id=`openstack project show demo -f value -c id`
echo '@cert-authority * '`openstack ssh ca show $demo_id -f value -c 'Host Public Key'` >> ~/.ssh/known_hosts
inv_id=`openstack project show invisible_to_admin --os-project-name invisible_to_admin -f value -c id`
echo '@cert-authority * '`openstack ssh ca show $inv_id -f value -c 'Host Public Key'` >> ~/.ssh/known_hosts

Above, note that the --os-project-name option is necessary because we sourced openrc with the "demo" project.

Your known_hosts file should now have one @cert-authority line for each project:

cat ~/.ssh/known_hosts
    @cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD...
    @cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKVCfrfD...

Launch your VM

Now launch a VM without a Key Pair. Unless you're using Dragonflow and Tatu's experimental PAT bastion feature, assign a floating IP to the VM. In this example we'll assume the VM's Floating IP is 172.24.4.8

Make sure to launch your VM on a private network that has a router attached, and that the router has a gateway. In other words, unless you're using a bastion, the VM's IP must be routable via the Floating IP.

If you look in Tatu API's log, you should see something like the following. There may be several calls to /v1/novavendordata because Nova queries Tatu once for each version of Nova metadata API. It's OK, Tatu returns the same data each time. The call to /noauth/hostcerts is the VM's request to Tatu to generate an SSH host certificate. The call does not use Keystone authentication but is protected by the one-time-token presented in "token_id":

journalctl --unit devstack@tatu-api.service
  Request POST /v1/novavendordata with body {... u'hostname': u'fluffy', u'boot-roles': u'admin,Member,anotherrole', u'image-id': ... u'project-id': ... u'instance-id': ...}
    produced response with status 201 Created location /hosttokens/489b555621f74494adf7089174563bfb
    and body {"api_endpoint": "http://172.24.4.1:18322", "auth_pub_key_user": ... "token": "489b555621f74494adf7089174563bfb", "root_principals": "",
              "ssh_port": 2222, "sudoers": "admin", "pam_sudo": true, "users": "admin,Member,anotherrole"}
  ...
  Request POST /noauth/hostcerts with body {u'token_id': '489b555621f74494adf7089174563bfb', u'pub_key': ... u'host_id': ...}
    produced response with status 200 OK location /hostcerts/717f9144e20e408380e174bda5855b3b/MD5:da:08:6f:d9:cc:b9:57:66:cb:b7:50:7f:d1:26:71:26
    and body {"created_at": "2018-03-14T18:27:58.000000", "hostname": "fluffy", "expires_at": "2019-03-14T18:27:58.000000", "cert": ...

SSH to your VM

If you launched your VM in the demo project, use the following ssh command. Note that the Linux user account must correspond to one of the principals in your certificate, which in turn corresponds to one of your roles in the project:

ssh -i ~/.ssh/demo_key Member@172.24.4.8

Thanks to the host's SSH certificate and the @cert-authority line in the client's known_hosts file, a man-in-the-middle (MITM) attack risk is eliminated, so you should no longer see this warning that you always ignore:

The authenticity of host '172.24.4.8 (172.24.4.8)' can't be established.
RSA key fingerprint is SHA256:FS2QGF4Ant/MHoUPxgO6N99uQss57lKkPclXDgFOLAU.
Are you sure you want to continue connecting (yes/no)?

Re-run the command with verbose output:

ssh -v -i ~/.ssh/demo_key Member@172.24.4.8

You should see the SSH host presenting its host certificate:

debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:FS2QGF4Ant/MHoUPxgO6N99uQss57lKkPclXDgFOLAU, serial 0 ID "otto_0" CA ssh-rsa SHA256:b0BD63oM4ks4BT2Cxlzz9WaV0HE+AqwEG7mnk3vJtz4 valid from 2018-03-09T04:32:35 to 2019-03-10T04:32:35
debug1: Host '172.24.4.8' is known and matches the RSA-CERT host certificate.
debug1: Found CA key in /root/.ssh/known_hosts:1

You should also see your SSH client presenting your user certificate. Note that your client first offers the public key, which is rejected, and then offers the certificate, which is accepted:

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/inv_key
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering RSA-CERT public key: /root/.ssh/inv_key-cert
debug1: Server accepts key: pkalg ssh-rsa-cert-v01@openssh.com blen 1088

Sudo privileges

Tatu's convention is that every Keystone role associated with a project should have a corresponding Linux user account on the VM. In addition, any role with 'admin' in its name should have a user account with sudo privileges.

When tatu's configuration key "pam_sudo" is set to False, then sudo calls are allowed without authentication. Tatu creates a file 130-admin in /etc/sudoers.d and containing this configuration:

admin ALL= NOPASSWD: ALL

In order to test this, go to Horizon, Identity->Projects and click on "Manage Members" for one of your projects. Now give one of your Keystone users the role 'admin'. You will have to generate a new SSH user certificate. You may also need to launch a new VM if none of the project's users previously had an admin role (because Tatu does not currently keep the VM's accounts up to date with the project's Keystone roles).

Using Uber's pam-ussh module

Uber's pam-ussh module handles authentication of sudo calls. You can enable it by setting "pam_sudo = True" in the "tatu" stanza in /etc/tatu/tatu.conf.

Tatu's conventions about sudo privileges (explained in the previous section) still apply, so go ahead and give one of your project's users a role with 'admin' in its name. Then launch a VM.

Uber's pam-ussh authenticates sudo calls by querying the client's SSH agent. You'll need to run ssh-agent, set some environment variables and ssh-add your private key:

ssh-agent
    SSH_AUTH_SOCK=/tmp/ssh-IrDH7qOuujNe/agent.17084; export SSH_AUTH_SOCK;
    SSH_AGENT_PID=17085; export SSH_AGENT_PID;
    echo Agent pid 17085;
# Set the environment variables by copy/pasting the previous command's output
SSH_AUTH_SOCK=/tmp/ssh-IrDH7qOuujNe/agent.17084; export SSH_AUTH_SOCK;
# Add your private key to the agent
ssh-add ~/.ssh/demo_key
ssh-add -l
    2048 SHA256:obvWOMbOuQyaqpvUI9+YxZiNCItlAL3JsQsZEEEx/6k /root/.ssh/demo_key (RSA)
    2048 SHA256:obvWOMbOuQyaqpvUI9+YxZiNCItlAL3JsQsZEEEx/6k /root/.ssh/demo_key (RSA-CERT)

When you launch ssh, remember to enable agent forwarding with the -A option, otherwise pam-ussh won't be able to query your agent. We won't need the -i option now because the agent will take care of trying the appropriate keys and certificates in its negotiation with the server. But let's use the -v option so we can see when pam-ussh does its authentication:

ssh -v -A admin@172.24.4.8
    ...
    debug1: Requesting authentication agent forwarding.
    ...
    Last login: Tue Mar 13 04:33:05 2018 from 172.24.4.1
[admin@dusty ~]$ sudo echo hello
    debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
    debug1: channel 1: new [authentication agent connection]
    debug1: confirm auth-agent@openssh.com
  hello
    debug1: channel 1: FORCE input drain
    debug1: channel 1: free: authentication agent connection, nchannels 2
[admin@dusty ~]$ sudo echo how are you
  how are you
[admin@dusty ~]$

What just happened? Afer login, the first time we ran sudo, there was another exchange between ssh server and client. Pam-ussh uses the agent AUTH_SOCK on the server to query the ssh-client for its certificates. The ssh-client gets them from the ssh-agent. Pam-ussh tries to find a valid ssh certificate (that has NOT been revoked - it should not be in the revoked-keys file on the server). Failing that, pam-ussh will give up and pass the torch to another pam module that does password-based authentication.

How did Tatu configure this on the VM? For each user that should be granted sudo privileges, Tatu created a file named like 130-admin in /etc/sudoers.d. Its contents look like this:

admin ALL= ALL
Defaults:admin timestamp_timeout=1

A few things to note:

• Compared to when pam_sudo is false, the "NOPASSWD:" option has been dropped;
• sudo is set to re-authenticate every 1 minute (thanks to timestamp_timeout) and that's why the second sudo call above didn't re-authenticate (unless you waited 60 seconds).

Finally, take a look at the PAM configuration:

[admin@dusty ~]$ cat /etc/pam.d/sudo
  #%PAM-1.0
  auth sufficient /lib64/security/pam_ussh.so ca_file=/etc/ssh/ca_user.pub authorized_principals=admin revoked_keys_file=/etc/ssh/revoked-keys
  auth       include      system-auth
  account    include      system-auth
  password   include      system-auth
  session    optional     pam_keyinit.so revoke
  session    required     pam_limits.so
  session    include      system-auth

Note that pam_ussh validation alone is sufficient to achieve validation. It's important that pam_ussh is placed before system-auth. If it were after, pam modules in system-auth would be called first and the user would have to fail to enter their password a few times before certificate-based authentication was attempted by pam_ussh.

Pam-ush's parameters specifies that only 'admin' account can authenticate with SSH certificates (others will have to use default mechanism, i.e. passwords, which Tatu does not provide); also, pam-ussh will check the revoked-keys file that Tatu's VM scripts are keeping up-to-date; and finally, certificate signatures are checked against the User CA public key stored in ca_user.pub