From 5790634ff48dd00444c051e20b7fe24cc82fb350 Mon Sep 17 00:00:00 2001 From: Martin Kopec Date: Fri, 25 Jun 2021 10:07:45 +0200 Subject: [PATCH] Update documentation, add playbook for developers The patch updates the user-faced documentation and adds a playbook together with a doc meant mainly for RefStack developers which helps with running the server instance locally. Change-Id: I268fdf717c8f0e2cd0c130c560306bf54e48f348 --- doc/source/index.rst | 1 - doc/source/run_in_docker.rst | 106 ++++++++++++++++++++++- doc/source/uploading_private_results.rst | 72 +++++++-------- doc/source/vendor_product.rst | 2 +- playbooks/README.rst | 12 +++ playbooks/run-refstack-in-container.yml | 52 +++++++++++ 6 files changed, 207 insertions(+), 38 deletions(-) create mode 100644 playbooks/README.rst create mode 100644 playbooks/run-refstack-in-container.yml diff --git a/doc/source/index.rst b/doc/source/index.rst index 41d1b742..32ce225d 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -42,4 +42,3 @@ Indices and tables ------------------ * :ref:`search` - diff --git a/doc/source/run_in_docker.rst b/doc/source/run_in_docker.rst index ddee704b..b57ac748 100644 --- a/doc/source/run_in_docker.rst +++ b/doc/source/run_in_docker.rst @@ -1,6 +1,111 @@ Run-in-docker manual ==================== +Option 1 - Using an ansible playbook +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +These steps are meant for RefStack developers to help them with setting up +a local refstack instance. + +In production RefStack server is managed by a set of playbooks and ansible roles +defined in `system-config `__ +repository. This option takes advantage of those. + +The RefStack server is running on Ubuntu 20.04 LTS in the production. + +You can find an ansible playbook in ``playbooks`` directory which semi-automates +the process of running refstack server in a container. + +Execute the playbook by:: + + $ ansible-playbook playbooks/run-refstack-in-container.yaml + +In order to avoid setting certificates and https protocol (it's simpler and more +than enough for a testing instance), edit +``/etc/apache2/sites-enabled/000-default.conf`` like following: + +* remove VirtualHost section for the port 80 and change the port of VirtualHost from 443 to 80 +* Turn off the SSLEngine (`SSLEngine on -> SSLEngine off`) +* Remove SSLCertificate lines + +and then restart the apache service so that it loads the new configuration:: + + $ systemctl restart apache2 + + +How to edit refstack files within the container +----------------------------------------------- + +List the running container by:: + + $ docker container list + +You can enter the container by:: + + $ sudo docker exec -it /bin/bash + +If you wanna install new packages like f.e. vim, do the following:: + $ apt update + $ apt install vim + +Edit what's needed, backend is installed under +``/usr/local/lib/python3.7/site-packages/refstack/`` and frontend source files +can be found at ``/refstack-ui`` + +After you made the changes, make pecan to reload the files served:: + + $ apt install procps # to install pkill command + $ pkill pecan + +Killing pecan will kick you out of the container, however, pecan serves the +edited files now and you may re-enter the container. + +Installing refstack with changes put for a review +------------------------------------------------- + +In order to do this, you will need to rebuild the refstack image built by the +playbook. + +Go to the location where the playbook downloaded system-config, default in +``/tmp/refstack-docker`` and edit the refstack's Dockerfile:: + + $ cd /tmp/refstack-docker + $ vim ./refstack-docker-files/Dockerfile + +Replace:: + + $ RUN git clone https://opendev.org/osf/refstack /tmp/src + +by:: + + $ RUN git clone https://opendev.org/osf/refstack.git /tmp/src \ + && cd /tmp/src && git fetch "https://review.opendev.org/osf/refstack" \ + refs/changes/37/ && git checkout -b \ + change-- FETCH_HEAD + +Then rebuild the image:: + + $ docker image build -f Dockerfile -t . + +Edit the ``docker-compose.yaml`` stored (by default) in +``/etc/refstack-docker/docker-compose.yaml`` and change the the image +(under `refstack-api`) to your image name and tag you set in the previous step. + +After then spin a new container using the new image:: + + $ cd /etc/refstack-docker + $ docker-compose down # if refstack container is already running + $ docker-compose up -d + +To see the server's logs use the following command:: + + $ docker container logs -f + + +Option 2 - Using a script +^^^^^^^^^^^^^^^^^^^^^^^^^ + +NOTE: This is currently outdated, follow the Option 1 for now. + The main purpose of the ``run-in-docker`` script is to provide a convenient way to create a local setup of RefStack inside a Docker container. It should be helpful for new developers and also for testing @@ -70,4 +175,3 @@ Useful in-container commands/aliases: files on the host - ``activate`` - activate the python virtual env - ``mysql`` - open the MySQL console - diff --git a/doc/source/uploading_private_results.rst b/doc/source/uploading_private_results.rst index d53f31a5..5ad6fa59 100644 --- a/doc/source/uploading_private_results.rst +++ b/doc/source/uploading_private_results.rst @@ -5,7 +5,7 @@ RefStack allows test results contributors to submit test results and have them displayed either anonymously, or identified with a vendor. As such, test results should be uploaded with validated users. Users will first log into RefStack with their OpenStack ID to upload their public -keys. RefStack test results can then be uploaded to RefStack using the +keys. RefStack test results need to be uploaded to RefStack using the corresponding private key. By default, the uploaded data isn't shared, but authorized users can decide to share the results with the community anonymously. @@ -19,10 +19,11 @@ Register an OpenStack ID The RefStack server uses OpenStack OpenID for user authentication. Therefore, the RefStack server requires that anyone who wants to upload test data to have an OpenStack ID. As you click on the Sign In/Sign Up -link on the RefStack pages, you will be redirected to the official -OpenStack user log in page where you can either log in with your -OpenStack ID or register for one. The registration page can also be -found directly through: https://www.openstack.org/join/register. +link on the `RefStack pages `__, you +will be redirected to the official OpenStack user log in page where you +can either log in with your OpenStack ID or register for one. +The registration page can also be found directly through: +https://www.openstack.org/join/register. Generate ssh keys locally ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -30,18 +31,18 @@ Generate ssh keys locally You will need to generate ssh keys locally. If your operating system is a Linux distro, then you can use the following instructions. -First check for existing keys with command: +First check for existing keys with command:: -- ``ls -al ~/.ssh`` + $ ls -al ~/.ssh If you see you already have existing public and private keys that you -want to use, you can skip this step; otherwise: +want to use, you can skip this step; otherwise:: -- ``ssh-keygen -m PEM -t rsa -b 4096 -C "youropenstackid"`` + $ ssh-keygen -m PEM -t rsa -b 4096 -C "youropenstackid" -The 'youropenstackid' string is the username you chose when you +The `youropenstackid` string is the username you chose when you registered for your OpenStack ID account. Enter the file name in which -to save the key (/home/you/.ssh/id\_rsa), then press enter. You will be +to save the key (``/home/you/.ssh/id\_rsa``), then press enter. You will be asked to enter a passphrase. Just press enter again as passphrase protected keys currently aren't supported. Your ssh keys will then be generated. @@ -49,15 +50,17 @@ generated. Sign Key with RefStack Client ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -\*\* IMPORTANT \*\* You must have the RefStack client on you computer to +**IMPORTANT** You must have the RefStack client on you computer to complete this step. +Follow `its README `__ on how to +install it. Generate a signature for your public key using your private key with -`refstack-client `__ +`refstack-client `__:: -- ``./refstack-client sign /path-of-sshkey-folder/key-file-name`` + $ ./refstack-client sign /path-of-sshkey-folder/key-file-name -The '/path-of-sshkey-folder' string is the path of the folder where the +The ``/path-of-sshkey-folder`` string is the path of the folder where the generated ssh keys are stored locally. The 'key-file-name' portion refers to the private key file name. If the command runs correctly, there will be output like below: @@ -74,37 +77,37 @@ Upload the ssh public key and the signature ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Sign into https://refstack.openstack.org with your OpenStack ID. Click -the "Profile" link in the upper right corner. Now click the "Import -public key" button on your profile page. A popup window with two entry +the `Profile` link in the upper right corner. Now click the `Import +public key` button on your profile page. A popup window with two entry fields will appear. Just copy and paste the key and signature generated in the previous step into the corresponding textboxes. -Note that the literal strings 'Public key:' and 'Self signature:' from -the refstack-client "sign" command output should not be copied/pasted -into the text boxes. Otherwise you will get an error like: +Note that the literal strings `Public key:` and `Self signature:` from +the ``refstack-client sign`` command output **should not** be copied/pasted +into the text boxes. Otherwise you will get an error like:: -- ``Bad Request Request doesnt correspond to schema`` + Bad Request Request doesn't correspond to schema -Once complete, click the 'Import public key' button. +Once complete, click the `Import public key` button. Upload the test result with refstack-client ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- ``./refstack-client upload /path_to_testresult_json_file --url https://refstack.openstack.org/api -i ~/.ssh/id_rsa`` +The results can be uploaded using the ``refstack-client`` by:: -\*\* NOTE \*\* Users may need to add the '--insecure' optional argument + $ ./refstack-client upload /path_to_testresult_json_file \ + --url https://refstack.openstack.org/api -i ~/.ssh/id_rsa + +**NOTE** Users may need to add the `--insecure` optional argument to the command string if certificate validation issues occur when -uploading test result. To use with insecure: +uploading test result. Usage with the insecure argument:: -- ``./refstack-client upload --insecure /path_to_testresult_json_file --url https://refstack.openstack.org/api -i ~/.ssh/id_rsa`` + $ ./refstack-client upload --insecure /path_to_testresult_json_file \ + --url https://refstack.openstack.org/api -i ~/.ssh/id_rsa`` -- The ``path_to_testresult_json_file`` here is the json file of your test result. - -- By default, it's in ``.tempest/.stestr/.json`` where refstack-client runs from. - -- Here '' is a serial number that matches its corresponding subunit file name. - -- If the command runs correctly, there will be output like below: +The ``path_to_testresult_json_file`` there is the json file of the test result. +By default, it's in ``.tempest/.stestr/.json``. If the command +runs correctly, there will be output like below: :: @@ -112,6 +115,5 @@ uploading test result. To use with insecure: Test results uploaded! URL: https://refstack.openstack.org/#/results/88a1e6f4-707d-4627-b658-b14b7e6ba70d. -You can find your uploaded test results by clicking the 'My Results' +You can find your uploaded test results by clicking the `My Results` link on the RefStack website. - diff --git a/doc/source/vendor_product.rst b/doc/source/vendor_product.rst index a3541c4e..2ba919b3 100755 --- a/doc/source/vendor_product.rst +++ b/doc/source/vendor_product.rst @@ -88,4 +88,4 @@ Product version A default version is created whenever a product is created. The name of the default version is blank. The default version is used for products that have -no version. Users can add new product versions to the product as needed. +no version. Users can add new product versions to the product as needed. diff --git a/playbooks/README.rst b/playbooks/README.rst new file mode 100644 index 00000000..1eba4b72 --- /dev/null +++ b/playbooks/README.rst @@ -0,0 +1,12 @@ +Playbook for running refstack locally +###################################### + +The playbook is meant for developers to help them with debugging and +reviewing new changes in the refstack project. + +The playbook semi-automates running the refstack server on the localhost. +It downloads refstack role and templates from +`system-config `__ repository +which is used for deploying and maintaining upstream servers, one of which is +refstack. Then it builds the refstack image and spins a container using the +refstack role. diff --git a/playbooks/run-refstack-in-container.yml b/playbooks/run-refstack-in-container.yml new file mode 100644 index 00000000..c6d26580 --- /dev/null +++ b/playbooks/run-refstack-in-container.yml @@ -0,0 +1,52 @@ +--- +- hosts: localhost + become: true + gather_facts: true + vars: + # dir where refstack files for buidling an image, running a container + # will be stored + refstack_dir: /tmp/refstack-docker + refstack_openid_endpoint: '' + # ip address of the machine you're running this on + refstack_url: # 'http://' + # the default credentials for the refstack database + refstack_db_username: refstack + refstack_db_password: Jz4ooq9TL7nc3hX3 + refstack_root_db_password: KbgY3r9HYnEYpgRP + tasks: + - name: Clone system-config repository + git: + repo: https://opendev.org/opendev/system-config.git + dest: "{{ refstack_dir }}/system-config" + + - name: Extract docker files + copy: + src: "{{ refstack_dir }}/system-config/docker/refstack/" + dest: "{{ refstack_dir }}/refstack-docker-files" + remote_src: yes + + - name: Extract refstack role + copy: + src: "{{ refstack_dir }}/system-config/playbooks/roles/refstack/" + dest: "{{ refstack_dir }}/refstack-role" + remote_src: yes + + - name: Delete the rest of system-config content + file: + state: absent + path: "{{ refstack_dir }}/system-config" + + - name: Install Docker + apt: + name: + - docker + - docker-compose + state: present + + - name: Build refstack image + command: docker image build -f Dockerfile -t refstack:1 . + args: + chdir: "{{ refstack_dir }}/refstack-docker-files" + + - include_role: + name: "{{ refstack_dir }}/refstack-role"