[RFE] Extend devmode.sh to deploy Openstack environment on OVB host clouds

[Ronelle Landy]

Background/Motivation:
----------------------

Hardware resources are finite and invariably, in demand. With the exception of functionality like hardware provisioning, the vast majority of features, updates, and bug fixes could be developed and tested on Openstack deployments running in OVB host clouds - where the usage of hardware resources is optimized. Utilizing OVB host clouds to deploy Openstack environments is a viable, verified alternative to requiring dedicated hardware for the purposes of development and testing.

The Tripleo Quickstart devmode.sh script, https://github.com/openstack/tripleo-quickstart/blob/master/devmode.sh provides developers with a one-line command to build and test changes
on a virthost.

We need to either extend devmode.sh, or build an equivalent script, to allow developers
to build and test changes using OVB.

Proposed change:
----------------

There are a number of OVB host cloud available for use - with assigned tenant access. Also, various groups are setting up host clouds using their own assigned hardware. Whatever the host cloud of choice, the user should be able to provide the script (via exports or prompts) credentials:

* os_username
* os_password
* os_tenant_name
* os_auth_url
* openstack_host_cloud_name (?)

and devmode.sh (or an equivalent script) should be able to build and test the change on the matching OVB cloud.

Desired outcome:
----------------

A user should be able to:
1. setup a TripleO (openstack) environment in an OVB host cloud with TripleO Quickstart from the command line
2. specify patches to be built into the deployment on the OVB stack for testing

Open questions/Considerations:
------------------------------
 - Since quotas in host clouds are also limited, the convention is for OVB jobs to clean up the environment and delete the OVB stack after a test has run. Since developers will using these environments for development and debugging, should we offer the option to skip the clean up and run the risk of polluting the environment and consuming resources?
 - There is no plan to add functionality to set up the OVB host cloud to devmode.sh. The assumption is that the host cloud is already patched, and configured with the required images, flavors etc. to begin a deployment.

[John Trowbridge (trown)]

Thanks for the excellent write up.

It seems like whether to add-on to devmode.sh or to create another script is also an open question. To me, the answer there would depend on how invasive the changes would need to be in order to add this.

For the teardown question, I think we should have an invocation of the script (devmode.sh or other) which does the teardown. It would then be the dev/user responsibility to clean up after themselves when they are done.

+1 to not including host cloud setup, that seems like a much different target audience.

[wes hayutin (weshayutin)]

Just adding some context to this RFE. The quickstart tool set already supports deploying in OVB environments and is running ovb based check gates in tripleo and internally. The focus of this RFE is to make it very simple for folks to not only create a dev/test tripleo env w/ ovb but to also integrate their patches into the build.

[Sagi (Sergey) Shnaidman (sshnaidm)]

I think it's great idea.
First of all we can keep it as much closer to CI as possible (both upstream and downstream), it'll help to reproduce OVB jobs too. Now it's possible only with virthost and vms, which has a few significant differences (like last vbmc issue, for example).
I agree that OVB setup is out of scope, but I think we'll need some tool to set up it for testing this devmode_ovb.sh script. Maybe it can go in parallel.
What regarding to cleaning environment, I think we can adopt the common behaviour in oooq now - teardown everything before and only then start to deploy a new stack.

[Ronelle Landy (rlandy)]

There are multiple pieces involved in the design.

We start by making sure the gating scenarios work for OVB:

Gating tripleo-quickstart-extras changes with dependencies on tripleo-quickstart changes is broken. The following two reviews are aimed at fixing that issue:

https://review.openstack.org/#/c/431764/ - Update ci-script to enable TripleO gate dependencies
https://review.openstack.org/#/c/431760/ - Clone tripleo-quickstart dependant change to tripleo-quickstart-gate-repo

(already +2'ed by adarazs)

Next we need to ensure OVB can work with delorean changes:

https://review.openstack.org/#/c/434004/ - Add option to build delorean changes and include the repo
https://review.openstack.org/#/c/425447 - Modify call to quickstart with options for repo include
(requires https://review.openstack.org/#/c/431764/ to merge first and then rebase)

Then there is the devmode-ovb.sh script:

https://review.openstack.org/#/c/444481/ - Add devmode-ovb script

This script is a modification of dev-mode.sh to call full-deploy-ovb.sh as we have the settings now.
This script could be simplified if we moved some settings which are currently in tripleo-environments to tripleo-quickstart-extras. The goals being that anyone running devmode-ovb.sh should not need to know about an internal repo tripleo-environments.

Proposal to move tripleo-environments settings related to OVB is covered in the next comment

[Ronelle Landy (rlandy)]

What is required to run devmode-ovb.sh?

=== 1. Access to an active tenant on a host cloud ===

Any user should have an rc file that can be sourced with the following environment variables:

 - OS_USERNAME
 - OS_PASSWORD
 - OS_TENANT_NAME
 - OS_AUTH_URL

This rc file could be sourced before devmode-ovb.sh is run. The current design requires the user to pass the path to the rc file and sources it in the script.

=== 2. Host cloud specific settings ===

Aside from the tenant access variables above, devmode-ovb.sh would also need to be passed settings related to the host cloud, such as:

 - the cloud name (to create a clouds.yaml file)
 - the images available in Glance to be used in the OVB stack deployment
 - dns servers
 - ntp servers
 - external network

For public clouds, these settings could be kept in a publicly available repo. For private clouds, the settings could be stored privately. As long as the settings file path is passed to devmode-ovb.sh, the script can run.
We could default the settings to the most commonly used host cloud without exposing user-specific tenants.

=== Example tenant on host cloud settings file ===

os_username: "{{ lookup('env','OS_USERNAME') }}"
os_password: "{{ lookup('env','OS_PASSWORD') }}"
os_tenant_name: "{{ lookup('env','OS_TENANT_NAME') }}"
os_auth_url: "{{ lookup('env','OS_AUTH_URL') }}"

cloud_name: mycloud
latest_guest_image:
    newton: centos-7
    ocata: centos-7
    master: centos-7
    rhos-9: rhel-7.3-server-x86_64-latest
    rhos-10: rhel-7.3-server-x86_64-latest

custom_nameserver: 8.8.8.8
pvt_nameserver: 8.8.8.8
undercloud_undercloud_nameservers: 8.8.8.8
external_net: '1.2.3.0/22'
dnservers: ["8.8.8.8", "4.4.4.4"]
ntp_server: 'pool.ntp.org'

=== 3. General OVB settings ===

After the OVB stack is deployed (see https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/ovb-manage-stack), the playbook run with quickstart.sh follows the baremetal deployment workflow. As such, OVB is considered as just another hardware environment, and we keep setting related to OVB in tripleo-environments, a private repo.

Since there is nothing private about these generalized OVB settings, the settings could be moved the a public repo (such as tripleo-quickstart-extras).
There is a review in tripleo-environments https://<code-eng>/gerrit/#/c/99022/ that removes any host cloud specific information and replaces that information with variables so that the settings could be stored in a host cloud specific file.

[Ronelle Landy (rlandy)]

=== Issues with moving the settings ===

The proposal to move the settings may need to move to a separate RFE.
Currently hardware-environments has a specific pattern to the directory structure. We would need to duplicate that pattern or modify the code in tripleo-quickstart-extras to not depend on that pattern.

[Ronelle Landy (rlandy)]

Devmode is up and running with OVB. Below are the log from a job setup running this:
https://thirdparty-logs.rdoproject.org/jenkins-tq-gate-devmode-master-ovb-rdocloud-public-bond-32/