get_discovery_url doesn't fully check for empty URL strings

[Martin Falatic]

I created a bay today and it threw an error in etcd on the m8s master:

Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal systemd[1]: Started Etcd Server.
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal etcd[2120]: 2015/09/01 22:32:37 etcd: listening for peers on http://10.0.0.5:2380
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal etcd[2120]: 2015/09/01 22:32:37 etcd: listening for client requests on http://0.0.0.0:2379
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal etcd[2120]: 2015/09/01 22:32:37 etcdserver: datadir is valid for the 2.0.1 format
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal etcd[2120]: 2015/09/01 22:32:37 etcd: stopping listening for client requests on http://0.0.0.0:2379
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal etcd[2120]: 2015/09/01 22:32:37 etcd: stopping listening for peers on http://10.0.0.5:2380
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal etcd[2120]: 2015/09/01 22:32:37 etcd: 10.0.0.5 has different advertised URLs in the cluster and advertised peer URLs list
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal systemd[1]: Unit etcd.service entered failed state.
Sep 01 22:32:37 k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp.novalocal systemd[1]: etcd.service failed.

The conf file showed the ETCD_DISCOVERY url to be empty:

[minion@k8-hkkfgoddp4-0-ku73jhlbrqbs-kube-master-zcnp3icurtsp ~]$ cat /etc/etcd/etcd.conf
ETCD_NAME="10.0.0.5"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_LISTEN_PEER_URLS="http://10.0.0.5:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.5:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.5:2380"
ETCD_DISCOVERY=""

This was also seen in magnum bay-show and the debug logs ("discovery_url": "")

In magnum/conductor/template_definition.py:375 there is a check for this. If the discovery_url attribute does not exist OR the attribute is empty further processing is done to get the URL via requests.get(discovery_endpoint).text. However, if that fails and returns an empty string no further checking or retrying is performed.

This call should fail explicitly if the URL is empty (possibly after retrying), so as to not create a bay without a discovery_url.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/219786

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/219786
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=d0c01ede2e823fd07530efc4a93669179673728f
Submitter: Jenkins
Branch: master

commit d0c01ede2e823fd07530efc4a93669179673728f
Author: Martin Falatic <email address hidden>
Date: Wed Sep 2 09:52:16 2015 -0700

    Enhanced error checking in get_discovery_url

    get_discovery_url should never return an empty
    URL. Added check and exception for this situation.

    Co-Authored-By: Vilobh Meshram <email address hidden>

    DocImpact
    Closes-Bug: #1491178
    Change-Id: I2d5b3996c0ee53f40399ca3fd0d6b987daa26f79