Comment 0 for bug 1667443

A simple 4-node deployment becomes inaccessible (namely glance) after one pass of the tempest smoke tests. Deployment is via juju charms (juju bundle/deployment file attached).

$ glance image-list
Error finding address for http://10.244.222.221:9292/versions: Unable to establish connection to http://10.244.222.221:9292/versions

$ telnet 10.244.222.221 9292
Trying 10.244.222.221...
Connected to 10.244.222.221.
Escape character is '^]'.
...

$ wget http://10.244.222.221:9292
--2017-02-23 19:10:21-- http://10.244.222.221:9292/
Connecting to 10.244.222.221:9292... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
...

There are 3 tempest failures (not the subject of this bug) and some cleanup is not being fully done (namely, there are 2 images left behind after tempest finishes, tempest-image-xxxxx and tempest-new-image-xxxxx).

Juju status output:

$ juju status
Model Controller Cloud/Region Version
nsantos maas maas 2.1-beta4

App Version Status Scale Charm Store Rev OS Notes
ceph-mon 10.2.5 active 1 ceph-mon jujucharms 7 ubuntu
ceph-osd 10.2.5 active 1 ceph-osd jujucharms 239 ubuntu
cinder 9.0.0 active 1 cinder jujucharms 258 ubuntu
glance 13.0.0 active 1 glance jujucharms 254 ubuntu
keystone 10.0.0 active 1 keystone jujucharms 262 ubuntu
mysql 5.6.21-25.8 active 1 percona-cluster jujucharms 247 ubuntu
neutron-api 9.0.0 active 1 neutron-api jujucharms 247 ubuntu
neutron-gateway 9.0.0 active 1 neutron-gateway jujucharms 232 ubuntu
neutron-openvswitch 9.0.0 active 2 neutron-openvswitch jujucharms 238 ubuntu
nova-cloud-controller 14.0.1 active 1 nova-cloud-controller jujucharms 292 ubuntu
nova-compute 14.0.1 active 2 nova-compute jujucharms 262 ubuntu
openstack-dashboard 10.0.0 active 1 openstack-dashboard jujucharms 243 ubuntu
rabbitmq-server 3.5.7 active 1 rabbitmq-server jujucharms 59 ubuntu
tempest 2.7.4 active 1 tempest jujucharms 31 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-mon/0* active idle 0/lxd/0 10.244.249.208 Unit is ready and clustered
ceph-osd/0* active idle 0 10.244.199.82 Unit is ready (1 OSD)
cinder/0* active idle 2 10.244.223.157 8776/tcp Unit is ready
glance/0* active idle 1/lxd/0 10.244.222.221 9292/tcp Unit is ready
keystone/0* active idle 2/lxd/0 10.244.223.199 5000/tcp Unit is ready
mysql/0* active idle 3/lxd/0 10.244.222.224 Unit is ready
neutron-api/0* active idle 0/lxd/1 10.244.222.137 9696/tcp Unit is ready
neutron-gateway/0* active idle 3 10.244.222.230 Unit is ready
nova-cloud-controller/0* active idle 1/lxd/1 10.244.222.207 8774/tcp Unit is ready
nova-compute/0* active idle 0 10.244.199.82 Unit is ready
  neutron-openvswitch/0* active idle 10.244.199.82 Unit is ready
nova-compute/1 active idle 1 10.244.222.120 Unit is ready
  neutron-openvswitch/1 active idle 10.244.222.120 Unit is ready
openstack-dashboard/0* active idle 2/lxd/1 10.244.197.71 80/tcp,443/tcp Unit is ready
rabbitmq-server/0* active idle 3/lxd/1 10.244.223.55 5672/tcp Unit is ready
tempest/0* active idle 0 10.244.199.82 Unit is ready

Machine State DNS Inst id Series AZ
0 started 10.244.199.82 4y3hea xenial default
0/lxd/0 started 10.244.249.208 juju-108abe-0-lxd-0 xenial
0/lxd/1 started 10.244.222.137 juju-108abe-0-lxd-1 xenial
1 started 10.244.222.120 4y4cr8 xenial default
1/lxd/0 started 10.244.222.221 juju-108abe-1-lxd-0 xenial
1/lxd/1 started 10.244.222.207 juju-108abe-1-lxd-1 xenial
2 started 10.244.223.157 4y4crn xenial default
2/lxd/0 started 10.244.223.199 juju-108abe-2-lxd-0 xenial
2/lxd/1 started 10.244.197.71 juju-108abe-2-lxd-1 xenial
3 started 10.244.222.230 4y4crr xenial default
3/lxd/0 started 10.244.222.224 juju-108abe-3-lxd-0 xenial
3/lxd/1 started 10.244.223.55 juju-108abe-3-lxd-1 xenial

Relation Provides Consumes Type
mon ceph-mon ceph-mon peer
mon ceph-mon ceph-osd regular
ceph ceph-mon cinder regular
ceph ceph-mon glance regular
ceph ceph-mon nova-compute regular
cluster cinder cinder peer
image-service cinder glance regular
identity-service cinder keystone regular
shared-db cinder mysql regular
cinder-volume-service cinder nova-cloud-controller regular
amqp cinder rabbitmq-server regular
cluster glance glance peer
identity-service glance keystone regular
shared-db glance mysql regular
image-service glance nova-cloud-controller regular
image-service glance nova-compute regular
cluster keystone keystone peer
shared-db keystone mysql regular
identity-service keystone neutron-api regular
identity-service keystone nova-cloud-controller regular
identity-service keystone openstack-dashboard regular
identity-admin keystone tempest regular
cluster mysql mysql peer
shared-db mysql neutron-api regular
shared-db mysql nova-cloud-controller regular
cluster neutron-api neutron-api peer
neutron-plugin-api neutron-api neutron-gateway regular
neutron-plugin-api neutron-api neutron-openvswitch regular
neutron-api neutron-api nova-cloud-controller regular
amqp neutron-api rabbitmq-server regular
cluster neutron-gateway neutron-gateway peer
quantum-network-service neutron-gateway nova-cloud-controller regular
amqp neutron-gateway rabbitmq-server regular
neutron-plugin neutron-openvswitch nova-compute regular
amqp neutron-openvswitch rabbitmq-server regular
cluster nova-cloud-controller nova-cloud-controller peer
cloud-compute nova-cloud-controller nova-compute regular
amqp nova-cloud-controller rabbitmq-server regular
neutron-plugin nova-compute neutron-openvswitch subordinate
compute-peer nova-compute nova-compute peer
amqp nova-compute rabbitmq-server regular
cluster openstack-dashboard openstack-dashboard peer
cluster rabbitmq-server rabbitmq-server peer