OpenStack Charm Test Infra

Mojo Designate HA spec fails with multiple services not running that should be

Bug #1842057 reported by Liam Young on 2019-08-30

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Charm Test Infra	Confirmed	High	Liam Young

Bug Description

next_designate_ha spec for disco-stein is failing with multiple services not running that should be. The problem appears to be after vault setup

https://pastebin.canonical.com/p/nDCnJksJRb/

http://10.245.162.58:8080/job/mojo_runner/19589/consoleFull

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2019-08-30:

Download full text (24.0 KiB)

Model Controller Cloud/Region Version SLA Timestamp
auto-osci-sv13 auto-osci-sv13 serverstack/serverstack 2.6.8 unsupported 02:58:09Z

Model           Controller      Cloud/Region             Version  SLA          Timestamp
auto-osci-sv13  auto-osci-sv13  serverstack/serverstack  2.6.8    unsupported  02:58:09Z

App                    Version         Status   Scale  Charm                  Store  Rev  OS      Notes
aodh                   8.0.0           active       3  aodh                   local    0  ubuntu  
aodh-hacluster                         active       3  hacluster              local    0  ubuntu  
barbican               8.0.0           blocked      1  barbican               local    0  ubuntu  
ceilometer             12.0.0          blocked      3  ceilometer             local    0  ubuntu  
ceilometer-agent       12.0.0          active       3  ceilometer-agent       local    0  ubuntu  
ceilometer-hacluster                   active       3  hacluster              local    1  ubuntu  
ceph-mon               13.2.6          active       3  ceph-mon               local    0  ubuntu  
ceph-osd               13.2.6          active       3  ceph-osd               local   15  ubuntu  
cinder                 14.0.0          blocked      3  cinder                 local  136  ubuntu  
cinder-ceph            14.0.0          active       3  cinder-ceph            local    2  ubuntu  
cinder-hacluster                       active       3  hacluster              local    2  ubuntu  
dashboard-hacluster                    active       3  hacluster              local    3  ubuntu  
designate              8.0.0           blocked      3  designate              local    0  ubuntu  
designate-bind         9.11.5.P1+dfsg  active       3  designate-bind         local    0  ubuntu  
designate-hacluster                    active       3  hacluster              local    4  ubuntu  
glance                 18.0.0          active       3  glance                 local  150  ubuntu  
glance-hacluster                       active       3  hacluster              local    5  ubuntu  
gnocchi                4.3.2           active       3  gnocchi                local    0  ubuntu  
gnocchi-hacluster                      active       3  hacluster              local    6  ubuntu  
heat                   12.0.0          active       3  heat                   local   12  ubuntu  
heat-hacluster                         active       3  hacluster              local    7  ubuntu  
keystone               15.0.0          active       3  keystone               local    0  ubuntu  
keystone-hacluster                     active       3  hacluster              local    8  ubuntu  
memcached                              active       1  memcached              local   69  ubuntu  
mysql                  5.7.20-29.24    active       3  percona-cluster        local   45  ubuntu  
mysql-hacluster                        active       3  hacluster              local    9  ubuntu  
neutron-api            14.0.2          active       3  neutron-api            local    0  ubuntu  
neutron-gateway        14.0.2          blocked      1  neutron-gateway        local   64  ubuntu  
neutron-hacluster                      active       3  hacluster              local   10  ubuntu  
neutron-openvswitch    14.0.2          active       3  neutron-openvswitch    local    0  ubuntu  
nova-cc-hacluster                      active       3  hacluster              local   11  ubuntu  
nova-cloud-controller  19.0.1          blocked      3  nova-cloud-controller  local  501  ubuntu  
nova-compute           19.0.1          active       3  nova-compute           local  133  ubuntu  
openstack-dashboard    15.0.0          active       3  openstack-dashboard    local   32  ubuntu  
rabbitmq-server        3.7.8           active       1  rabbitmq-server        local  150  ubuntu  
swift-hacluster                        active       3  hacluster              local   12  ubuntu  
swift-proxy            2.21.0          active       3  swift-proxy            local  147  ubuntu  
swift-storage-z1       2.21.0          active       1  swift-storage          local   90  ubuntu  
swift-storage-z2       2.21.0          active       1  swift-storage          local   91  ubuntu  
swift-storage-z3       2.21.0          active       1  swift-storage          local   92  ubuntu  
tempest                2.20.1          active       1  tempest                local    0  ubuntu  
vault                  1.1.1           active       3  vault                  local    0  ubuntu  
vault-hacluster                        active       3  hacluster              local   13  ubuntu

Unit                       Workload  Agent  Machine  Public address  Ports                       Message
aodh/0*                    active    idle   0        172.17.113.9    8042/tcp                    Unit is ready
  aodh-hacluster/2         active    idle            172.17.113.9                                Unit is ready and clustered
aodh/1                     active    idle   1        172.17.113.25   8042/tcp                    Unit is ready
  aodh-hacluster/1         active    idle            172.17.113.25                               Unit is ready and clustered
aodh/2                     active    idle   2        172.17.113.4    8042/tcp                    Unit is ready
  aodh-hacluster/0*        active    idle            172.17.113.4                                Unit is ready and clustered
barbican/0*                blocked   idle   3        172.17.113.7    9311/tcp,9312/tcp           Services not running that should be: barbican-worker
ceilometer/0*              blocked   idle   4        172.17.113.26                               Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
  ceilometer-hacluster/1*  active    idle            172.17.113.26                               Unit is ready and clustered
ceilometer/1               blocked   idle   5        172.17.113.20                               Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
  ceilometer-hacluster/2   active    idle            172.17.113.20                               Unit is ready and clustered
ceilometer/2               blocked   idle   6        172.17.113.16                               Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
  ceilometer-hacluster/0   active    idle            172.17.113.16                               Unit is ready and clustered
ceph-mon/0                 active    idle   7        172.17.113.35                               Unit is ready and clustered
ceph-mon/1*                active    idle   8        172.17.113.5                                Unit is ready and clustered
ceph-mon/2                 active    idle   9        172.17.113.13                               Unit is ready and clustered
ceph-osd/0                 active    idle   10       172.17.113.11                               Unit is ready (1 OSD)
ceph-osd/1*                active    idle   11       172.17.113.28                               Unit is ready (1 OSD)
ceph-osd/2                 active    idle   12       172.17.113.21                               Unit is ready (1 OSD)
cinder/0                   blocked   idle   13       172.17.113.17   8776/tcp                    Services not running that should be: cinder-scheduler
  cinder-ceph/1            active    idle            172.17.113.17                               Unit is ready
  cinder-hacluster/2       active    idle            172.17.113.17                               Unit is ready and clustered
cinder/1                   blocked   idle   14       172.17.113.15   8776/tcp                    Services not running that should be: cinder-scheduler
  cinder-ceph/2            active    idle            172.17.113.15                               Unit is ready
  cinder-hacluster/1       active    idle            172.17.113.15                               Unit is ready and clustered
cinder/2*                  blocked   idle   15       172.17.113.12   8776/tcp                    Services not running that should be: cinder-scheduler
  cinder-ceph/0*           active    idle            172.17.113.12                               Unit is ready
  cinder-hacluster/0*      active    idle            172.17.113.12                               Unit is ready and clustered
designate-bind/0           active    idle   19       172.17.113.3                                Unit is ready
designate-bind/1           active    idle   20       172.17.113.33                               Unit is ready
designate-bind/2*          active    idle   21       172.17.113.36                               Unit is ready
designate/0                blocked   idle   16       172.17.113.44   9001/tcp                    Services not running that should be: designate-mdns, designate-producer, designate-worker
  designate-hacluster/2    active    idle            172.17.113.44                               Unit is ready and clustered
designate/1*               blocked   idle   17       172.17.113.27   9001/tcp                    Services not running that should be: designate-mdns, designate-producer, designate-worker
  designate-hacluster/0*   active    idle            172.17.113.27                               Unit is ready and clustered
designate/2                blocked   idle   18       172.17.113.32   9001/tcp                    Services not running that should be: designate-mdns, designate-producer, designate-worker
  designate-hacluster/1    active    idle            172.17.113.32                               Unit is ready and clustered
glance/0                   active    idle   22       172.17.113.24   9292/tcp                    Unit is ready
  glance-hacluster/0*      active    idle            172.17.113.24                               Unit is ready and clustered
glance/1*                  active    idle   23       172.17.113.29   9292/tcp                    Unit is ready
  glance-hacluster/2       active    idle            172.17.113.29                               Unit is ready and clustered
glance/2                   active    idle   24       172.17.113.31   9292/tcp                    Unit is ready
  glance-hacluster/1       active    idle            172.17.113.31                               Unit is ready and clustered
gnocchi/0                  active    idle   25       172.17.113.8    8041/tcp                    Unit is ready
  gnocchi-hacluster/2      active    idle            172.17.113.8                                Unit is ready and clustered
gnocchi/1*                 active    idle   26       172.17.113.22   8041/tcp                    Unit is ready
  gnocchi-hacluster/0*     active    idle            172.17.113.22                               Unit is ready and clustered
gnocchi/2                  active    idle   27       172.17.113.18   8041/tcp                    Unit is ready
  gnocchi-hacluster/1      active    idle            172.17.113.18                               Unit is ready and clustered
heat/0                     active    idle   28       172.17.113.40   8000/tcp,8004/tcp           Unit is ready
  heat-hacluster/1         active    idle            172.17.113.40                               Unit is ready and clustered
heat/1*                    active    idle   29       172.17.113.64   8000/tcp,8004/tcp           Unit is ready
  heat-hacluster/2         active    idle            172.17.113.64                               Unit is ready and clustered
heat/2                     active    idle   30       172.17.113.14   8000/tcp,8004/tcp           Unit is ready
  heat-hacluster/0*        active    idle            172.17.113.14                               Unit is ready and clustered
keystone/0*                active    idle   31       172.17.113.57   5000/tcp                    Unit is ready
  keystone-hacluster/0*    active    idle            172.17.113.57                               Unit is ready and clustered
keystone/1                 active    idle   32       172.17.113.51   5000/tcp                    Unit is ready
  keystone-hacluster/2     active    idle            172.17.113.51                               Unit is ready and clustered
keystone/2                 active    idle   33       172.17.113.52   5000/tcp                    Unit is ready
  keystone-hacluster/1     active    idle            172.17.113.52                               Unit is ready and clustered
memcached/0*               active    idle   34       172.17.113.10   11211/tcp                   Unit is ready
mysql/0                    active    idle   35       172.17.113.37   3306/tcp                    Unit is ready
  mysql-hacluster/1        active    idle            172.17.113.37                               Unit is ready and clustered
mysql/1                    active    idle   36       172.17.113.66   3306/tcp                    Unit is ready
  mysql-hacluster/2        active    idle            172.17.113.66                               Unit is ready and clustered
mysql/2*                   active    idle   37       172.17.113.39   3306/tcp                    Unit is ready
  mysql-hacluster/0*       active    idle            172.17.113.39                               Unit is ready and clustered
neutron-api/0              active    idle   38       172.17.113.45   9696/tcp                    Unit is ready
  neutron-hacluster/2      active    idle            172.17.113.45                               Unit is ready and clustered
neutron-api/1              active    idle   39       172.17.113.34   9696/tcp                    Unit is ready
  neutron-hacluster/1      active    idle            172.17.113.34                               Unit is ready and clustered
neutron-api/2*             active    idle   40       172.17.113.38   9696/tcp                    Unit is ready
  neutron-hacluster/0*     active    idle            172.17.113.38                               Unit is ready and clustered
neutron-gateway/0*         blocked   idle   41       172.17.113.61                               Services not running that should be: neutron-metering-agent, neutron-lbaasv2-agent, neutron-openvswitch-agent, neutron-dhcp-agent
nova-cloud-controller/0    blocked   idle   42       172.17.113.74   8774/tcp,8775/tcp,8778/tcp  Services not running that should be: nova-scheduler
  nova-cc-hacluster/0*     active    idle            172.17.113.74                               Unit is ready and clustered
nova-cloud-controller/1*   blocked   idle   43       172.17.113.58   8774/tcp,8775/tcp,8778/tcp  Services not running that should be: nova-scheduler
  nova-cc-hacluster/1      active    idle            172.17.113.58                               Unit is ready and clustered
nova-cloud-controller/2    blocked   idle   44       172.17.113.71   8774/tcp,8775/tcp,8778/tcp  Services not running that should be: nova-scheduler
  nova-cc-hacluster/2      active    idle            172.17.113.71                               Unit is ready and clustered
nova-compute/0             active    idle   45       172.17.113.77                               Unit is ready
  ceilometer-agent/2       active    idle            172.17.113.77                               Unit is ready
  neutron-openvswitch/2    active    idle            172.17.113.77                               Unit is ready
nova-compute/1*            active    idle   46       172.17.113.47                               Unit is ready
  ceilometer-agent/0*      active    idle            172.17.113.47                               Unit is ready
  neutron-openvswitch/0*   active    idle            172.17.113.47                               Unit is ready
nova-compute/2             active    idle   47       172.17.113.78                               Unit is ready
  ceilometer-agent/1       active    idle            172.17.113.78                               Unit is ready
  neutron-openvswitch/1    active    idle            172.17.113.78                               Unit is ready
openstack-dashboard/0*     active    idle   48       172.17.113.60   80/tcp,443/tcp              Unit is ready
  dashboard-hacluster/1*   active    idle            172.17.113.60                               Unit is ready and clustered
openstack-dashboard/1      active    idle   49       172.17.113.63   80/tcp,443/tcp              Unit is ready
  dashboard-hacluster/2    active    idle            172.17.113.63                               Unit is ready and clustered
openstack-dashboard/2      active    idle   50       172.17.113.68   80/tcp,443/tcp              Unit is ready
  dashboard-hacluster/0    active    idle            172.17.113.68                               Unit is ready and clustered
rabbitmq-server/0*         active    idle   51       172.17.113.87   5671/tcp,5672/tcp           Unit is ready
swift-proxy/0*             active    idle   52       172.17.113.48   8080/tcp                    Unit is ready
  swift-hacluster/0*       active    idle            172.17.113.48                               Unit is ready and clustered
swift-proxy/1              active    idle   53       172.17.113.41   8080/tcp                    Unit is ready
  swift-hacluster/1        active    idle            172.17.113.41                               Unit is ready and clustered
swift-proxy/2              active    idle   54       172.17.113.65   8080/tcp                    Unit is ready
  swift-hacluster/2        active    idle            172.17.113.65                               Unit is ready and clustered
swift-storage-z1/0*        active    idle   55       172.17.113.23                               Unit is ready
swift-storage-z2/0*        active    idle   56       172.17.113.19                               Unit is ready
swift-storage-z3/0*        active    idle   57       172.17.113.69                               Unit is ready
tempest/0*                 active    idle   58       172.17.113.43                               Unit is ready
vault/0                    active    idle   59       172.17.113.67   8200/tcp                    Unit is ready (active: true, mlock: enabled)
  vault-hacluster/2        active    idle            172.17.113.67                               Unit is ready and clustered
vault/1*                   active    idle   60       172.17.113.54   8200/tcp                    Unit is ready (active: true, mlock: enabled)
  vault-hacluster/1        active    idle            172.17.113.54                               Unit is ready and clustered
vault/2                    active    idle   61       172.17.113.93   8200/tcp                    Unit is ready (active: true, mlock: enabled)
  vault-hacluster/0*       active    idle            172.17.113.93                               Unit is ready and clustered

Machine  State    DNS            Inst id                               Series  AZ    Message
0        started  172.17.113.9   db70895a-df1e-4739-a1d2-3869ba1f10c6  disco   nova  ACTIVE
1        started  172.17.113.25  69d52376-3794-41ce-9a3f-0dcc81e3d6de  disco   nova  ACTIVE
2        started  172.17.113.4   b703ea13-92bf-41d6-8fea-f55ee3ac3c80  disco   nova  ACTIVE
3        started  172.17.113.7   93ea4ccb-256a-4d42-85ab-9a55faeffa76  disco   nova  ACTIVE
4        started  172.17.113.26  bce1af0c-6b6b-4891-a3ab-67fc06f96e3f  disco   nova  ACTIVE
5        started  172.17.113.20  d17a5aa6-5837-448c-aca4-7537150944d9  disco   nova  ACTIVE
6        started  172.17.113.16  9b63ed07-cd0e-4001-b752-087029dda26f  disco   nova  ACTIVE
7        started  172.17.113.35  94afeccb-af5c-4fa6-93f2-6a4ddef2a413  disco   nova  ACTIVE
8        started  172.17.113.5   d08d0154-e3f7-43cd-beed-67c863179814  disco   nova  ACTIVE
9        started  172.17.113.13  a39e449f-6b83-4247-afed-07b2b4a102fc  disco   nova  ACTIVE
10       started  172.17.113.11  5b309184-178d-4173-8e76-adfd8fd2661b  disco   nova  ACTIVE
11       started  172.17.113.28  87c4b46c-af4f-4308-9f82-34452c3c599b  disco   nova  ACTIVE
12       started  172.17.113.21  139cfae6-8b31-410a-bbdb-547a78350f03  disco   nova  ACTIVE
13       started  172.17.113.17  441e470f-3d4b-4189-adfb-7eb76e6525c8  disco   nova  ACTIVE
14       started  172.17.113.15  6280a1ed-7ee4-4107-83df-b587d8b5d286  disco   nova  ACTIVE
15       started  172.17.113.12  20dd49b4-4794-4dbc-86ed-4c65100d9e2d  disco   nova  ACTIVE
16       started  172.17.113.44  16bb0015-da73-4ea6-88ce-6029459b1eb8  disco   nova  ACTIVE
17       started  172.17.113.27  e1eea084-398e-4bde-bb9a-e3f5be23b966  disco   nova  ACTIVE
18       started  172.17.113.32  866a4bbb-34ab-4841-9864-3d204170fb6c  disco   nova  ACTIVE
19       started  172.17.113.3   c6c34539-2047-4b1a-b2ac-851b886c6b50  disco   nova  ACTIVE
20       started  172.17.113.33  34b551f7-f327-48b9-83ac-a69f7ad4a992  disco   nova  ACTIVE
21       started  172.17.113.36  0035bfdb-3675-4fd8-a38e-5922f7755db1  disco   nova  ACTIVE
22       started  172.17.113.24  39bf81c5-fb93-466c-b7fb-f6e772499eca  disco   nova  ACTIVE
23       started  172.17.113.29  8bd56079-3997-49f9-9bc4-aa75f2e005ff  disco   nova  ACTIVE
24       started  172.17.113.31  b2343fa2-19fc-45f3-bec9-9149fe0497bd  disco   nova  ACTIVE
25       started  172.17.113.8   2ff4a045-e41d-4025-ae0b-16e5086f2fd9  disco   nova  ACTIVE
26       started  172.17.113.22  e189bbd1-4d7d-46e0-ae1b-8c723383219a  disco   nova  ACTIVE
27       started  172.17.113.18  20763cb1-063a-49a3-8cee-9cdf54192f86  disco   nova  ACTIVE
28       started  172.17.113.40  6e8fb254-0094-450a-8876-3a6dd56b0530  disco   nova  ACTIVE
29       started  172.17.113.64  24ac2f37-fbf7-44fd-b538-28a4096a3c38  disco   nova  ACTIVE
30       started  172.17.113.14  3207779a-d042-45f6-b5c1-a2b6cfa100af  disco   nova  ACTIVE
31       started  172.17.113.57  96a1b525-7110-4afe-9f05-de1bfde39c76  disco   nova  ACTIVE
32       started  172.17.113.51  c8b12fa8-c21a-41b6-899a-8db470f51f65  disco   nova  ACTIVE
33       started  172.17.113.52  09c782a0-f05e-476e-8c3f-f4dd5b260455  disco   nova  ACTIVE
34       started  172.17.113.10  d08c45d5-f0ed-4914-bca2-502103948699  disco   nova  ACTIVE
35       started  172.17.113.37  ab5d1e43-ba98-4275-8a0e-98e8276cdcf9  disco   nova  ACTIVE
36       started  172.17.113.66  0f6d807b-4c05-4f76-9fab-a59dfed98662  disco   nova  ACTIVE
37       started  172.17.113.39  a4dac60a-066b-4330-949f-7e930abcc20e  disco   nova  ACTIVE
38       started  172.17.113.45  21f0c2f3-815b-40e0-a0e8-fbdab9d3dbca  disco   nova  ACTIVE
39       started  172.17.113.34  c44ec2bd-51f5-432d-bc1e-79ab5dc6fbb1  disco   nova  ACTIVE
40       started  172.17.113.38  2cf339da-511f-47d3-a863-e2061a06b377  disco   nova  ACTIVE
41       started  172.17.113.61  867a21cb-051a-4536-9706-8adddd661730  disco   nova  ACTIVE
42       started  172.17.113.74  d10673bf-99c4-42c0-a33b-79b058d2422f  disco   nova  ACTIVE
43       started  172.17.113.58  f81f6a9b-c81b-41b0-b489-1fd7e5f837c3  disco   nova  ACTIVE
44       started  172.17.113.71  f5436e1c-4b6c-4f8e-9b8f-5eb72ae86d58  disco   nova  ACTIVE
45       started  172.17.113.77  65cbcf1b-0e6a-4a82-9b03-bb96cdc2309a  disco   nova  ACTIVE
46       started  172.17.113.47  60abbd19-1b3c-47ee-a51d-87b102a8ed4a  disco   nova  ACTIVE
47       started  172.17.113.78  f8b5e174-0256-4298-9ccd-8c2207c2fd94  disco   nova  ACTIVE
48       started  172.17.113.60  e6b107c7-a47a-48d3-9c7c-9e0c0e3374e5  disco   nova  ACTIVE
49       started  172.17.113.63  790665d3-1080-4546-bbf3-f14ecd5b8376  disco   nova  ACTIVE
50       started  172.17.113.68  5eebaa99-b24d-4d2a-841c-e5e1faf60443  disco   nova  ACTIVE
51       started  172.17.113.87  22c1ede9-38fd-4f8a-a7d4-3d3b41caaaca  disco   nova  ACTIVE
52       started  172.17.113.48  38f60a2e-72b0-4841-a6c2-510b3356e1d3  disco   nova  ACTIVE
53       started  172.17.113.41  5aa13f0b-e992-42c0-8838-48b7c8a15d75  disco   nova  ACTIVE
54       started  172.17.113.65  6b2cc6fe-af0b-48b9-bc94-50e1ffe32301  disco   nova  ACTIVE
55       started  172.17.113.23  5326071c-1e4e-496f-a0df-cd1b73f44248  disco   nova  ACTIVE
56       started  172.17.113.19  2520d847-478b-405b-b1c6-db2cafec7bf0  disco   nova  ACTIVE
57       started  172.17.113.69  bfb50af9-c3b3-4bc1-bf31-87fa4252be52  disco   nova  ACTIVE
58       started  172.17.113.43  f2728b89-d6a2-42d7-8488-4cd0f666762a  disco   nova  ACTIVE
59       started  172.17.113.67  48899a08-ddb4-435a-921c-9eaa84e89845  disco   nova  ACTIVE
60       started  172.17.113.54  6f3681cb-1f0f-4e6f-9779-02b599b2ac52  disco   nova  ACTIVE
61       started  172.17.113.93  6c58bcce-175a-4427-a1ca-afcaeb568805  disco   nova  ACTIVE

Liam Young (gnuoy) on 2019-09-02

Changed in charm-test-infra:
status:	New → Confirmed
importance:	Undecided → High
assignee:	nobody → Liam Young (gnuoy)

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1842056

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.