| 2021-09-01 05:13:13 |
Eric Chen |
bug |
|
|
added bug |
| 2021-09-01 05:36:45 |
Eric Chen |
attachment added |
|
juju-crashdump-5e17f8a2-4b96-4419-a3d6-942991666a73-2.tar.xz https://bugs.launchpad.net/charm-keystone/+bug/1942289/+attachment/5522075/+files/juju-crashdump-5e17f8a2-4b96-4419-a3d6-942991666a73-2.tar.xz |
|
| 2021-09-01 07:04:19 |
Eric Chen |
description |
Description
==========
Keystone HA is enabled.
After I scale up 1 keystone unit and remove the keystone unit with vip.
The cluster become "Database not initialised".
Versions
===============================
Cloud focal-wallaby
Deploy openstack from https://github.com/openstack-charmers/openstack-bundles
(git commit: 3f74a67704ed7654e0f574ccfd724dc3a224c9f8)
Modify the charm version and turn on ovn in bundle.yaml
diff --git a/stable/openstack-base/bundle.yaml b/stable/openstack-base/bundle.yaml
index 39297cf..c52ecb2 100644
--- a/stable/openstack-base/bundle.yaml
+++ b/stable/openstack-base/bundle.yaml
@@ -219,7 +219,7 @@ applications:
annotations:
gui-x: '300'
gui-y: '1270'
- charm: cs:keystone-323
+ charm: cs:keystone-326
num_units: 1
options:
worker-multiplier: *worker-multiplier
@@ -359,7 +359,7 @@ applications:
# top of this file.
options:
ovn-bridge-mappings: physnet1:br-ex
- bridge-interface-mappings: *data-port
+ #bridge-interface-mappings: *data-port
vault-mysql-router:
annotations:
gui-x: '1535'
Reproduce Step
===================
1. Prepare the Openstack environment like above in serverstack.
2. Turn on Keystone HA
juju add-unit -n 2 keystone
juju config keystone vip=10.5.55.1
juju deploy --config cluster_count=3 hacluster keystone-hacluster
juju add-relation keystone-hacluster:ha keystone:ha
Everything is still good except the certificate issue in LP: #1930763
3. Scale up 1 keystone unit
juju add-unit -n 1 keystone
4. Remove the keystone unit with VIP
juju remove-unit keystone/28
After that, the cluster status become "blcoked"
Unit Workload Agent Machine Public address Ports Message
keystone/29 blocked idle 122 10.5.2.26 5000/tcp Database not initialised
keystone-hacluster/18 active idle 10.5.2.26 Unit is ready and clustered
keystone-mysql-router/50 active idle 10.5.2.26 Unit is ready
keystone/30* blocked idle 123 10.5.3.123 5000/tcp Database not initialised
keystone-hacluster/17 active idle 10.5.3.123 Unit is ready and clustered
keystone-mysql-router/49* active idle 10.5.3.123 Unit is ready
keystone/31 blocked idle 124 10.5.2.152 5000/tcp Database not initialised
keystone-hacluster/19* active idle 10.5.2.152 Unit is ready and clustered
keystone-mysql-router/51 active idle 10.5.2.152 Unit is ready
Detail Information
=================
I find related log in keystone/29.
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 INFO unit.keystone/29.juju-log server.go:314 Keystone charm unit not ready - deferring identity-relation updates
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peers this unit is: NOTREADY
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peer behind relation cluster:311 that keystone/29 is NOTREADY
At the meantime, one of the mysql-innodb-cluster unit become error.
This error can be fixed after this command
juju resolve --no-retry mysql-innodb-cluster/4
But keystone is still stay in blocked status.
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/3 active idle 69 10.5.3.213 Unit is ready: Mode: R/W, Cluster is ONLIN
E and can tolerate up to ONE failure. mysql-innodb-cluster/4 error idle 70 10.5.1.215 hook failed: "update-status"
mysql-innodb-cluster/5* active idle 71 10.5.1.138 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
I will attach juju crashdump later, it's too big to upload now.
Expected result
===============================
keystone cluster action complete and the unit is "active" state. |
Description
==========
Keystone HA is enabled.
After I scale up 1 keystone unit and remove the keystone unit with vip.
The cluster become "Database not initialised".
Versions
===============================
Cloud focal-wallaby
Deploy openstack from https://github.com/openstack-charmers/openstack-bundles
(git commit: 3f74a67704ed7654e0f574ccfd724dc3a224c9f8)
Modify the charm version and turn on ovn in bundle.yaml
diff --git a/stable/openstack-base/bundle.yaml b/stable/openstack-base/bundle.yaml
index 39297cf..c52ecb2 100644
--- a/stable/openstack-base/bundle.yaml
+++ b/stable/openstack-base/bundle.yaml
@@ -219,7 +219,7 @@ applications:
annotations:
gui-x: '300'
gui-y: '1270'
- charm: cs:keystone-323
+ charm: cs:keystone-326
num_units: 1
options:
worker-multiplier: *worker-multiplier
@@ -359,7 +359,7 @@ applications:
# top of this file.
options:
ovn-bridge-mappings: physnet1:br-ex
- bridge-interface-mappings: *data-port
+ #bridge-interface-mappings: *data-port
vault-mysql-router:
annotations:
gui-x: '1535'
Reproduce Step
===================
1. Prepare the Openstack environment like above in serverstack.
2. Turn on Keystone HA
juju add-unit -n 2 keystone
juju config keystone vip=10.5.55.1
juju deploy --config cluster_count=3 hacluster keystone-hacluster
juju add-relation keystone-hacluster:ha keystone:ha
Everything is good except the certificate issue in LP: #1930763
3. Scale up 1 keystone unit
juju add-unit -n 1 keystone
4. Remove the keystone unit with VIP
juju remove-unit keystone/28
After that, the cluster status become "blcoked"
Unit Workload Agent Machine Public address Ports Message
keystone/29 blocked idle 122 10.5.2.26 5000/tcp Database not initialised
keystone-hacluster/18 active idle 10.5.2.26 Unit is ready and clustered
keystone-mysql-router/50 active idle 10.5.2.26 Unit is ready
keystone/30* blocked idle 123 10.5.3.123 5000/tcp Database not initialised
keystone-hacluster/17 active idle 10.5.3.123 Unit is ready and clustered
keystone-mysql-router/49* active idle 10.5.3.123 Unit is ready
keystone/31 blocked idle 124 10.5.2.152 5000/tcp Database not initialised
keystone-hacluster/19* active idle 10.5.2.152 Unit is ready and clustered
keystone-mysql-router/51 active idle 10.5.2.152 Unit is ready
Detail Information
=================
I find related log in keystone/29.
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 INFO unit.keystone/29.juju-log server.go:314 Keystone charm unit not ready - deferring identity-relation updates
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peers this unit is: NOTREADY
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peer behind relation cluster:311 that keystone/29 is NOTREADY
At the meantime, one of the mysql-innodb-cluster unit become error.
This error can be fixed after this command
juju resolve --no-retry mysql-innodb-cluster/4
But keystone is still stay in blocked status.
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/3 active idle 69 10.5.3.213 Unit is ready: Mode: R/W, Cluster is ONLIN
E and can tolerate up to ONE failure. mysql-innodb-cluster/4 error idle 70 10.5.1.215 hook failed: "update-status"
mysql-innodb-cluster/5* active idle 71 10.5.1.138 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
I will attach juju crashdump later, it's too big to upload now.
Expected result
===============================
keystone cluster action complete and the unit is "active" state. |
|
| 2021-09-01 07:05:57 |
Eric Chen |
description |
Description
==========
Keystone HA is enabled.
After I scale up 1 keystone unit and remove the keystone unit with vip.
The cluster become "Database not initialised".
Versions
===============================
Cloud focal-wallaby
Deploy openstack from https://github.com/openstack-charmers/openstack-bundles
(git commit: 3f74a67704ed7654e0f574ccfd724dc3a224c9f8)
Modify the charm version and turn on ovn in bundle.yaml
diff --git a/stable/openstack-base/bundle.yaml b/stable/openstack-base/bundle.yaml
index 39297cf..c52ecb2 100644
--- a/stable/openstack-base/bundle.yaml
+++ b/stable/openstack-base/bundle.yaml
@@ -219,7 +219,7 @@ applications:
annotations:
gui-x: '300'
gui-y: '1270'
- charm: cs:keystone-323
+ charm: cs:keystone-326
num_units: 1
options:
worker-multiplier: *worker-multiplier
@@ -359,7 +359,7 @@ applications:
# top of this file.
options:
ovn-bridge-mappings: physnet1:br-ex
- bridge-interface-mappings: *data-port
+ #bridge-interface-mappings: *data-port
vault-mysql-router:
annotations:
gui-x: '1535'
Reproduce Step
===================
1. Prepare the Openstack environment like above in serverstack.
2. Turn on Keystone HA
juju add-unit -n 2 keystone
juju config keystone vip=10.5.55.1
juju deploy --config cluster_count=3 hacluster keystone-hacluster
juju add-relation keystone-hacluster:ha keystone:ha
Everything is good except the certificate issue in LP: #1930763
3. Scale up 1 keystone unit
juju add-unit -n 1 keystone
4. Remove the keystone unit with VIP
juju remove-unit keystone/28
After that, the cluster status become "blcoked"
Unit Workload Agent Machine Public address Ports Message
keystone/29 blocked idle 122 10.5.2.26 5000/tcp Database not initialised
keystone-hacluster/18 active idle 10.5.2.26 Unit is ready and clustered
keystone-mysql-router/50 active idle 10.5.2.26 Unit is ready
keystone/30* blocked idle 123 10.5.3.123 5000/tcp Database not initialised
keystone-hacluster/17 active idle 10.5.3.123 Unit is ready and clustered
keystone-mysql-router/49* active idle 10.5.3.123 Unit is ready
keystone/31 blocked idle 124 10.5.2.152 5000/tcp Database not initialised
keystone-hacluster/19* active idle 10.5.2.152 Unit is ready and clustered
keystone-mysql-router/51 active idle 10.5.2.152 Unit is ready
Detail Information
=================
I find related log in keystone/29.
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 INFO unit.keystone/29.juju-log server.go:314 Keystone charm unit not ready - deferring identity-relation updates
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peers this unit is: NOTREADY
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peer behind relation cluster:311 that keystone/29 is NOTREADY
At the meantime, one of the mysql-innodb-cluster unit become error.
This error can be fixed after this command
juju resolve --no-retry mysql-innodb-cluster/4
But keystone is still stay in blocked status.
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/3 active idle 69 10.5.3.213 Unit is ready: Mode: R/W, Cluster is ONLIN
E and can tolerate up to ONE failure. mysql-innodb-cluster/4 error idle 70 10.5.1.215 hook failed: "update-status"
mysql-innodb-cluster/5* active idle 71 10.5.1.138 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
I will attach juju crashdump later, it's too big to upload now.
Expected result
===============================
keystone cluster action complete and the unit is "active" state. |
Description
==========
Keystone HA is enabled.
After I scale up 1 keystone unit and remove the keystone unit with vip.
The cluster become "Database not initialised".
Versions
===============================
Cloud focal-wallaby
Deploy openstack from https://github.com/openstack-charmers/openstack-bundles
(git commit: 3f74a67704ed7654e0f574ccfd724dc3a224c9f8)
Modify the charm version and turn on ovn in bundle.yaml
diff --git a/stable/openstack-base/bundle.yaml b/stable/openstack-base/bundle.yaml
index 39297cf..c52ecb2 100644
--- a/stable/openstack-base/bundle.yaml
+++ b/stable/openstack-base/bundle.yaml
@@ -219,7 +219,7 @@ applications:
annotations:
gui-x: '300'
gui-y: '1270'
- charm: cs:keystone-323
+ charm: cs:keystone-326
num_units: 1
options:
worker-multiplier: *worker-multiplier
@@ -359,7 +359,7 @@ applications:
# top of this file.
options:
ovn-bridge-mappings: physnet1:br-ex
- bridge-interface-mappings: *data-port
+ #bridge-interface-mappings: *data-port
vault-mysql-router:
annotations:
gui-x: '1535'
Reproduce Step
===================
1. Prepare the Openstack environment like above in serverstack.
2. Turn on Keystone HA
juju add-unit -n 2 keystone
juju config keystone vip=10.5.55.1
juju deploy --config cluster_count=3 hacluster keystone-hacluster
juju add-relation keystone-hacluster:ha keystone:ha
Everything is good except the certificate issue in LP: #1930763
3. Scale up 1 keystone unit
juju add-unit -n 1 keystone
4. Remove the keystone unit with VIP
juju remove-unit keystone/28
After that, the cluster status become "blcoked"
```
Unit Workload Agent Machine Public address Ports Message
keystone/29 blocked idle 122 10.5.2.26 5000/tcp Database not initialised
keystone-hacluster/18 active idle 10.5.2.26 Unit is ready and clustered
keystone-mysql-router/50 active idle 10.5.2.26 Unit is ready
keystone/30* blocked idle 123 10.5.3.123 5000/tcp Database not initialised
keystone-hacluster/17 active idle 10.5.3.123 Unit is ready and clustered
keystone-mysql-router/49* active idle 10.5.3.123 Unit is ready
keystone/31 blocked idle 124 10.5.2.152 5000/tcp Database not initialised
keystone-hacluster/19* active idle 10.5.2.152 Unit is ready and clustered
keystone-mysql-router/51 active idle 10.5.2.152 Unit is ready
```
Detail Information
=================
I find related log in keystone/29.
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 INFO unit.keystone/29.juju-log server.go:314 Keystone charm unit not ready - deferring identity-relation updates
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Checking for maintenance notifications
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 This unit (keystone/29) is in allowed unit list from keystone-mysql-router/50
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 db-initialised key missing, assuming db is not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database initialised: False
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Database not initialised
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peers this unit is: NOTREADY
2021-09-01 03:28:42 DEBUG unit.keystone/29.juju-log server.go:314 Telling peer behind relation cluster:311 that keystone/29 is NOTREADY
At the meantime, one of the mysql-innodb-cluster unit become error.
This error can be fixed after this command
juju resolve --no-retry mysql-innodb-cluster/4
But keystone is still stay in blocked status.
Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/3 active idle 69 10.5.3.213 Unit is ready: Mode: R/W, Cluster is ONLIN
E and can tolerate up to ONE failure. mysql-innodb-cluster/4 error idle 70 10.5.1.215 hook failed: "update-status"
mysql-innodb-cluster/5* active idle 71 10.5.1.138 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
I will attach juju crashdump later, it's too big to upload now.
Expected result
===============================
keystone cluster action complete and the unit is "active" state. |
|
| 2021-09-01 07:25:38 |
Eric Chen |
attachment added |
|
juju-crashdump-64d59817-537c-4110-bb74-dbff24a03386.tar.xz https://bugs.launchpad.net/charm-keystone/+bug/1942289/+attachment/5522137/+files/juju-crashdump-64d59817-537c-4110-bb74-dbff24a03386.tar.xz |
|
| 2021-09-01 13:07:24 |
Aurelien Lourot |
charm-keystone: status |
New |
Triaged |
|
| 2021-09-01 13:07:26 |
Aurelien Lourot |
charm-keystone: importance |
Undecided |
Medium |
|
| 2021-09-27 03:01:11 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
| 2022-04-07 21:39:46 |
Garrett Neugent |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2022-04-11 03:44:11 |
Xav Paice |
bug task added |
|
juju |
|
| 2022-04-11 07:29:59 |
Canonical Juju QA Bot |
juju: status |
New |
Invalid |
|
| 2023-03-27 11:48:09 |
Alex Kavanagh |
tags |
|
good-first-bug |
|
| 2023-10-03 15:32:28 |
Phan Trung Thanh |
charm-keystone: assignee |
|
Phan Trung Thanh (tphan025) |
|
| 2023-10-03 21:47:37 |
OpenStack Infra |
charm-keystone: status |
Triaged |
In Progress |
|
| 2024-04-23 14:29:53 |
Thobias Trevisan |
bug |
|
|
added subscriber Thobias Trevisan |
