etcd3 driver missing in binary images
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| kolla |
Medium
|
Radosław Piliszek | ||
| Stein |
Medium
|
Radosław Piliszek | ||
| Train |
Medium
|
Radosław Piliszek | ||
| Ussuri |
Medium
|
Radosław Piliszek | ||
| kolla-ansible |
Medium
|
Radosław Piliszek | ||
| Stein |
Medium
|
Radosław Piliszek | ||
| Train |
Medium
|
Radosław Piliszek | ||
| Ussuri |
Medium
|
Radosław Piliszek |
Bug Description
O/S: Ubuntu 18.04
OpenStack: Stein
Deploy: Kolla-ansible with ceph
H/W: 1 deploy, 3 controller, 4 compute/storage with 7 disks for ceph
[What happened:]
After successful installation, cinder-volume and cinder-backup went down. If I restart the services using the docker restart command, there services on the horizon dashboard stays up for about 30 seconds and then goes back down.
Ceph has been deployed successfully, and Glance and Nova services that use ceph work fine.
[reproduce:] yes.
[Error Logs]
<dashboard log>
The following error occurred when creating a VM instance on Dashboard.
-------
Error: Failed to perform requested operation on instance "TVM", the instance has an error status: Please try again later [Error: Build of instance 65c3fe77-
-------
<cinder-volume log>
The following error occurred repeatedly for the cinder-volume service.
-------
2019-11-11 13:59:56.161 5416 INFO cinder.service [-] Starting cinder-volume node (version 14.0.1)
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.168 5416 ERROR oslo_service.
2019-11-11 13:59:56.172 5416 DEBUG oslo_concurrenc
2019-11-11 13:59:56.173 5416 DEBUG oslo_concurrenc
2019-11-11 13:59:56.177 6 INFO oslo_service.
2019-11-11 13:59:56.181 6 DEBUG oslo_service.
2019-11-11 13:59:56.186 5417 INFO cinder.se
-------
<cinder-backup log>
The following error occurred repeatedly for the cinder-backup service.
-------
2019-11-11 14:03:11.625 5797 INFO cinder.service [-] Starting cinder-backup node (version 14.0.1)
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.630 5797 ERROR oslo_service.
2019-11-11 14:03:11.633 5797 DEBUG oslo_concurrenc
2019-11-11 14:03:11.634 5797 DEBUG oslo_concurrenc
2019-11-11 14:03:11.641 6 INFO oslo_service.
2019-11-11 14:03:11.642 6 INFO oslo_service.
-------
<detail info about system>
(virtualenv) root@deploy:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https:/
SUPPORT_URL="https:/
BUG_REPORT_URL="https:/
PRIVACY_
VERSION_
UBUNTU_
(virtualenv) root@deploy:~# uname -a
Linux deploy 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
(virtualenv) root@deploy:~#
root@controller
Client: Docker Engine - Community
Version: 19.03.4
API version: 1.40
Go version: go1.12.10
Git commit: 9013bf583a
Built: Fri Oct 18 15:54:09 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.4
API version: 1.40 (minimum version 1.12)
Go version: go1.12.10
Git commit: 9013bf583a
Built: Fri Oct 18 15:52:40 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f
docker-init:
Version: 0.18.0
GitCommit: fec3683
root@controller
Kolla-ansible version: stein
Docker image Install type: binary
I am using official images from Docker hub.
global.yml, multinode files attached below.
BAHK MOON KEE (mkbahk) wrote : | #1 |
Radosław Piliszek (yoctozepto) wrote : | #3 |
This is the effect of enabling etcd - cinder coordination backend became etcd and it seems not available in the Ubuntu images.
Since you are using ceph, and hence not need coordination, as a workaround set:
cinder_
in globals.yml
description: | updated |
BAHK MOON KEE (mkbahk) wrote : | #4 |
Works fine. This was not a bug. The wrong setting was the cause. Thank you very much.
Radosław Piliszek (yoctozepto) wrote : | #5 |
I am glad it helped you.
OTOH, this is a bug - we need to ship etcd support in Ubuntu cinder images since we "support" etcd for coordination.
Radosław Piliszek (yoctozepto) wrote : | #6 |
It seems we don't have etcd3 in binary centos nor ubuntu images (nor debian for completeness). Hence it is broken with binary. Let's do what I planned to do - default to no coordination - less surprises.
no longer affects: | kolla/rocky |
no longer affects: | kolla/stein |
no longer affects: | kolla/train |
no longer affects: | kolla/ussuri |
Changed in kolla: | |
status: | New → Won't Fix |
Changed in kolla-ansible: | |
status: | New → Triaged |
importance: | Undecided → Medium |
assignee: | nobody → Radosław Piliszek (yoctozepto) |
no longer affects: | kolla/ussuri |
no longer affects: | kolla/train |
no longer affects: | kolla/stein |
Fix proposed to branch: master
Review: https:/
Changed in kolla-ansible: | |
status: | Triaged → In Progress |
description: | updated |
description: | updated |
Ok, so after a bit of discussion and a bit of investigation, I decided to fix the problem in a different way - we can include etcd3gw as etcd3 driver which seems more popular and can be included in binary builds. It also fixes the instability problems introduced by etcd3 - all the more reason to switch it.
summary: |
- cinder-volume, cinder-backup service down with CEPH + etcd3 driver missing in ubuntu binary |
Changed in kolla: | |
status: | Won't Fix → Triaged |
milestone: | none → 10.0.0 |
Fix proposed to branch: master
Review: https:/
Changed in kolla: | |
assignee: | nobody → Radosław Piliszek (yoctozepto) |
status: | Triaged → In Progress |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 0186c5e3cafeae0
Author: Radosław Piliszek <email address hidden>
Date: Tue Dec 3 14:28:26 2019 +0100
Install etcd3gw to fix Ubuntu binary tooz coordination
Change-Id: Ib56e62d1fb4d0f
Closes-bug: #1852086
Changed in kolla: | |
status: | In Progress → Fix Released |
Fix proposed to branch: stable/train
Review: https:/
Fix proposed to branch: stable/stein
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit 19c38d9c9d6416b
Author: Radosław Piliszek <email address hidden>
Date: Tue Dec 3 14:28:26 2019 +0100
Install etcd3gw to fix Ubuntu binary tooz coordination
Change-Id: Ib56e62d1fb4d0f
Closes-bug: #1852086
(cherry picked from commit 0186c5e3cafeae0
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 38191924fe04004
Author: Radosław Piliszek <email address hidden>
Date: Tue Dec 3 14:28:26 2019 +0100
Install etcd3gw to fix Ubuntu binary tooz coordination
Change-Id: Ib56e62d1fb4d0f
Closes-bug: #1852086
(cherry picked from commit 0186c5e3cafeae0
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 58b5acbf65013f4
Author: Radosław Piliszek <email address hidden>
Date: Fri Nov 15 09:38:43 2019 +0100
Default to etcd3gw driver for etcd-based coordination
To fix instability and availability issues:
etcd3 is not available in repos for binary kolla images.
etcd3 does not support eventlet-based services [1].
[1] https:/
Change-Id: I430bab735da204
Closes-bug: #1852086
Closes-bug: #1854932
Changed in kolla-ansible: | |
status: | In Progress → Fix Released |
Fix proposed to branch: stable/train
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit bfd1bde4e0ed1fe
Author: Radosław Piliszek <email address hidden>
Date: Fri Nov 15 09:38:43 2019 +0100
Default to etcd3gw driver for etcd-based coordination
To fix instability and availability issues:
etcd3 is not available in repos for binary kolla images.
etcd3 does not support eventlet-based services [1].
[1] https:/
Change-Id: I430bab735da204
Closes-bug: #1852086
Closes-bug: #1854932
(cherry picked from commit 58b5acbf65013f4
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit a0190747485bbb5
Author: Radosław Piliszek <email address hidden>
Date: Fri Nov 15 09:38:43 2019 +0100
Default to etcd3gw driver for etcd-based coordination
To fix instability and availability issues:
etcd3 is not available in repos for binary kolla images.
etcd3 does not support eventlet-based services [1].
[1] https:/
Change-Id: I430bab735da204
Closes-bug: #1852086
Closes-bug: #1854932
(cherry picked from commit 58b5acbf65013f4
(cherry picked from commit bfd1bde4e0ed1fe
This issue was fixed in the openstack/
This issue was fixed in the openstack/kolla 9.0.0.0rc2 release candidate.
Radosław Piliszek (yoctozepto) wrote : | #21 |
Changed subject as it affected centos binary too for the very same reason.
summary: |
- etcd3 driver missing in ubuntu binary + etcd3 driver missing in binary images |
Radosław Piliszek (yoctozepto) wrote : | #22 |
This issue was fixed in the openstack/
This issue was fixed in the openstack/kolla 8.0.2 release.
here is globals.yml also..