What happened:
I upgraded my Train test cluster to Ussuri following these instructions:
https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure
The upgrade completed successfully with no failures, and the existing VMs are fine, but new VM build fails with rados.Rados.connect\nrados.PermissionDeniedError:
http://paste.openstack.org/show/805424/
I'm running external ceph so I looked at this document:
OpenStack Docs: External Ceph<https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html>
It says that I need the following in /etc/kolla/config/glance/ceph.conf:
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
I didn't have that, so I added it and then redeployed, but still can't build VMs. I tried adding the same to all copies of ceph.conf and redeployed again, but that didn't help. I see some cryptic talk about ceph in the release notes but it's not obvious what I'm being asked to change:
https://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes
I read the bug that it refers to:
https://bugs.launchpad.net/kolla-ansible/+bug/1904062
But I already have "backend_host=rbd:volumes" so I don't think I'm hitting that.
Also I read these sections but I don't see anything obvious here that needs to be changed. My config files are in the standard locations.
* For cinder (cinder-volume and cinder-backup), glance-api and manila keyrings behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (ceph.*), instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in /etc/kolla/config do not match default values for introduced variables.
* The default behavior for generating the cinder.conf template has changed. An rbd-1 section will be generated when external Ceph functionality is used, i.e. cinder_backend_ceph is set to true. Previously it was only included when Kolla Ansible internal Ceph deployment mechanism was used.
* The rbd section of nova.conf for nova-compute is now generated when nova_backend is set to "rbd". Previously it was only generated when both enable_ceph was "yes" and nova_backend was set to "rbd".
My ceph keys have the default name and are in the default locations. I have cinder_backend_ceph: "yes". I don't have a nova_backend setting but I have nova_backend_ceph: "yes"
I added nova_backend: "rbd" and redeployed and now I get a different error: rados.Rados.connect\nrados.ObjectNotFound
http://paste.openstack.org/show/805425/
What I expected to happen: VMs build without errors after upgrade
How to reproduce it:
Install kolla-ansible Train on Centos 7. Upgrade Centos 7 to 8, upgrade Train to Ussuri, then build a VM.
Environment:
[root@chrnc-void-testupgrade-control-1-replace ~]# cat /etc/*rele*
CentOS Linux release 8.2.2004 (Core)
Derived from Red Hat Enterprise Linux 8.2 (Source)
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"
CentOS Linux release 8.2.2004 (Core)
CentOS Linux release 8.2.2004 (Core)
cpe:/o:centos:centos:8
[root@chrnc-void-testupgrade-control-1-replace ~]# uname -a
Linux chrnc-void-testupgrade-control-1-replace.dev.chtrse.com 4.18.0-193.6.3.el8_2.x86_64 #1 SMP Wed Jun 10 11:09:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@chrnc-void-testupgrade-control-1-replace ~]# docker version
Client: Docker Engine - Community
Version: 20.10.6
API version: 1.41
Go version: go1.13.15
Git commit: 370c289
Built: Fri Apr 9 22:45:33 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.6
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8728dd2
Built: Fri Apr 9 22:43:57 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Kolla-Ansible version: Ussuri
kolla_install_type: "source"
Official images
(openstack) [root@chrnc-void-testupgrade-build ~]# grep -v ^# /etc/kolla/globals.yml|grep -v ^$
---
config_strategy: "COPY_ALWAYS"
virtualenv: /opt/kolla/venv
virtualenv_site_packages: yes
kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "train"
node_custom_config: "/etc/kolla/config"
kolla_internal_vip_address: "172.16.0.100"
network_interface: "eth0"
kolla_external_vip_interface: "eth0"
neutron_external_interface: "eth1"
neutron_plugin_agent: "openvswitch"
keepalived_virtual_router_id: "51"
kolla_enable_tls_internal: "no"
openstack_region_name: "chrnc-void-testupgrade"
multiple_regions_names: ["{{ openstack_region_name }}"]
enable_openstack_core: "yes"
rabbitmq_use_3_7_24_on_centos7: true
elasticsearch_use_v6: true
kibana_use_v6: true
enable_central_logging: "yes"
enable_ceph: "no"
enable_chrony: "no"
enable_cinder: "yes"
enable_cinder_backup: "yes"
enable_fluentd: "yes"
enable_grafana: "no"
enable_mariabackup: "yes"
enable_masakari: "yes"
enable_neutron_agent_ha: "no"
enable_neutron_bgp_dragent: "yes"
enable_neutron_provider_networks: "yes"
enable_prometheus: "yes"
rabbitmq_server_additional_erl_args: "+S 1:1"
external_ceph_cephx_enabled: "yes"
glance_backend_ceph: "yes"
glance_backend_file: "no"
glance_backend_swift: "no"
glance_backend_vmware: "no"
cinder_backend_ceph: "yes"
cinder_backup_driver: "ceph"
nova_backend_ceph: "yes"
nova_compute_virt_type: "qemu"
nova_safety_upgrade: "yes"
(openstack) [root@chrnc-void-testupgrade-build glance]# cat /etc/kolla/config/glance/ceph.conf
# Please do not change this file directly since it is managed by Ansible and will be overwritten
[global]
cluster network = 172.16.0.0/22
fsid = 30c52736-0f41-4bfc-a5a6-90657bb3315d
mon host = [v2:172.16.3.111:3300,v1:172.16.3.111:6789],[v2:172.16.1.89:3300,v1:172.16.1.89:6789],[v2:172.16.0.213:3300,v1:172.16.0.213:6789]
mon initial members = chrnc-void-testupgrade-ceph-1,chrnc-void-testupgrade-ceph-2,chrnc-void-testupgrade-ceph-0
osd pool default crush rule = -1
public network = 172.16.0.0/22
[osd]
osd memory target = 11037206118
(openstack) [root@chrnc-void-testupgrade-build glance]# ssh ceph0 "ceph status"
cluster:
id: 30c52736-0f41-4bfc-a5a6-90657bb3315d
health: HEALTH_OK
services:
mon: 3 daemons, quorum chrnc-void-testupgrade-ceph-0,chrnc-void-testupgrade-ceph-2,chrnc-void-testupgrade-ceph-1 (age 4w)
mgr: chrnc-void-testupgrade-ceph-1(active, since 4w), standbys: chrnc-void-testupgrade-ceph-0, chrnc-void-testupgrade-ceph-2
osd: 3 osds: 3 up (since 4w), 3 in (since 4w)
data:
pools: 3 pools, 48 pgs
objects: 2.05k objects, 2.9 GiB
usage: 6.1 GiB used, 24 GiB / 30 GiB avail
pgs: 48 active+clean
Hi Albert. To verify that the keys in the containers are valid you could docker exec into nova_compute & cinder_volume, and try running some ceph commands using each of the keys.