bootstrap-servers failed to enable ceph apt repository

[Yushawn Tang]

This might be the complaint of release file of the ceph repository. Both pacific and quincy releases seem not having that file.

The environments use external ceph storage and didn't have this problem two weeks ago around 7/14/2023 though nothing in the environment was changed since then.

The compute nodes that have this problem are all running Ubuntu 22.04 upgraded recently. Manually running apt-add-repository produces the similar message but the ceph repo can be successfully added to the repo conf file on the nodes nonetheless.

bootstrap-servers error message
=============================================

fatal: [kao11.cec.edu]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "codename": null,
            "filename": "ceph",
            "install_python_apt": true,
            "mode": null,
            "repo": "deb https://download.ceph.com/debian-pacific/ jammy main",
            "state": "present",
            "update_cache": true,
            "update_cache_retries": 5,
            "update_cache_retry_max_delay": 12,
            "validate_certs": true
        }
    },
    "msg": "Failed to update apt cache: E:The repository 'https://download.ceph.com/debian-pacific jammy Release' does not have a Release file., W:Updating from such a repository can't be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., W:https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details."
}

manually adding ceph repository
=================================================

$ sudo apt-add-repository 'deb https://download.ceph.com/debian-pacific/ jammy main'
Repository: 'deb https://download.ceph.com/debian-pacific/ jammy main'
Description:
Archive for codename: jammy components: main
More info: https://download.ceph.com/debian-pacific/
Adding repository.
Press [ENTER] to continue or Ctrl-c to cancel.
Adding deb entry to /etc/apt/sources.list.d/archive_uri-https_download_ceph_com_debian-pacific_-jammy.list
Adding disabled deb-src entry to /etc/apt/sources.list.d/archive_uri-https_download_ceph_com_debian-pacific_-jammy.list
Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease
Ign:2 https://download.ceph.com/debian-pacific jammy InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Err:4 https://download.ceph.com/debian-pacific jammy Release
  404 Not Found [IP: 158.69.68.124 443]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [859 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [209 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [668 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [961 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [208 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [21.4 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [650 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [152 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [768 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [139 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.3 kB]
Reading package lists... Done
E: The repository 'https://download.ceph.com/debian-pacific jammy Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

[Yushawn Tang (galaxyling)]

This is likely missing ceph repositories for Ubuntu 22.04.

[Yushawn Tang (galaxyling)]

This problem is more likely the ceph repository problem for ubuntu 22.04 and it happens to affect our environment, that is ubuntu 22.04 nodes, external ceph storage and zun enabled.

Also in our /etc/kolla/globals.yml this was enabled according to the document of kolla-ansible 16.1.0 external ceph.

zun_configure_for_cinder_ceph: "yes"

I figure we don't need that feature and turned it off. Bootstrap-servers was successful.