nova_libvirt: Error reading secret: internal error: invalid base64 with Windows line endings in Ceph keyrings

[Gerald Schneider]

My nova_libvirt container can't read the secrets.

user@compute01:~$ docker exec nova_libvirt virsh secret-list
 UUID Usage
---------------

The files themselves are inside the container:

user@compute01:~$ docker exec nova_libvirt ls -l /etc/libvirt/secrets/
total 16
-rw------- 1 root root 41 Mar 12 11:09 9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64
-rw------- 1 root root 170 Mar 12 11:09 9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.xml
-rw------- 1 root root 41 Mar 12 11:09 e2a1f3fb-c212-42c3-b3b0-49348c6971ef.base64
-rw------- 1 root root 168 Mar 12 11:09 e2a1f3fb-c212-42c3-b3b0-49348c6971ef.xml

Error messages from the logs:

user@compute01:~# tail /var/log/kolla/libvirt/libvirtd.log
2020-03-12 10:09:03.314+0000: 24841: error : virSecretLoadValue:878 : internal error: invalid base64 in '/etc/libvirt/secrets/e2a1f3fb-c212-42c3-b3b0-49348c6971ef.base64'
2020-03-12 10:09:03.314+0000: 24841: error : virSecretLoadAllConfigs:960 : Error reading secret: internal error: invalid base64 in '/etc/libvirt/secrets/e2a1f3fb-c212-42c3-b3b0-49348c6971ef.base64'
2020-03-12 10:09:03.314+0000: 24841: error : virSecretLoadValue:878 : internal error: invalid base64 in '/etc/libvirt/secrets/9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64'
2020-03-12 10:09:03.314+0000: 24841: error : virSecretLoadAllConfigs:960 : Error reading secret: internal error: invalid base64 in '/etc/libvirt/secrets/9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64'

As far as I can tell the .base64 files contain the same string as the `key =` value in the corresponding .keyring file.

I'm using the as of now current images:

user@compute01:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kolla/ubuntu-binary-nova-compute train 36fdde8be0b4 30 hours ago 919MB
kolla/ubuntu-binary-neutron-openvswitch-agent train 3f116af323a0 30 hours ago 574MB
kolla/ubuntu-binary-nova-ssh train b4e4d123a4ff 30 hours ago 657MB
kolla/ubuntu-binary-kolla-toolbox train 7b99730386b4 30 hours ago 916MB
kolla/ubuntu-binary-openvswitch-db-server train b7b8c14cc27d 30 hours ago 251MB
kolla/ubuntu-binary-openvswitch-vswitchd train 8962d7724f49 30 hours ago 251MB
kolla/ubuntu-binary-fluentd train c0b1ad62df58 30 hours ago 564MB
kolla/ubuntu-binary-nova-libvirt train fc3e1720212f 30 hours ago 795MB
kolla/ubuntu-binary-cron train d1e2d9326c5f 30 hours ago 235MB

**Environment**:
* OS (e.g. from /etc/os-release): Ubuntu 18.04.4 LTS
* Kernel (e.g. `uname -a`): Linux compute01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
* Docker version if applicable (e.g. `docker version`): 19.03.8
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): pip 9.0.1
* Docker image Install type (source/binary): binary
* Docker image distribution: ubuntu
* Are you using official images from Docker Hub or self built? official
* If self built - Kolla version and environment used to build:

[Mark Goddard (mgoddard)]

Hi, have you tried using the 'base64' command to verify that the files are not valid? Also check the values in the keyring files?

[Mark Goddard (mgoddard)]

Also, are you using kolla ceph or external ceph?

[Gerald Schneider (geschnei)]

Yes I have tried base64, it doesn't show any error:

(nova-libvirt)[root@compute01 secrets]# base64 -d 9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64

vq=^  ~4
ч(nova-libvirt)[root@compute01 secrets]#

I'm using external ceph. cinder and glance can connect to it, I can create images and volumes.

[Mark Goddard (mgoddard)]

Strange. Perhaps it's a permissions issue? Can the files be read by the nova user?

[Gerald Schneider (geschnei)]

libvirtd runs as root in the docker container. The files are owned by root. Just to make sure I ran `chmod 644 /etc/libvirt/secrets/*`, but the result is the same.

[Gerald Schneider (geschnei)]

Yesterday I ran `kolla-ansible destroy`, `git pull` on kolla-ansible, `kolla-ansible pull` and deployed the whole cluster again to restart fresh.

Currently running `base64 -d` inside the container fails:

(nova-libvirt)[root@compute01 /]# base64 -d /etc/libvirt/secrets/9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64

vq=^  ~4
чbase64: invalid input

Checking the file contents of that file I get:

root@compute01:~# file /etc/kolla/nova-libvirt/secrets/9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64
/etc/kolla/nova-libvirt/secrets/9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64: ASCII text, with CR line terminators

On the production cluster this command shows only `ASCII text`, without any kind of line terminators.

I removed the line endings in earlier trials, which resulted in a working `base64 -d`, but libvirtd still complained about the invalid bas64.

For now I'm just leaving it as it is.

When I open the file with vi and copy the string inside (without the line ending) I can successfully decode it again.

root@compute01:~# vi /etc/kolla/nova-libvirt/secrets/9d4dacc6-9c75-49cb-bda2-c2e5e14f8e95.base64
root@compute01:~# echo "AQB2cT1eAAAAABAAfqCQkjSdB5ocsICE4tGH2g==" |base64 -d

vq=^  ~4
чroot@compute01:~#

So, maybe the line endings are a problem?

[Mark Goddard (mgoddard)]

Strange - windows line endings in the base64 file. Do your ceph keyring files have CR line endings?

[Gerald Schneider (geschnei)]

Well, that's a little embarrassing. I actually investigated the line endings earlier before posting here, but seems like I forgot to correct the line endings in the nova keyring files of all things.

After recreating the keyring files (using vi on the kolla machine this time) and running reconfigure nova virtlib can load the secrets and it works.

You can consider this "bug" solved. On the other hand, this could be something that could be corrected in the regex that is used to create the secret files in ansible.

[Mark Goddard (mgoddard)]

Glad we got to the bottom of it. I wonder if we could catch it somehow. There is the 'validate' parameter to the copy module, we could validate that the keyring files do not have windows line endings when copying into place. Or we could add a precheck. Is that something you would have time to look at?