Setting release file causes cloud-init failure to launch

[Eric Miller]

Kolla-Ansible 8.0.1 Stein installation on CentOS

We ran into an issue where Ubuntu images would not launch Cloud-Init when deployed. This started after we added the file:

/etc/nova/release

with the contents:

[Nova]
vendor = Company
product = Company Cloud
package = ""

This was specified in the Kolla configuration file in /etc/kolla/config/release, which is copied to /etc/nova/release on all compute nodes.

All images deployed fine previously, which have the appropriate os_distro and os_type to "ubuntu" and "linux", respectively.

Deployment testing is pretty simple, such as:

openstack server create --image 72270019-df0d-4d7e-a90c-dbc1d64b98db --network TestNetwork001 --flavor t5sd.4xlarge --key-name testkey --security-group AllowSSHInbound TestUbuntuServer001

After adding a floating IP, we tested connecting via SSH. The connection failed.

After removing all items from the "release" file above, so only the stanza [Nova] existed, and re-deploying Kolla-Ansible, verifying that the empty release file was installed on all compute nodes, and finally restarting all Nova containers, the problem goes away.

Eric

[Mark Goddard (mgoddard)]

Have you checked cloud-init logs?

[Eric Miller (erickmiller)]

Hi Mark,

I did not, since I could not login. The console logs showed that Cloud-Init never ran.

However, I am re-running the test with a volume-based boot drive and will try again, then mount that drive to a different VM to get to the logs in case there is evidence as to why it didn't run. I'll be back.

[Eric Miller (erickmiller)]

The /var/log/cloud-init.log file does not exist. Cloud-init is definitely installed. I used a standard Ubuntu 18.04 cloud image which has cloud-init pre-installed, and all of the cloud-init files are on the filesystem, including /usr/bin/cloud-init.

[Mark Goddard (mgoddard)]

That's quite strange. Given that the release file is accessed via metadata, I'd expect cloud-init to have run. Perhaps the error is on the nova side?

[Eric Miller (erickmiller)]

What is more bizarre is that CentOS images work fine, while Ubuntu images fail to run cloud-init.

[Dincer Celik (dincercelik)]

I've tried to reproduce this on my environments which run stein on ubuntu and couldn't face the same issue. I used the official cloud images of centos 6/7 and ubuntu 16.04/18.04.

@Eric, which type of images are you using for kolla? If they are binary, do you have chance to try source images for nova?

[Dincer Celik (dincercelik)]

Also, please try to push a cloud-config data to create a user for debugging cloud-init logs. This will also prove whether cloud-init has been run.

#cloud-config

users:
  - default
  - name: test
    plain_text_passwd: "test123"
    home: /home/test
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    lock_passwd: False

[Sylvain Bauza (sylvain-bauza)]

Honestly, I don't know what to say from a Nova side if cloud-init isn't correctly running on the guest.

The only interaction that would impact Nova would be the metadata service, which could be unavailable, or, or... but if the guest just doesn't run cloud-init and with other official images it does work properly, then I don't think this is a Nova problem.

Moving the bug to Opinion for the Nova project, but please change the status back to New once you find evidence that Nova is messing up with metadata or whatever else but not the guest image.