Nova Doesn't Set Instance Passwords

[Boris Lukashev]

I've filed bugs against two deployment stacks regarding this(https://bugs.launchpad.net/charm-nova-compute/+bug/1933057 and https://bugs.launchpad.net/kolla-ansible/+bug/1942654), and it was suggested that i file here as well since this appears unrelated to the manner in which openstack is deployed.
Using the various deployment mechanisms, i've found that in Wallaby with libvirt+kvm compute services and OVN networking, instance passwords are never set. Passwords can not be set by setting them in Horizon, in the CLI, nor are they randomly set when not set by the operator in any way - resulting in a security issue on Windows images (and others) which have default passwords baked in under the expectation that they will be reset by cloud-init at the first boot to something other than the well known default of the image (https://owasp.org/www-community/vulnerabilities/Use_of_hard-coded_password). The instance metadata password URI is always empty. Userdata however can be used to set passwords at boot time, though this requires manual intervention to create the relevant userdata by the operator which still leaves that hard coded password vuln in play.
Current openstack deployment is Kolla-Ansible 12.2, but this happened with the Juju stack as well, so its probably a Wallaby (or prior recent release) issue.

[Balazs Gibizer (balazs-gibizer)]

@Boris: Thanks for reporting the bug!

I think I can reproduce the problem with devstack on master. At least the "The instance metadata password URI is always empty." part of it. The metadata service looks into the instance.system_metadata for password [1]. But simply booting an instance does not store the password there.

I also checked the what happens if config drive is used instead, e.g. creating a server with --use-config-drive flag in the openstack CLI. Then I can see that the admin password I provided on at boot ends up stored in the meta_data.json on the config drive.

Do you have info about an OpenStack version where the password via the metadata service worked?

[1] https://github.com/openstack/nova/blob/402fe188b4e7ff76109e8a5ea1f24a5e915eaa09/nova/api/metadata/password.py#L37

[Boris Lukashev (rageltman)]

Thanks for looking into this @Balazs. Both Liberty and Mitaka worked for the metadata service, the userdata, and the config drive mechanism.

[Balazs Gibizer (balazs-gibizer)]

I dig through Mitaka and Liberty codepath that is providing password to the metadata service and it looks very similar to what we have on master. The logic was [1] in Liberty too. So I also dig instance.system_metadata usage that supposed to store password in sysmeta. But I found nothing in Liberty / Mitaka that would actually store a password there (except for xen virt driver but you are using libvirt).

I have only found one somewhat related commit [2] in the whole history. But this is actually fixing password storage in metadata if the password is explicitly changes via changePassword server action API call[3].

Honestly I'm stuck. I cannot really create a mitaka devstack any more as it is a so old release. And I cannot find any definite code that changed around this since.

I hope others will have ideas.

[1] https://github.com/openstack/nova/blob/402fe188b4e7ff76109e8a5ea1f24a5e915eaa09/nova/api/metadata/password.py#L37
[2] https://review.opendev.org/c/openstack/nova/+/543032
[3] https://docs.openstack.org/api-ref/compute/#change-administrative-password-changepassword-action

[Boris Lukashev (rageltman)]

So i have Mitaka and Liberty clouds still live (modified over time, obviously). If there's anything you want run in them (non-destructive, these are prod), happy to try and get relevant data/outputs. Our older clouds were Fuel-based by the way, might be worth seeing if Mirantis made any changes to Nova/Neutron for this.
My initial assumption was that this was the same sort of mess that we've got with VPNaaS in OVN - no place anymore to run some function/service/agent responsible for delivering that content to the appropriate endpoints.

[Balazs Gibizer (balazs-gibizer)]

Thanks for offer. As a first step it would be nice to see the following:

0) turn on the debug logging for the nova-metadata service
1) create an instance with an admin password
2) check that the admin password is visible from inside the guest via the metadata service. Provide the logs from the metadata service for this action.
3) dump the content of the instance_system_metadata table of the nova database for the uuid of the instance you created at #1)
    select * from instance_system_metadata where instance_uuid='<instance uuid>';
Feel free to sanitize the value column of the output before posting it if it contains sensitive information.

Also if you think Mirantis delivered customized Nova for you then look for any changes that might be relevant in nova/api/metadata/ directory compared to mitaka-eol[1] or liberty-eol tags upstream. If not then you can widen the search of changes in the whole nova codebase.

[1] https://github.com/openstack/nova/tree/mitaka-eol/nova/api/metadata

[Boris Lukashev (rageltman)]

Horizon didn't have the option to actually set a password back then (see attached), but the CLI did take an `--admin-pass` param.
Will mess with a compute node tonight or over the weekend to minimize chance of impact on anything actually in there, should have that output for you shortly.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]