/usr/bin/nova network-list executed before the 'nova' service user gets his role

[Attila Fazekas]

In a nova network setup packstack/puppet-nova tries to execute '/usr/bin/nova network-list' with the credentials of the nova services user.

The error message looks like:
ERROR : Error appeared during Puppet run: 192.168.1.13_controller.pp
Error: /Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]: Could not evaluate: Execution of '/usr/bin/nova network-list' returned 1: ERROR (Unauthorized): The request you have made requires authentication. (HTTP 401) (Request-ID: req-bc659daf-9ec7-4eb9-9c31-c398e84c703e)

According the network dump the following happened.

1. The nova user was created
2. The /usr/bin/nova network-list executed
3. The admin role is added to the nova user (~ 30 sec later)
4. I read my console and saw the above error.

So looks like the 'role' is assigned after the actual error situation happened.

So the client is most likely rejected by keystone because the service user did not had the role in time.

[YaZug (jon-schlueter)]

was able to get same error with "packstack --allinone --os-neutron-install n" had it fail and then run it again and it completed

[Alan Pevec (apevec)]

Any updates, can you push WIP gerrit review even if not finished yet?

[YaZug (jon-schlueter)]

This still re-produces with openstack-packstack-8.0.0-0.20160708221207.2522b5c.el7

[Ivan Chavero (imcsk8) (ichavero-ichavero)]

The openstack command was throwing a warning[1] to stdout, the puppet-openstackclient was not filtering this and was failing. Applying this two patches [2][3] fixes the issue in puppet.

Checking if the problem persists.

[1] WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
[2] https://review.openstack.org/#/c/343868/
[3] https://review.openstack.org/#/c/343877/

[Alan Pevec (apevec)]

There must be more, it reproduces with --os-neutron-install n and not with default Neutron enabled, which is tested in the Packstack upstream gate.

[Ivan Chavero (imcsk8) (ichavero-ichavero)]

I'm doing more tests, i also found the glance package is still changing the glance-api-paste.ini to
glance-api-dist-paste.ini and also found that the python-cotyledon package is not being installed (i checked the gnocchi spec file for RDO and its added as a dependency).

[Ivan Chavero (imcsk8) (ichavero-ichavero)]

I've managed to recreate the error, checking a fix

[Ivan Chavero (imcsk8) (ichavero-ichavero)]

Nova network is now deprecated, should we continue with this bug?

https://github.com/openstack/nova/blob/master/releasenotes/notes/deprecate_nova_network-093e937dcdb7fc57.yaml

[YaZug (jon-schlueter)]

Until it is removed from nova or packstack drops nova networking support all together this is a bug, it shouldn't be that difficult a fix, it's ordering issue from what I can tell.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/344664

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jon Schlueter (<email address hidden>) on branch: master
Review: https://review.openstack.org/344351
Reason: https://review.openstack.org/#/c/344664/ provides more targeted solution for this particular problem

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/344664
Committed: https://git.openstack.org/cgit/openstack/packstack/commit/?id=ae579f64869637dbc280eb02ed1dfa428b4730ad
Submitter: Jenkins
Branch: master

commit ae579f64869637dbc280eb02ed1dfa428b4730ad
Author: Ivan Chavero <email address hidden>
Date: Wed Jul 20 02:12:00 2016 -0600

    Fix user creation for nova network

    When nova network is enabled Packstack fails because
    the nova user is not yet created while checking the
    list of networks.

    Change-Id: Iac66addbfce1ca27f7a10be106d16a3c102c2540
    Closes-bug: #1602675