reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] with fake driver

[Matt Simonin]

I'm facing an issue when I use the reconfigure action, on this particular task :

TASK: [neutron | Restart the neutron_openvswitch_agent container] *************
skipping: [parasilo-28-kavlan-4.rennes.grid5000.fr]
skipping: [parasilo-8-kavlan-4.rennes.grid5000.fr]
fatal: [parasilo-5-kavlan-4.rennes.grid5000.fr] => error while evaluating conditional: openvswitch_agent_envs['KOLLA_CONFIG_STRATEGY'] != 'COPY_ONCE'

My guess is that the tests here
https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/neutron/tasks/do_reconfigure.yml#L237
and here
https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/do_reconfigure.yml#L138

should be the same.

Further information

# Parameters

* kolla code version : stable/mitaka

* with nova fake_driver

* globals.yml :
enable_heat: 'no'
enable_nova_fake: 'yes'
kolla_base_distro: centos
kolla_internal_vip_address: 10.24.63.249
network_interface: eth0
neutron_external_interface: eth1
num_nova_fake_per_node: 1
openstack_release: 2.0.1

* topology
'compute': [Host('parasilo-5-kavlan-4.rennes.grid5000.fr')], <- FAKE DRIVER
'control': [Host('parasilo-28-kavlan-4.rennes.grid5000.fr')],
'network': [Host('parasilo-8-kavlan-4.rennes.grid5000.fr')]},

# Step to reproduce
kolla-ansible deploy && kolla-ansible-reconfigure

# Some more debug

## add a debug just after the task [ neutron | Container config strategy for the neutron_openvswitch_agent container ]

to see who's is getting the var openvswitch_agent_envs filled :

https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/neutron/tasks/do_reconfigure.yml#L116

TASK: [neutron | Container config strategy for the neutron_openvswitch_agent container] ***
skipping: [parasilo-28-kavlan-4.rennes.grid5000.fr]
skipping: [parasilo-5-kavlan-4.rennes.grid5000.fr]
ok: [parasilo-8-kavlan-4.rennes.grid5000.fr]

-> only the network

## add a debug just before the task [neutron | Restart the neutron_openvswitch_agent container]
to see who's is trying to use the openvswitch_agent_envs

https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/neutron/tasks/do_reconfigure.yml#L229

-> It tries to use the openvswitch_agent_envs on the compute

TASK: [neutron | debug var=openvswitch_agent_envs] ****************************
ok: [parasilo-28-kavlan-4.rennes.grid5000.fr] => {
    "var": {
        "openvswitch_agent_envs": {
            "changed": false,
            "skipped": true
        }
    }
}
ok: [parasilo-5-kavlan-4.rennes.grid5000.fr] => {
    "var": {
        "openvswitch_agent_envs": {
            "changed": false,
            "skipped": true
        }
    }
}
ok: [parasilo-8-kavlan-4.rennes.grid5000.fr] => {
    "var": {
        "openvswitch_agent_envs": {
            "KOLLA_BASE_DISTRO": "centos",
            "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
            "KOLLA_INSTALL_METATYPE": "rdo",
            "KOLLA_INSTALL_TYPE": "binary",
            "KOLLA_SERVICE_NAME": "neutron-openvswitch-agent",
            "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "PS1": "$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ ",
            "changed": false,
            "invocation": {
                "module_args": "",
                "module_complex_args": {
                    "action": "get_container_env",
                    "name": "neutron_openvswitch_agent"
                },
                "module_name": "kolla_docker"
            }
        }
    }
}

TASK: [neutron | Restart the neutron_openvswitch_agent container] *************
skipping: [parasilo-28-kavlan-4.rennes.grid5000.fr]
skipping: [parasilo-8-kavlan-4.rennes.grid5000.fr]
fatal: [parasilo-5-kavlan-4.rennes.grid5000.fr] => error while evaluating conditional: openvswitch_agent_envs['KOLLA_CONFIG_STRATEGY'] != 'COPY_ONCE'

[Steven Dake (sdake)]

Matt,

I don't run into this but I'll ask another cr to see if they do.

Thanks for the report and analysis!

[Jeffrey Zhang (jeffrey4l)]

I haven't seen this issue, too. Could u post you whole inventory file out?

thanks.

[Matt Simonin (matthieu-simonin)]

From my side, I'm confirming the behaviour with stable/mitaka (99683643653f4859e47a157647463b6eed074b04).

My globals.yaml (fake_driver enabled)

enable_heat: 'no'
kolla_base_distro: centos
kolla_internal_vip_address: 10.8.63.249
network_interface: eth0
neutron_external_interface: eth1
enable_nova_fake: 'yes'
num_nova_fake_per_node: 2
openstack_release: 2.0.2

Here's also my inventory file (no storage):

[storage]
[control]
chimint-12-kavlan-4.lille.grid5000.fr ansible_ssh_user=root
[compute]
chimint-15-kavlan-4.lille.grid5000.fr ansible_ssh_user=root
chimint-7-kavlan-4.lille.grid5000.fr ansible_ssh_user=root
[network]
chimint-3-kavlan-4.lille.grid5000.fr ansible_ssh_user=root

# You can explicitly specify which hosts run each project by updating the
# groups in the sections below. Common services are grouped together.
[kibana:children]
control

[elasticsearch:children]
control

[haproxy:children]
network

[mariadb:children]
control

[rabbitmq:children]
control

[mongodb:children]
control

[keystone:children]
control

[glance:children]
control

[nova:children]
control

[neutron:children]
network

[cinder:children]
control

[memcached:children]
control

[horizon:children]
control

[swift:children]
control

[heat:children]
control

[murano:children]
control

[ironic:children]
control

[ceph-mon:children]
control

[ceph-rgw:children]
control

[ceph-osd:children]
storage

[magnum:children]
control

[mistral:children]
control

[manila:children]
control

[ceilometer:children]
control

# Additional control implemented here. These groups allow you to control which
# services run on which hosts at a per-service level.
#
# Word of caution: Some services are required to run on the same host to
# function appropriately. For example, neutron-metadata-agent must run on the
# same host as the l3-agent and (depending on configuration) the dhcp-agent.

# Glance
[glance-api:children]
glance

[glance-registry:children]
glance

# Nova
[nova-api:children]
nova

[nova-conductor:children]
nova

[nova-consoleauth:children]
nova

[nova-novncproxy:children]
nova

[nova-scheduler:children]
nova

[nova-spicehtml5proxy:children]
nova

[nova-compute-ironic:children]
nova

# Neutron
[neutron-server:children]
control

[neutron-dhcp-agent:children]
neutron

[neutron-l3-agent:children]
neutron

[neutron-lbaas-agent:children]
neutron

[neutron-metadata-agent:children]
neutron

# Cinder
[cinder-api:children]
cinder

[cinder-backup:children]
storage

[cinder-scheduler:children]
cinder

[cinder-volume:children]
storage

# iSCSI
[iscsid:children]
compute

[tgtd:children]
storage

# Manila
[manila-api:children]
manila

[manila-scheduler:children]
manila

[manila-share:children]
storage

# Swift
[swift-proxy-server:children]
swift

[swift-account-server:children]
storage

[swift-container-server:children]
storage

[swift-object-server:children]
storage

# Heat
[heat-api:children]
heat

[heat-api-cfn:children]
heat

[heat-engine:children]
heat

# Murano
[murano-api:children]
murano

[murano-engine:children]
murano

# Ironic
[ironic-api:children]
ironic

[ironic-conductor:children]
ironic

[ironic-inspector:children]
ironic

[ironic-pxe:children]
ironic

# Magnum
[magnum-api:childre...

[Steven Dake (sdake)]

please join us on IRC for a live debugging session on #openstack-kolla on freenode irc. We run 24 hrs a day. Please reference this bug when asking for help. We want to fix this bug for you, but we can't reproduce it and I believe its an environmental problem.

[Matt Simonin (matthieu-simonin)]

I'll join. Thank you.

[Steven Dake (sdake)]

Matt,

Our deadline for 3.0.0 is oct 18th, so if you could join IRC asap so we can investigate this (which may be an environmental problem), it would be appreciated. Note this is marked as critical for 3.0.0 (Newton).

Regards
-steve

[Matt Simonin (matthieu-simonin)]

Hi Steve,

I'll join for sure on Monday 17. morning (I'm UTC + 2).
Hopefully I'll be able to join earlier and get this solve for 3.0.0.

Best,

Matt

[Steven Dake (sdake)]

Hui would you mind looking at this:
http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-10-18.log.html#t2016-10-18T13:43:43

[Steven Dake (sdake)]

Used for evals, increasing priority.

[Matt Simonin (matthieu-simonin)]

To add some more feedbacks :

When using fake drivers on 2.0.3

- reconfigure report this error

When NOT using fake drivers on 2.0.3

- reconfigure is working well

[Jeffrey Zhang (jeffrey4l)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (OCATA, PIKE, QUEENS, ROCKY, ROCKY).
  Valid example: CONFIRMED FOR: OCATA