neutron-openvswitch charm errors after node reboots

[Gema Gomez]

Neutron-openvswitch charm errors every time after a node reboots:
         neutron-openvswitch/2:
            upgrading-from: local:trusty/neutron-openvswitch-0
            agent-state: error
            agent-state-info: 'hook failed: "config-changed"'
            agent-version: 1.20.14

I have a deployment done with: http://bazaar.launchpad.net/~openstack-charm-testers/+junk/ceilometer-ha/view/head:/default-mongo-ha-only.yaml

Where lp:charms/trusty/neutron-openvswitch errors every time I reboot the compute nodes.

Please, find sos report from a VM displaying this behaviour here: http://people.canonical.com/~gemagomez/sosreport-gg-openvswitch.tar.xz

[Mario Splivalo (mariosplivalo)]

Hi, Gema.

I'm failing to reproduce this - constant rebooting of VMs only lead me to induced/forced mysql and keystone failures, but ceilometer and openvswitch units work fine.

As explained in lp1423961, I will try this at my local maas/openstack deploy and will post results here.

[Scott Pendleton (scott.pendleton)]

I also have this issue.

When I try to resolve --retry I get this

neutron-openvswitch/0 maintenance executing 1.25.6 plabs-os-cn2.maas (config-changed) Installing apt packages

and then it goes back to

neutron-openvswitch/0 error idle 1.25.6 plabs-os-cn2.maas hook failed: "config-changed"

It is definitely impacting the system and not just a notification error as my openvswitch agents on the compute nodes don't appear to be communicating with agent on the neutron-gateway and as a result none of the tap interfaces getting created. None of the instances can reach the router. Everything was functional prior to the reboot of the compute node.

I also get nothing in the openvswtich-agent.log after that happens. I don't know if agent causing the hook to fail or the failure to hook into is causing it to be misconfigured.

[Scott Pendleton (scott.pendleton)]

While doing a hook debug I get this when I run neutron_ovs_hooks.py

Traceback (most recent call last):
  File "./hooks/neutron_ovs_hooks.py", line 155, in <module>
    main()
  File "./hooks/neutron_ovs_hooks.py", line 151, in main
    assess_status(CONFIGS)
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/neutron_ovs_utils.py", line 585, in assess_status
    assess_status_func(configs)()
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1766, in _assess_status_func
    state, message = _determine_os_workload_status(*args, **kwargs)
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1166, in _determine_os_workload_status
    configs, required_interfaces)
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1220, in _ows_check_generic_interfaces
    required_interfaces)
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1501, in incomplete_relation_data
    complete_ctxts = configs.complete_contexts()
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 293, in complete_contexts
    for i in six.itervalues(self.templates)]
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 118, in complete_contexts
    self.context()
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 103, in context
    _ctxt = context()
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/context.py", line 1354, in __call__
    mappings = super(PhyNICMTUContext, self).__call__()
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/context.py", line 1333, in __call__
    portmap = parse_data_port_mappings(ports)
  File "/var/lib/juju/agents/unit-neutron-openvswitch-0/charm/hooks/charmhelpers/contrib/openstack/neutron.py", line 355, in parse_data_port_mappings
    _mappings = {mappings.split()[0]: default_bridge}
IndexError: list index out of range

[Launchpad Janitor (janitor)]

[Expired for OpenStack neutron-openvswitch charm because there has been no activity for 60 days.]