acceptance: neutron fails to start server service

[Emilien Macchi]

This is a new error that happened very lately, using RDO liberty packaging:

With current state of beaker manifests, we have this error:
No providers specified for 'LOADBALANCER' service, exiting

Source: http://logs.openstack.org/50/216950/5/check/gate-puppet-neutron-puppet-beaker-rspec-dsvm-centos7/9e7e510/logs/neutron/server.txt.gz#_2015-09-01_12_40_22_734

That means neutron-server can't start correctly.

This is probably a misconfiguration in our manifests or a packaging issue in Neutron, because we don't have the issue in Trusty jobs.

RDO packaging version: 7.0.0.0b3-dev606

[Emilien Macchi (emilienm)]

I need to dig more but it would be an orchestration issue where neutron server tries to start while neutron_lbaas.conf file (pvovided by neutron lbaas) is not here yet.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/219393

[Emilien Macchi (emilienm)]

having 2015-09-02 04:04:31.531 | openvswitch-switch : Depends: init-system-helpers (>= 1.18~) but 1.14 is to be installed

on trusty...

[Javier Peña (jpena-c)]

In RDO, I have found a workaround to start the service by doing this:

ln -s /etc/neutron/neutron_lbaas.conf /etc/neutron/conf.d/neutron-server/neutron_lbaas.conf

After that, neutron-server can start.

This is definitely a packaging issue in the RDO side, that should work out of the box.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/219393
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=ef7de8d4cd947df94823428043d316d584ca2ea9
Submitter: Jenkins
Branch: master

commit ef7de8d4cd947df94823428043d316d584ca2ea9
Author: Emilien Macchi <email address hidden>
Date: Tue Sep 1 14:32:18 2015 -0400

    acceptance: disable lbaas service plugin

    Neutron packaging seems having issues when LBaaS is enabled in
    service_plugins Neutron parameter.

    As a temporary workaround, we disable it so we can unblock our CI, but
    keep LBaaS service installed & working, so we continue the class
    testing.

    Once packaging is fixed, we will enable the service plugin again.

    It also drop the device_driver parameter which is already the default
    one in puppet-neutron.

    Partial-bug: #1490990
    Change-Id: Ibd90e5e159f9a1fbff3990241795894e0bdca3cc

[Ihar Hrachyshka (ihar-hrachyshka)]

No, it's not a packaging issue, it's a neutron bug.

Since https://review.openstack.org/#/c/202207/23/neutron/services/provider_configuration.py, neutron presumes that config-dir points to /etc/neutron. But oslo.config allows to use multiple config-dir options. F.e. Delorean uses this to allow users to configure their services by dropping a custom .conf file into a pre-designated service dir, instead of modifying stock neutron.conf and other files: https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-server.service#L8

So now, neutron fails to locate service conf file because config-dir points to the latest config-dir argument and not /etc/neutron.

[Ihar Hrachyshka (ihar-hrachyshka)]

I guess another project that is involved here is oslo.config since it does not allow to get all config-dirs passed to an executable, instead rewriting the single value on each --config-dir argument occurrence:

https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L1276

I believe oslo.config should provide a way to get the ordered list of all config dirs and files.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/220070

[Miguel Angel Ajo (mangelajo)]

I agree with ihar on #7, if oslo.config doesn't allow several config dirs, we shall fix it.

[Doug Hellmann (doug-hellmann)]

It's not clear multiple configuration directories were ever supported. The --config-file option can be repeated with multiple files, but the --config-dir option seems to want just one value, as indicated in the code and the help string. So this may be a misunderstanding of intent. It probably makes sense to support multiple directories, for the use case described here, so we should give some thought to how to do that. The docstring for ConfigDirAction makes me think it is expecting to be used more than once, so perhaps the setattr() can be replaced with some more complicated logic to extend the path. OTOH, that would change the type of the config_dir value in the namespace, so it might actually be better to add a new --config-path option that can be used to set multiple values and to eventually deprecate the current --config-dir option.

[Doug Hellmann (doug-hellmann)]

Note that although the directory names aren't available, repeating the --config-dir option does actually trigger parsing the config files discovered in each directory. Another approach is to store the config-path built up by repeating --config-dir using a different variable on the namespace. Of course the problem there is ensuring that we pick a unique attribute name that wouldn't be overwritten by some other option.

[Ihar Hrachyshka (ihar-hrachyshka)]

The reason why I adopted --config-dir for multiple directories is that in official docs:

http://docs.openstack.org/developer/oslo.config/cfg.html#loading-config-files

it's said: "The order of configuration files inside the same configuration directory is defined by the alphabetic sorting order of their file names." that made me think it supports multiple dirs, since it refers to 'same' dir.

I am fine with --config-path approach, though I believe we won't see it in L. So there without it, there are two ways to approach:

- reverting all config-dir usages from RDO and RHEL-OSP for Liberty. It's not clear how we maintain backwards compatibility though.
- in RDO/RHEL-OSP, moving config-dir /etc/neutron to the end of arguments list. This breaks the publicly announced ordering guarantee.
- revert neutron change for L and work with oslo.config team on --config-path feature in M, then adopt it in M.

I am not confident whether revert is a viable option, need to talk to @armax.

[Ihar Hrachyshka (ihar-hrachyshka)]

TripleO relies on it too: https://github.com/openstack/tripleo-image-elements/blob/master/elements/nova-api/install.d/nova-source-install/80-nova-api#L15

[Ihar Hrachyshka (ihar-hrachyshka)]

The feature was discussed in March and no one raised any concern: http://lists.openstack.org/pipermail/openstack-dev/2015-March/thread.html#59088

[Doug Hellmann (doug-hellmann)]

https://review.openstack.org/220210 demonstrates that oslo.config would merge multi-value options coming from different sources. Some other tests in that same test class demonstrate that options in the same group are merged properly, too. It's not clear why the neutron team thought they needed to build custom merging logic to handle this.

I'm left wondering exactly what's going on here. Can someone explain how the existing behavior in oslo.config didn't meet neutron's needs? Was this a matter of not realizing that the desired feature was already there, or is it an issue with the way it works not meeting neutron's needs? The ConfigOpts class doesn't report where it looked for those files, which is an oversight but shouldn't be necessary if we're relying on oslo.config to handle everything.

[Armando Migliaccio (armando-migliaccio)]

It's my understanding that the issue stems from the different ways options can passed to the CLI.

For instance, --config-dir is relied in distros like RDO, but it is not used in DevStack when loading the neutron server. Furthermore, when loading the server with extension modules (like lbaas, vpnaas, ...), their respective config files are not passed to the CLI at all. For this reason, today we rely on a custom logic to load the option from the *aas config file, but that existing logic is no longer meant to be used for merging the opts.

Bottom line, oslo.config rocks and there's nothing wrong with it. What we'll have to do is revise how Devstack handles the configuration for *-aas modules, and eventually get rid of this extra logic that was introduced when we split the services out of Neutron. Having said that...this is a Neutron matter.

HTH

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/220254

[Doug Hellmann (doug-hellmann)]

@Armando, it should be possible to either pass multiple config files on the command line, or to put the files in a directory together and pass that, or put them in separate directories and have both scanned. So maybe all that needs to change is how the service is being invoked? Let me know if you would like to talk through options. IRC or the ML may be better for that.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/220070

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220254
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f347939fd6c7b5a9e93af2007a0c01d00f29dc2b
Submitter: Jenkins
Branch: master

commit f347939fd6c7b5a9e93af2007a0c01d00f29dc2b
Author: armando-migliaccio <email address hidden>
Date: Thu Sep 3 10:29:12 2015 -0700

    Make sure service providers can be loaded correctly

    This patch fixes a regression where, if neutron was loaded using
    --config-dir, the service_providers option was no longer available.

    We bring the logic back (removed by 61121c5f2af), alongside the ability
    to load the option auto-magically. This is especially required for DevStack
    deployments as of today, because neutron-server is only loaded by passing
    --config-file (...)neutron.conf and --config-file (...)ml2_conf.ini

    Change-Id: I9bfaed9e19a5506e27795a0b7ad47f4c31fefa40
    Closes-bug: #1490990

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/220566

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220566
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=319c87e55126386505b5382118f30b579531318b
Submitter: Jenkins
Branch: master

commit 319c87e55126386505b5382118f30b579531318b
Author: Emilien Macchi <email address hidden>
Date: Fri Sep 4 11:50:51 2015 -0400

    acceptance: enable loadbalancer service_plugin

    Thanks to http://git.openstack.org/cgit/openstack/neutron/commit/?id=f4e980f7d5608b594e64d5ab78c594d61a664955
    Neutron should now be able to enable load loadbalancer service plugin in
    RDO packaging.

    This patch aims to re-enable the test again to validate Liberty can
    enable LBaaS service.

    Change-Id: I1375980cfefd5dd5bf1fc814907eb2746201d568
    Closes-Bug: #1490990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220210
Committed: https://git.openstack.org/cgit/openstack/oslo.config/commit/?id=b270375dd86f6fd2fdd2d0bb28eb43469fcbfb11
Submitter: Jenkins
Branch: master

commit b270375dd86f6fd2fdd2d0bb28eb43469fcbfb11
Author: Doug Hellmann <email address hidden>
Date: Thu Sep 3 15:52:08 2015 +0000

    Demonstrate that MultiStr values from multiple sources are merged

    Add a test to show that when we encounter the same MultiStrOption in
    several sources, the values are merged into one list.

    Related-Bug: #1490990
    Change-Id: Ie0324572191f084a3646d8bfb1d214d14221244d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

[OpenStack Infra (hudson-openstack)]

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334