neutron bootstrap container crashes when optional plugins are enabled

[Martin Matyáš]

Situation:
build from source, at least one optional neutron plugin enabled (sfc, lbaas) during build time as described here:
http://docs.openstack.org/developer/kolla/networking-guide.html#networking-sfc

means inserted sections
[neutron-server-plugin-networking-sfc]
or
[neutron-server-plugin-neutron-lbaas]

into /etc/kolla/kolla-build.conf

build proceeds fine, it seems that plugins are correctly baked into
/plugins
directory of neutron-server.

Then, attempt for deployment crashes on
TASK [neutron : Running Neutron bootstrap container] ***************************
fatal: [kolla-control-01 -> kolla-control-01]: FAILED! => {"changed": true, "failed": true, "msg": "Container exited with non-zero return code"}

docker logs bootstrap_neutron:
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Copying /var/lib/kolla/config_files/neutron.conf to /etc/neutron/neutron.conf
INFO:__main__:Setting permissions for /etc/neutron/neutron.conf
INFO:__main__:Copying /var/lib/kolla/config_files/neutron_lbaas.conf to /etc/neutron/neutron_lbaas.conf
INFO:__main__:Setting permissions for /etc/neutron/neutron_lbaas.conf
INFO:__main__:Copying /var/lib/kolla/config_files/neutron_vpnaas.conf to /etc/neutron/neutron_vpnaas.conf
INFO:__main__:Setting permissions for /etc/neutron/neutron_vpnaas.conf
INFO:__main__:Copying /var/lib/kolla/config_files/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini
INFO:__main__:Setting permissions for /etc/neutron/plugins/ml2/ml2_conf.ini
INFO:__main__:Writing out command to execute
Traceback (most recent call last):
  File "/var/lib/kolla/venv/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 686, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 205, in do_upgrade
    run_sanity_checks(config, revision)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 670, in run_sanity_checks
    script_dir.run_env()
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/alembic/script/base.py", line 407, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 24, in <module>
    from neutron.db.migration.models import head # noqa
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/migration/models/head.py", line 65, in <module>
    utils.import_modules_recursively(os.path.dirname(models.__file__))
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 865, in import_modules_recursively
    modules.extend(import_modules_recursively(dir_))
  File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 861, in import_modules_recursively
    importlib.import_module(module)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named setup

I tried to analyze, and as far as I understand that:
there seems to be really stupid bug in
https://github.com/openstack/neutron/blob/71f2d2bc90aaf9ee696cd4e4f29879ba6c5703b5/neutron/common/utils.py#L865

which should recursively import stuff, but full path not prepended prepend to recurse the subdir. Coincidentally, there is 'plugin' path to be recursed, and we have the same 'plugin' directory in root. So, our plugin directory is recursed instead. And as a consequence, setup.py is attempted from lbaas/sfc plugins to be imported, which of course fails.

[Martin Matyáš (martinx-maty)]

fixed in stable/newton and mester branches of neutron:
https://github.com/openstack/neutron/commit/676257cea09b736328caacc61182e27d369e254b