neutron-db-manage fails during OSA installation

Bug #1777655 reported by Maksim Malchuk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Maksim Malchuk

Bug Description

neutron-db-manage fails during OSA installation of AIO on 17.0.5 (stable/queens) on the TASK [os_neutron : Perform a DB expand] with error:

2018-06-18 19:25:22,535 p=24955 u=root | fatal: [aio1_neutron_server_container-1cdfa253]: FAILED! => {
    "changed": true,
    "cmd": [
        "/openstack/venvs/neutron-17.0.5/bin/neutron-db-manage",
        "upgrade",
        "--expand"
    ],
    "delta": "0:00:04.772890",
    "end": "2018-06-18 19:25:22.503808",
    "failed": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/openstack/venvs/neutron-17.0.5/bin/neutron-db-manage upgrade --expand",
            "_uses_shell": false,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2018-06-18 19:25:17.730918",
    "stderr": "Traceback (most recent call last):\n File \"/openstack/venvs/neutron-17.0.5/bin/neutron-db-manage\", line 11, in <module>\n sys.exit(main())\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/cli.py\", line 653, in main\n return_val |= bool(CONF.command.func(config, CONF.command.name))\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/cli.py\", line 172, in do_upgrade\n run_sanity_checks(config, revision)\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/cli.py\", line 637, in run_sanity_checks\n script_dir.run_env()\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/alembic/script/base.py\", line 425, in run_env\n util.load_python_file(self.dir, 'env.py')\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/alembic/util/pyfiles.py\", line 81, in load_python_file\n module = load_module_py(module_id, path)\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/alembic/util/compat.py\", line 141, in load_module_py\n mod = imp.load_source(module_id, path, fp)\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py\", line 24, in <module>\n from neutron.db.migration.models import head # noqa\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/models/head.py\", line 29, in <module>\n from neutron.db import agentschedulers_db # noqa\n File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py\", line 23, in <module>\n from neutron_lib.exceptions import dhcpagentscheduler as das_exc\nImportError: cannot import name dhcpagentscheduler",
    "stderr_lines": [
        "Traceback (most recent call last):",
        " File \"/openstack/venvs/neutron-17.0.5/bin/neutron-db-manage\", line 11, in <module>",
        " sys.exit(main())",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/cli.py\", line 653, in main",
        " return_val |= bool(CONF.command.func(config, CONF.command.name))",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/cli.py\", line 172, in do_upgrade",
        " run_sanity_checks(config, revision)",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/cli.py\", line 637, in run_sanity_checks",
        " script_dir.run_env()",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/alembic/script/base.py\", line 425, in run_env",
        " util.load_python_file(self.dir, 'env.py')",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/alembic/util/pyfiles.py\", line 81, in load_python_file",
        " module = load_module_py(module_id, path)",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/alembic/util/compat.py\", line 141, in load_module_py",
        " mod = imp.load_source(module_id, path, fp)",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py\", line 24, in <module>",
        " from neutron.db.migration.models import head # noqa",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/migration/models/head.py\", line 29, in <module>",
        " from neutron.db import agentschedulers_db # noqa",
        " File \"/openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py\", line 23, in <module>",
        " from neutron_lib.exceptions import dhcpagentscheduler as das_exc",
        "ImportError: cannot import name dhcpagentscheduler"
    ],
    "stdout": "",
    "stdout_lines": []
}

This is because of the one of the previous tasks uprade the neutron pip package from the Queens to the Rocky one version:

2018-06-18 19:22:54,594 p=24955 u=root | TASK [os_neutron : Install optional pip packages] *************************************************************************************************************
******************************************
2018-06-18 19:22:54,595 p=24955 u=root | task path: /etc/ansible/roles/os_neutron/tasks/neutron_install.yml:136
2018-06-18 19:22:54,595 p=24955 u=root | Monday 18 June 2018 19:22:54 +0000 (0:00:01.128) 1:54:17.571 ***********
2018-06-18 19:22:54,715 p=24955 u=root | container_name: "aio1_neutron_server_container-1cdfa253"
2018-06-18 19:22:54,716 p=24955 u=root | physical_host: "aio1"
2018-06-18 19:22:54,716 p=24955 u=root | Container confirmed
2018-06-18 19:22:54,716 p=24955 u=root | Container type "lxc"
2018-06-18 19:22:54,720 p=24955 u=root | Using module file /opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/packaging/language/pip.py
2018-06-18 19:22:54,721 p=24955 u=root | container_name: "aio1_neutron_server_container-1cdfa253"
2018-06-18 19:22:54,721 p=24955 u=root | physical_host: "aio1"
2018-06-18 19:22:54,721 p=24955 u=root | Container confirmed
2018-06-18 19:22:54,721 p=24955 u=root | Container type "lxc"
2018-06-18 19:24:44,065 p=24955 u=root | changed: [aio1_neutron_server_container-1cdfa253] => {
    "attempts": 1,
    "changed": true,
    "cmd": "/openstack/venvs/neutron-17.0.5/bin/pip2 install -U --constraint /opt/global-requirement-pins.txt --constraint https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints
.txt?id=23a4a4b5c9b17a31a367edbd2e28954e8d9bb231 neutron_lbaas",
    "failed": false,
    "invocation": {
        "module_args": {
            "chdir": null,
            "editable": false,
            "executable": null,
            "extra_args": " --constraint /opt/global-requirement-pins.txt --constraint https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=23a4a4b5c9b17a31a367edbd2e289
54e8d9bb231 ",
            "name": [
                "neutron_lbaas"
            ],
            "requirements": null,
            "state": "latest",
            "umask": null,
            "use_mirrors": true,
            "version": null,
            "virtualenv": "/openstack/venvs/neutron-17.0.5",
            "virtualenv_command": "virtualenv",
            "virtualenv_python": null,
            "virtualenv_site_packages": false
        }
    },
    "name": [
        "neutron_lbaas"
    ],
    "requirements": null,
    "state": "latest",
[...cut...]
    "stdout_lines": [
        "Collecting neutron_lbaas",
        " Downloading https://files.pythonhosted.org/packages/fc/14/da12b62fe644d470aae1f4d23d2aa732c0e3d840b140e42719ccab11392f/neutron_lbaas-13.0.0.0b2-py2.py3-none-any.whl (367kB)",
        "Requirement already up-to-date: oslo.messaging===5.35.0 in /openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages (from -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=23a4a4b5c9b17a31a367edbd2e28954e8d9bb231 (line 145))",
        "Requirement already up-to-date: oslo.reports===1.26.0 in /openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages (from -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=23a4a4b5c9b17a31a367edbd2e28954e8d9bb231 (line 349))",
        "Requirement already up-to-date: neutron-lib===1.13.0 in /openstack/venvs/neutron-17.0.5/lib/python2.7/site-packages (from -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=23a4a4b5c9b17a31a367edbd2e28954e8d9bb231 (line 58))",
        "Collecting pyasn1===0.4.2 (from -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?id=23a4a4b5c9b17a31a367edbd2e28954e8d9bb231 (line 389))",
        " Downloading https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl (71kB)",
        "Collecting neutron>=12.0.0 (from neutron_lbaas)",
        " Downloading https://files.pythonhosted.org/packages/77/1f/9e8f66e3cc28ceb69d9cbb423e010c695816f7b69b312d7e39846dc076c3/neutron-13.0.0.0b2-py2.py3-none-any.whl (3.1MB)",
[...cut...]
        "Installing collected packages: python-barbicanclient, pyasn1, pyasn1-modules, neutron, neutron-lbaas, argparse",
        " Found existing installation: neutron 12.0.3.dev9",
        " Uninstalling neutron-12.0.3.dev9:",
        " Successfully uninstalled neutron-12.0.3.dev9",
        " Found existing installation: argparse 1.2.1",
        " Not uninstalling argparse at /usr/lib/python2.7, outside environment /openstack/venvs/neutron-17.0.5",
        "Successfully installed argparse-1.4.0 neutron-13.0.0.0b2 neutron-lbaas-13.0.0.0b2 pyasn1-0.4.2 pyasn1-modules-0.2.1 python-barbicanclient-4.6.0"
    ],
    "version": null,
    "virtualenv": "/openstack/venvs/neutron-17.0.5"
}

as we can see the correct version neutron-12.0.3.dev9 was replaced with neutron-13.0.0.0b2.

Changed in openstack-ansible:
assignee: nobody → Maksim Malchuk (mmalchuk)
status: New → In Progress
Revision history for this message
Maksim Malchuk (mmalchuk) wrote :
Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
Michael Johnson (johnsom) wrote :

This was the incorrect fix. Neutron plugins cannot be added to the upper-constraints.txt file in releases prior to Rocky. The ansible code should be fixed to install the correct branch/version of neutron-lbaas.

Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

Michael, thanks for the comment and the commit revert. The issue indeed should be fixed on the deployment side and I've prepared the bugfix, which actually adds an ability to correctly build from the source in the environments without Internet access with own PyPi indexes and local constraints files.

Commit for the Rocky: https://review.openstack.org/#/c/577652/
and for the Queens: https://review.openstack.org/#/c/577653/

Both well tested on my CI and on the Zuul.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/requirements queens-eol

This issue was fixed in the openstack/requirements queens-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.