Could not start Service[neutron-server]

[Egor Kotko]

{"build_id": "2014-05-30_00-41-31", "mirantis": "yes", "build_number": "331", "nailgun_sha": "0828946af6a2e757fb018c2f69f7f81f0ea4b7f1", "ostf_sha": "3690709d95c1f42506b6d955b44221910cec8527", "fuelmain_sha": "7fbe99c86a4fffcb5ebc5b86046dcf1c20776f61", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "c64ffdc5e7bfe7fa8eb971be1fea2260ee849ab5"}

Steps to reproduce:
- Create environment trough UI/CLI -
Centos HA 3 Controllers, 2 Cephs (for Volumes and Images ), 2 Computes

Expected result:
Deployed cluster

Actual result:
Cluster deployed with error in puppet log

Puppet log:
(/Stage[main]/Neutron::Server/Service[neutron-server]/ensure) change from stopped to running failed: Could not start Service[neutron-server]: Execution of '/sbin/service neutron-server start' returned 6: at /etc/puppet/modules/neutron/manifests/server.pp:93

[Andrey Sledzinskiy (asledzinskiy)]

Get the same problem, logs are attached

[Vladimir Kuklin (vkuklin)]

need a reproducer on the newer ISO.

[Artem Panchenko (apanchenko-8)]

Reproduced on ISO # 336:

{"build_id": "2014-06-02_18-24-56", "mirantis": "yes", "build_number": "336", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "8ed4269439efd09027e1e4e89d49e366604375ff", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "7a0bc0c733afdc02f30dc848579c8cce12b64ebd"}

Here is the part of puppet logs (node-1):

<27>Jun 2 19:05:30 node-1 puppet-apply[30524]: (/Stage[main]/Neutron::Server/Service[neutron-server]/ensure) change from stopped to running failed: Could not start Service[neutron-server]: Execution of '/sbin/service neutron-server start' returned 6: at /etc/puppet/modules/neutron/manifests/server.pp:93

and commands outputs:

[root@fuelweb ~]# ssh node-1 "/sbin/service neutron-server start; echo $?"

neutron-db-check: ERROR: neutron database does not have version information. You must run the "neutron-db-manage stamp" command before starting neutron services.
6

[root@fuelweb ~]# ssh node-1 "neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp head"

No handlers could be found for logger "neutron.common.legacy"
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.

[root@fuelweb ~]# ssh node-1 "/sbin/service neutron-server start; echo $?"

Starting neutron: [ OK ]
0

[Ryan Moe (rmoe)]

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head

That command needs to be run on an empty neutron database prior to starting neutron-server. This should be done in puppet like in upstream. When the database is empty when upgrade runs it sets up the version correctly. stamp is only needed when trying to run upgrade with existing tables.

[Vladimir Kuklin (vkuklin)]

Marking as confirmed for Havana, but we need info from Fuel Hardening team, whether this bug affects current master and Icehouse branch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/97395

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/97397

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/97398

[Dmitry Borodaenko (angdraug)]

This problem only exists in 4.1 starting with Havana 2013.2.3, and was never reproduced on Icehouse so far. Marking as Incomplete for 5.0 and 5.1.

[Dmitry Borodaenko (angdraug)]

Correction: we're likely to encounter this problem in 5.0.1 as well, a non-zero DB schema version number is required to reproduce.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97395
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5d87d59db50a64a73c0846b97e55d3498075d5c4
Submitter: Jenkins
Branch: stable/4.1

commit 5d87d59db50a64a73c0846b97e55d3498075d5c4
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server starts

    Co-authored-by: Ryan Moe <email address hidden>
    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421
    Closes-bug: #1324859

[Tatyanka (tatyana-leontovich)]

verified {"build_id": "2014-06-04_09-16-08", "mirantis": "yes", "build_number": "341", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "7ed0f85acc0bab4b9157703a618b8cc9fd7de3e1", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "0e96fc5a340cd57f75c454ea8536471379299494"}

[Vladimir Kuklin (vkuklin)]

this bug should be confirmed for 2014.1.1, I guess. Marking as Confirmed for a while. Recheck needed.

[Vladimir Kuklin (vkuklin)]

it looks like, the fix is related not only to db migrations but also to default plugin ini config file.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97398
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f94feb46c15e3e2f8af6d227803808902723bc68
Submitter: Jenkins
Branch: stable/5.0

commit f94feb46c15e3e2f8af6d227803808902723bc68
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server starts

    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421
    Closes-bug: #1324859

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97397
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c104235821c917395732c9dca7f1542102fcb6a6
Submitter: Jenkins
Branch: master

commit c104235821c917395732c9dca7f1542102fcb6a6
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server

    - Depends on merge of 2014.1.1 (or newer)
    - neutron with alembic migrations

    Closes-bug: #1324859
    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421

[Nastya Urlapova (aurlapova)]

{
build_id: "2014-08-14_19-52-36",
mirantis: "yes",
build_number: "170",
ostf_sha: "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
nailgun_sha: "af3d1922bfc21345f81be3454115ab6139675c35",
production: "docker",
api: "1.0",
fuelmain_sha: "fd58828f404e4298ed338e8f44c6a326cebd31de",
astute_sha: "6db5f5031b74e67b92fcac1f7998eaa296d68025",
release: "5.0.1",
fuellib_sha: "a31dbac8fff9cf6bc4cd0d23459670e34b27a9ab"
}