Could not start Service[neutron-server]

Bug #1324859 reported by Egor Kotko on 2014-05-30
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Vladimir Kuklin
4.1.x
Critical
Fuel Library (Deprecated)
5.0.x
Critical
Sergey Kolekonov
Mirantis OpenStack
High
Sergey Kolekonov
5.0.x
Critical
Vladimir Kuklin
5.1.x
High
Sergey Kolekonov

Bug Description

{"build_id": "2014-05-30_00-41-31", "mirantis": "yes", "build_number": "331", "nailgun_sha": "0828946af6a2e757fb018c2f69f7f81f0ea4b7f1", "ostf_sha": "3690709d95c1f42506b6d955b44221910cec8527", "fuelmain_sha": "7fbe99c86a4fffcb5ebc5b86046dcf1c20776f61", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "c64ffdc5e7bfe7fa8eb971be1fea2260ee849ab5"}

Steps to reproduce:
- Create environment trough UI/CLI -
Centos HA 3 Controllers, 2 Cephs (for Volumes and Images ), 2 Computes

Expected result:
Deployed cluster

Actual result:
Cluster deployed with error in puppet log

Puppet log:
(/Stage[main]/Neutron::Server/Service[neutron-server]/ensure) change from stopped to running failed: Could not start Service[neutron-server]: Execution of '/sbin/service neutron-server start' returned 6: at /etc/puppet/modules/neutron/manifests/server.pp:93

Egor Kotko (ykotko) wrote :
Egor Kotko (ykotko) wrote :

Get the same problem, logs are attached

Changed in fuel:
status: New → Confirmed
Vladimir Kuklin (vkuklin) wrote :

need a reproducer on the newer ISO.

no longer affects: fuel/4.1.x
Changed in fuel:
status: Confirmed → Incomplete
assignee: nobody → Fuel Library Team (fuel-library)
Artem Panchenko (apanchenko-8) wrote :

Reproduced on ISO # 336:

{"build_id": "2014-06-02_18-24-56", "mirantis": "yes", "build_number": "336", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "8ed4269439efd09027e1e4e89d49e366604375ff", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "7a0bc0c733afdc02f30dc848579c8cce12b64ebd"}

Here is the part of puppet logs (node-1):

<27>Jun 2 19:05:30 node-1 puppet-apply[30524]: (/Stage[main]/Neutron::Server/Service[neutron-server]/ensure) change from stopped to running failed: Could not start Service[neutron-server]: Execution of '/sbin/service neutron-server start' returned 6: at /etc/puppet/modules/neutron/manifests/server.pp:93

and commands outputs:

[root@fuelweb ~]# ssh node-1 "/sbin/service neutron-server start; echo $?"

neutron-db-check: ERROR: neutron database does not have version information. You must run the "neutron-db-manage stamp" command before starting neutron services.
6

[root@fuelweb ~]# ssh node-1 "neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini stamp head"

No handlers could be found for logger "neutron.common.legacy"
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.

[root@fuelweb ~]# ssh node-1 "/sbin/service neutron-server start; echo $?"

Starting neutron: [ OK ]
0

Changed in fuel:
status: Incomplete → Confirmed
Ryan Moe (rmoe) wrote :

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head

That command needs to be run on an empty neutron database prior to starting neutron-server. This should be done in puppet like in upstream. When the database is empty when upgrade runs it sets up the version correctly. stamp is only needed when trying to run upgrade with existing tables.

Vladimir Kuklin (vkuklin) wrote :

Marking as confirmed for Havana, but we need info from Fuel Hardening team, whether this bug affects current master and Icehouse branch.

Changed in fuel:
milestone: 4.1.1 → none

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/97395

Changed in fuel:
assignee: Fuel Hardening Team (fuel-hardening) → Vladimir Kuklin (vkuklin)
status: New → In Progress
Dmitry Borodaenko (angdraug) wrote :

This problem only exists in 4.1 starting with Havana 2013.2.3, and was never reproduced on Icehouse so far. Marking as Incomplete for 5.0 and 5.1.

Dmitry Borodaenko (angdraug) wrote :

Correction: we're likely to encounter this problem in 5.0.1 as well, a non-zero DB schema version number is required to reproduce.

Reviewed: https://review.openstack.org/97395
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5d87d59db50a64a73c0846b97e55d3498075d5c4
Submitter: Jenkins
Branch: stable/4.1

commit 5d87d59db50a64a73c0846b97e55d3498075d5c4
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server starts

    Co-authored-by: Ryan Moe <email address hidden>
    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421
    Closes-bug: #1324859

Tatyanka (tatyana-leontovich) wrote :

verified {"build_id": "2014-06-04_09-16-08", "mirantis": "yes", "build_number": "341", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "7ed0f85acc0bab4b9157703a618b8cc9fd7de3e1", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "0e96fc5a340cd57f75c454ea8536471379299494"}

Vladimir Kuklin (vkuklin) wrote :

this bug should be confirmed for 2014.1.1, I guess. Marking as Confirmed for a while. Recheck needed.

Changed in fuel:
status: Confirmed → In Progress
Vladimir Kuklin (vkuklin) wrote :

it looks like, the fix is related not only to db migrations but also to default plugin ini config file.

Changed in mos:
milestone: none → 5.0.1
importance: Undecided → Critical
status: New → In Progress
assignee: nobody → Vladimir Kuklin (vkuklin)

Reviewed: https://review.openstack.org/97398
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f94feb46c15e3e2f8af6d227803808902723bc68
Submitter: Jenkins
Branch: stable/5.0

commit f94feb46c15e3e2f8af6d227803808902723bc68
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server starts

    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421
    Closes-bug: #1324859

Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Sergey Kolekonov (skolekonov)
Changed in fuel:
assignee: Sergey Kolekonov (skolekonov) → Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
assignee: Sergii Golovatiuk (sgolovatiuk) → Sergey Kolekonov (skolekonov)
Dmitry Pyzhov (dpyzhov) on 2014-07-15
no longer affects: fuel/5.1.x
Changed in fuel:
assignee: Sergey Kolekonov (skolekonov) → Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
assignee: Sergii Golovatiuk (sgolovatiuk) → Vladimir Kuklin (vkuklin)

Reviewed: https://review.openstack.org/97397
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c104235821c917395732c9dca7f1542102fcb6a6
Submitter: Jenkins
Branch: master

commit c104235821c917395732c9dca7f1542102fcb6a6
Author: Vladimir Kuklin <email address hidden>
Date: Tue Jun 3 09:36:10 2014 +0400

    Run database migration before neutron-server

    - Depends on merge of 2014.1.1 (or newer)
    - neutron with alembic migrations

    Closes-bug: #1324859
    Change-Id: Iafce36930c1c73e3e64e34d94ca1e52f850a1421

Changed in fuel:
status: In Progress → Fix Committed
Nastya Urlapova (aurlapova) wrote :

{
build_id: "2014-08-14_19-52-36",
mirantis: "yes",
build_number: "170",
ostf_sha: "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
nailgun_sha: "af3d1922bfc21345f81be3454115ab6139675c35",
production: "docker",
api: "1.0",
fuelmain_sha: "fd58828f404e4298ed338e8f44c6a326cebd31de",
astute_sha: "6db5f5031b74e67b92fcac1f7998eaa296d68025",
release: "5.0.1",
fuellib_sha: "a31dbac8fff9cf6bc4cd0d23459670e34b27a9ab"
}

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers