[neutron] ha neutron gre : puppet failed to restart neutron api

Bug #1355758 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
In Progress
High
Sergey Vasilenko

Bug Description

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "424"
  build_id: "2014-08-12_02-01-17"
  astute_sha: "b52910642d6de941444901b0f20e95ebbcb2b2e9"
  fuellib_sha: "54d834afccf1f8a97e4a82d7d081e7d1a1e068a1"
  ostf_sha: "d2a894d228c1f3c22595a77f04b1e00d09d8e463"
  nailgun_sha: "c7e00e5a00499d9f8dec608541dad1d745a8dd2e"
  fuelmain_sha: "9d4463400b4924159c978af43855e48bcf2a84b2"

http://paste.openstack.org/show/93809/
Steps to reproduce:

Deploy Centos ha gre using system tests

Expected Result:
Cluster ready. Ostf tests are passed

Actual Result:
one of controller(node-3) in error state
with failed attempts to restart neutron server
root@node-3 ~]# /sbin/service neutron-server restart
/sbin/service: line 66: 10026 Terminated env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS}

also next errors presents in rabbit logs for this node
http://paste.openstack.org/show/93811/

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
summary: - [neutron] Simple neutron gre : puppet failed to restart neutron api
+ [neutron] ha neutron gre : puppet failed to restart neutron api
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

RabbitMQ should not be the source of issue, according to the logs it had been reassembled its cluster w/o issues:

2014-08-12T07:43:22.419595 node-3 ./node-3.test.domain.local/lrmd.log:2014-08-12T07:43:22.419595+01:00 warning: WARNING: p_rabbitmq-server: start_rmq_server_app(): RMQ-server app can't start without Mnesia cleaning.
2014-08-12T07:43:22.991261 node-3 ./node-3.test.domain.local/lrmd.log:2014-08-12T07:43:22.991261+01:00 info: INFO: p_rabbitm
q-server: try_to_start_rmq_app(): begin.
2014-08-12T07:43:23.374082 node-3 ./node-3.test.domain.local/rabbitmq.log:2014-08-12T07:43:23.374082+01:00 notice: {error,{inconsistent_cluster,"Node 'rabbit@node-3' thinks it's clustered with node 'rabbit@node-5', but 'rabbit@node-5' disagrees"}}
2014-08-12T07:43:23.533744 node-3 ./node-3.test.domain.local/lrmd.log:2014-08-12T07:43:23.533744+01:00 info: INFO: p_rabbitmq-server: try_to_start_rmq_app(): start_app was successful.
2014-08-12T07:43:23.533866 node-3 ./node-3.test.domain.local/lrmd.log:2014-08-12T07:43:23.533866+01:00 info: INFO: p_rabbitmq-server: try_to_start_rmq_app(): waiting for start to finish.
2014-08-12T07:43:26.512117 node-3 ./node-3.test.domain.local/lrmd.log:2014-08-12T07:43:26.512117+01:00 info: INFO: p_rabbitmq-server: start_rmq_server_app(): RMQ-server app stopped succesfully.
2014-08-12T07:43:26.512303 node-3 ./node-3.test.domain.local/lrmd.log:2014-08-12T07:43:26.512303+01:00 info: INFO: p_rabbitmq-server: start_rmq_server_app(): RMQ-server app Mnesia cleaned successfully.
...

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Galera issues were also detected ~18 min before Neutron server failed restart
2014-08-12T07:32:08.757823+01:00 err: (/Stage[main]/Galera/Exec[wait-for-synced-state]/returns) change from notrun to 0 failed: /usr/bin/mysql -uwsrep_sst -ppassword -Nbe "show status like 'wsrep_local_state_comment'" | /bin/grep -q Synced && sleep 10 returned 1 instead of one of [0]

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

The entire deployment obviously will fail, then galera was not synced, setting as a dup for https://bugs.launchpad.net/fuel/+bug/1356812

Revision history for this message
Sergey Vasilenko (xenolog) wrote :

This is not a duplicate bug #1354479.

[root@node-3 ~]# /sbin/service neutron-server restart
/sbin/service: line 66: 10026 Terminated env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS}

happens because init script neutron-server kill himself at stop action.

stop() {
    echo -n $"Stopping $prog: "
    killproc -p $pidfile $prog
    if pgrep $prog-server &>/dev/null ; then
        sleep 2
        pgrep $prog-server &>/dev/null && \
        killall $prog-server
    fi
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergey Vasilenko (xenolog)
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: http://gerrit.mirantis.com/21165
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21165/centos

Revision history for this message
Alexander Ignatov (aignatov) wrote :

Sergey, don't we need to apply this change for 5.0.2 as well? Looking at the gerrit I see the patch which could expose the same behaviour as in 5.1 https://gerrit.mirantis.com/#/c/20676/1/rpm/SOURCES/neutron-server.init

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: http://gerrit.mirantis.com/21165
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos

Revision history for this message
Sergey Vasilenko (xenolog) wrote :

we need carefully check all init scripts on all repo versions for self-killing feature.
may be not only in Neulron.

Changed in fuel:
status: New → Confirmed
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: http://gerrit.mirantis.com/21310
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21310/centos

Changed in fuel:
assignee: Sergey Vasilenko (xenolog) → Dmitry Borodaenko (dborodaenko)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

bug #1362839 created to review and fix other init scripts for the same kind of problem.

Changed in fuel:
assignee: Dmitry Borodaenko (dborodaenko) → Sergey Vasilenko (xenolog)
status: Confirmed → In Progress
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

http://gerrit.mirantis.com/21310 is still in review, assigning back to xenolog with status = In Progress.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

so, Dmitry, did you review this request? is it ok or not?

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21440
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-21440/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21446
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21446/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21446
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21446/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: http://gerrit.mirantis.com/21310
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21310/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21446
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21446/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21440
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-21440/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package glance has been built from changeset: http://gerrit.mirantis.com/21528
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-21528/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package swift has been built from changeset: http://gerrit.mirantis.com/21535
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-21535/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package keystone has been built from changeset: http://gerrit.mirantis.com/21537
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21537/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: http://gerrit.mirantis.com/21310
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21310/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package keystone has been built from changeset: http://gerrit.mirantis.com/21540
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-21540/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: http://gerrit.mirantis.com/21310
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21310/centos

Revision history for this message
Sergey Vasilenko (xenolog) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package cinder has been built from changeset: http://gerrit.mirantis.com/21568
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-21568/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package neutron has been built from changeset: http://gerrit.mirantis.com/21310
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21310/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21446
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-21446/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package swift has been built from changeset: http://gerrit.mirantis.com/21535
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-21535/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/21445
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.2-stable-21445/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.