Bug #1333292 “[osci] nova-api hangs during openstack updating” : Bugs : Fuel for OpenStack

Revision history for this message

Ihor Kalnytskyi (ikalnytskyi) wrote on 2014-06-23:

#1

log.log Edit (3.3 KiB, text/plain)

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-06-23:

#2

It looks like "rsyslog" restart causes this, just like described here: https://bugs.launchpad.net/nova/+bug/1101404

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-06-23:

#3

Tested it wit Igor, if we disable rsyslog restart (put exit 0 in the beggining) then Opnestack update works fine.

Revision history for this message

Ihor Kalnytskyi (ikalnytskyi) wrote on 2014-06-23:

#4

What if rsyslog will be updated? We have to restart it in that case.

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-06-23:

#5

Steps to reproduce

1) Run on controller
/etc/init.d/rsyslog restart
/etc/init.d/openstack-nova-api restart

2) Check /var/log/nova-all.log, it will have:
error: [Errno 98] Address already in use

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-06-23:

#6

related bug in nova https://bugs.launchpad.net/nova/+bug/1101404

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-06-23:

#7

I see 3 possible ways to fix this:

1) Patch CentOS init scripts for API services to apply some additional checks/kill of stuck service in stop() function (the easiest and fastest way).
2) Reconnfigure all local logging on OS nodes to log via TCP/UDP intead of /dev/log (would require refactoring of our rsyslog config files since running through UDP breaks log matching).
3) Find and apply a patch for the problem python module so python logging could handle recreation of /dev/log fine.

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-06-23:

#8

Solutions
1) fix init.d/openstack* (add pgrep check and killall after killproc in stop() action)
2) patch system python2.6 handlers.py handlers.SysLogHandler
3) patch Oslo-incubator log.py (replace '/dev/log' for ('localhost', 514)

The (2) barely would have been accepted in upstream
The (3) is an ugly kludge and will require logging formaters updating as well.
The (1) looks like the best solution - we make a kludge in init.d for 'broken' service only (nova-api)

Changed in fuel:
status:	Confirmed → Triaged
assignee:	Aleksandr Didenko (adidenko) → Fuel OSCI Team (fuel-osci)

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2014-06-23:

#9

Looks like we should pick the option 1

Revision history for this message

Ihor Kalnytskyi (ikalnytskyi) wrote on 2014-06-23:

#10

I think the second solution (patch handlers.py) would not work. I have some research and figure out that our Python 2.6.6 already has this fix

As for third solution (/dev/log -> localist, 514) I'm not sure too, since the root of problem is in eventlet library. AFAIK, eventlet patchs both unixsocket and tcp/udp socket.

Well, we have only first solution then.

Roman Vyalov (r0mikiam) on 2014-06-24

Changed in fuel:
assignee:	Fuel OSCI Team (fuel-osci) → MOS Nova (mos-nova)

Revision history for this message

Ihor Kalnytskyi (ikalnytskyi) wrote on 2014-06-25:

#11

Please note, that the issue occurs not only for nova. For instance, I had same issue for cinder-api. So we have to fix init scripts for all opestack services, I guess.

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2014-06-25:

#12

This has nothing to do with Nova, but rather a CPython <-> eventlet interaction issue. MOS Compute can't help here. I'd suggest you patch CPython/eventlet version we use.

Changed in fuel:
assignee:	MOS Nova (mos-nova) → nobody

Revision history for this message

Aleksandr Didenko (adidenko) wrote on 2014-06-25:

#13

Something like this should do the trick:

--- /etc/init.d/openstack-nova-api.origin 2013-10-28 12:03:21.000000000 +0000
+++ /etc/init.d/openstack-nova-api 2014-06-25 14:45:50.045890202 +0000
@@ -49,6 +49,11 @@
stop() {
     echo -n $"Stopping $prog: "
     killproc -p $pidfile $prog
+ if pgrep nova-$suffix &>/dev/null ; then
+ sleep 2
+ pgrep nova-$suffix &>/dev/null && \
+ killall nova-$suffix
+ fi
     retval=$?
     echo
     [ $retval -eq 0 ] && rm -f $lockfile

I've tested it and it works fine, nova-api gets restarted OK.

Revision history for this message

Ihor Kalnytskyi (ikalnytskyi) wrote on 2014-06-25:

#14

We need it not only for nova-api, but for all services, since I've met this bug for cinder-api,nova-api and nova-conductor.

Changed in fuel:
assignee:	nobody → Fuel OSCI Team (fuel-osci)
no longer affects:	fuel/5.0.x

Roman Vyalov (r0mikiam) on 2014-06-25

Changed in fuel:
assignee:	Fuel OSCI Team (fuel-osci) → Sergey Otpuschennikov (sotpuschennikov)

Sergey Otpuschennikov (sotpuschennikov) on 2014-06-27

Changed in fuel:
status:	Triaged → In Progress

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#15

Package nova has been built from changeset: http://gerrit.mirantis.com/17043
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17043/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#16

Package cinder has been built from changeset: http://gerrit.mirantis.com/17044
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17044/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#17

Package glance has been built from changeset: http://gerrit.mirantis.com/17045
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17045/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#18

Package heat has been built from changeset: http://gerrit.mirantis.com/17046
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17046/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#20

Package nova has been built from changeset: http://gerrit.mirantis.com/17043
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17043/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#21

Package cinder has been built from changeset: http://gerrit.mirantis.com/17044
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17044/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#22

Package glance has been built from changeset: http://gerrit.mirantis.com/17045
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17045/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#23

Package neutron has been built from changeset: http://gerrit.mirantis.com/17055
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17055/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#24

Package keystone has been built from changeset: http://gerrit.mirantis.com/17057
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17057/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#25

Package murano-api has been built from changeset: http://gerrit.mirantis.com/17058
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17058/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#26

Package ceilometer has been built from changeset: http://gerrit.mirantis.com/17060
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17060/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2014-06-30:

#27

Package sahara has been built from changeset: http://gerrit.mirantis.com/17061
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17061/centos

Revision history for this message

Sergey Otpuschennikov (sotpuschennikov) wrote on 2014-06-30:

#28

Need review, please:
https://gerrit.mirantis.com/#/c/17043/
https://gerrit.mirantis.com/#/c/17044/
https://gerrit.mirantis.com/#/c/17045/
https://gerrit.mirantis.com/#/c/17046/
https://gerrit.mirantis.com/#/c/17055/
https://gerrit.mirantis.com/#/c/17057/
https://gerrit.mirantis.com/#/c/17058/
https://gerrit.mirantis.com/#/c/17060/
https://gerrit.mirantis.com/#/c/17061/

Revision history for this message

Tatyanka (tatyana-leontovich) wrote on 2014-07-01:

#29

Actually I build the iso with this one repo http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-17061/centos and init.d script for nova do not contain the patch ( Here is the iso link http://jenkins-product.srt.mirantis.net:8080/job/fuel_iso_with_gerrit_commits_docker/259/parameters/
May be I use wrong repo here?

Revision history for this message

Tatyanka (tatyana-leontovich) wrote on 2014-07-02:

#30

verified all reviews on {"build_id": "2014-07-02_14-56-30", "ostf_sha": "1e8d4775979b04450652836c19c97a30fbe8a22e", "build_number": "269", "api": "1.0", "nailgun_sha": "b52eb2bfc1e680365a19525a881acbc8a2e883c4", "production": "docker", "fuelmain_sha": "79c15f66c64d102f1920a9d104eca590b2a54c7c", "astute_sha": "4b5a0003f6c372229500f31b4a4e4d6708b154fb", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "5777e388a61ca4da8744e33cf2e6b215959ff1f9"}. works fine