syslog logging to /dev/log race condition

[Dmitry Mescheryakov]

Link to the original bug: https://bugs.launchpad.net/mos/+bug/1101404

[Roman Podoliaka (rpodolyaka)]

This is purely an eventlet/python2.6 interaction issue, not sure what we can do in Nova.

[Roman Podoliaka (rpodolyaka)]

MOS Linux team, could you please evaluate how hard it would be to patch python2.6 version we use in CentOS? (upstream bug contains detailed information on this)

[Alexei Sheplyakov (asheplyakov)]

CentOS/RH contains quite a number of important packages which depend on python: yum, anaconda, etc.
Therefore rebuilding python package might trigger the rebuild of those packages (recently ~60 of packages has been
rebuilt as the side effect of the kernel update). Actually there's no need to rebuild anything (the fix is in the *.py file),
but I don't know if the build system is smart enough to figure that out. (By the way, what is the best way to prevent
reverse dependencies from being rebuilt?)

Also the logging patch just papers over the problem: the real bug is wrong ENOTCONN handling by eventlet.
I'm afraid we'll run into the similar difficult to debug problems (that is, sporadic failures to reconnect when a service
restarts) in other OpenStack components.

[Alexei Sheplyakov (asheplyakov)]

Yet another idea: can we just monkey patch logging.log in oslo?

[Alexei Sheplyakov (asheplyakov)]

The current version of python shipped with CentOS 6.5 already includes the fix (or rather a work around):

$ cat python-2.6.6-fix-SysLogHandler-error-handling.patch
diff -up Python-2.6.6/Lib/logging/handlers.py.15179 Python-2.6.6/Lib/logging/handlers.py
--- Python-2.6.6/Lib/logging/handlers.py.15179 2010-05-24 05:43:13.000000000 -0400
+++ Python-2.6.6/Lib/logging/handlers.py 2012-10-11 20:45:18.849380832 -0400
@@ -727,7 +727,11 @@ class SysLogHandler(logging.Handler):
         except socket.error:
             self.socket.close()
             self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
- self.socket.connect(address)
+ try:
+ self.socket.connect(address)
+ except socket.error:
+ self.socket.close()
+ raise

     # curious: when talking to the unix-domain '/dev/log' socket, a
     # zero-terminator seems to be required. this string is placed

[Michael Semenov (msemenov)]

We are using upstream package in 5.1. It contans the fix. So, moving to Invalid.

[Roman Podoliaka (rpodolyaka)]

Seems that Ubuntu 12.04 is affected

[Roman Podoliaka (rpodolyaka)]

This patch should be applied to cpython - http://hg.python.org/cpython/rev/99f0c0207faa

[Alexei Sheplyakov (asheplyakov)]

Ubuntu 12.04 uses python 2.7 which includes the above mentioned fix.

Here is the relevant fragment of the code from /usr/lib/python2.7/logging/handlers.py

 737 def _connect_unixsocket(self, address):
 738 self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
 739 # syslog may require either DGRAM or STREAM sockets
 740 try:
 741 self.socket.connect(address)
 742 except socket.error:
 743 self.socket.close()
 744 self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
 745 self.socket.connect(address)
 736 try:
 737 self.socket.connect(address)
 738 except socket.error:
 739 self.socket.close()
 740 raise

[Ryan Moe (rmoe)]

This still exists in Ubuntu on 5.1. Our python 2.7 package does not contain the fix in #8. Do we also need this fix http://hg.python.org/cpython/rev/d91da96a55bf?

[Dmitry Borodaenko (angdraug)]

http://fuel-repository.mirantis.com/fwm/5.1/ubuntu/pool/main/python2.7-minimal_2.7.3-0ubuntu3.5_amd64.deb

and

http://packages.ubuntu.com/precise/amd64/python2.7-minimal/download

are the exact same packages down to md5sum, and neither includes the fixes for this problem.

[Kostiantyn Danylov (kdanylov)]

waiting for python code land in fuel git to add a patch. Probably would fix Aug 14

[Kostiantyn Danylov (kdanylov)]

python code pushed to git. https://gerrit.mirantis.com/#/c/20443/. Working on patch

[OSCI Robot (oscirobot)]

Package python has been built from changeset: http://gerrit.mirantis.com/20443
DEB Repository URL: http:///osci-obs.vm.mirantis.net:82/ubuntu-fuel-master-20443/ubuntu

[Kostiantyn Danylov (kdanylov)]

patch http://hg.python.org/cpython/rev/d91da96a55bf is in repo - https://gerrit.mirantis.com/#/c/20443/

[OSCI Robot (oscirobot)]

Package python has been built from changeset: http://gerrit.mirantis.com/20443
DEB Repository URL: http:///osci-obs.vm.mirantis.net:82/ubuntu-fuel-master-20443/ubuntu

[OSCI Robot (oscirobot)]

Package python has been built from changeset: http://gerrit.mirantis.com/20443
DEB Repository URL: http:///osci-obs.vm.mirantis.net:82/ubuntu-fuel-master/ubuntu

[Vladimir Sharshov (vsharshov)]

Reproduced again. Details: https://bugs.launchpad.net/fuel/+bug/1356924

  release: "5.1"
  api: "1.0"
  build_number: "436"
  build_id: "2014-08-14_02-01-17"
  astute_sha: "b52910642d6de941444901b0f20e95ebbcb2b2e9"
  fuellib_sha: "4b085bfbf7be973f0aa29d9d5e4f3ebd5bf789a1"
  ostf_sha: "d2a894d228c1f3c22595a77f04b1e00d09d8e463"
  nailgun_sha: "b5bdd19c2dbeb26ce3bd88270d09f5e7541a3aea"
  fuelmain_sha: "9f327045cdd72d406d89063393a499635be5e3d4"

[Dmitry Borodaenko (angdraug)]

ISO #457 and onward include the python packages built from changesets quoted above.

[OSCI Robot (oscirobot)]

Package python-eventlet has been built from changeset: https://review.fuel-infra.org/230
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-230/ubuntu

[OSCI Robot (oscirobot)]

Package python-eventlet has been built from changeset: https://review.fuel-infra.org/231
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-231/centos

[OSCI Robot (oscirobot)]

Package python-eventlet has been built from changeset: https://review.fuel-infra.org/230
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

[OSCI Robot (oscirobot)]

Package python-eventlet has been built from changeset: https://review.fuel-infra.org/231
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

[James Page (james-page)]

Associated Ubuntu bug 1081022

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Igor Yozhikov <email address hidden>
Review: https://review.fuel-infra.org/7698

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Igor Yozhikov <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/7698
Reason: Doesn't required anymore