Services not running that should be: cinder-scheduler, neutron-lbaasv2-agent, neutron-dhcp-agent, neutron-metering-agent

Bug #1834565 reported by Chris MacNaughton on 2019-06-28
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack cinder charm
Undecided
Unassigned
OpenStack neutron-gateway charm
Undecided
Unassigned
Ubuntu Cloud Archive
Status tracked in Stein
Stein
High
Unassigned
Train
High
Unassigned
python-eventlet (Ubuntu)
Status tracked in Eoan
Disco
High
James Page
Eoan
High
Unassigned

Bug Description

[Impact]
Unable to enable TLS termination of any endpoints for OpenStack with Python 3.7.

[Test Case]
sudo apt install python3-eventlet
python3 << EOF
import eventlet
eventlet.monkey_patch()

import socket
import ssl

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
wrappedSocket = ssl.wrap_socket(sock)
EOF

Traceback (most recent call last):
  File "<stdin>", line 8, in <module>
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 403, in wrap_socket
    return GreenSSLSocket(sock, *a, **kw)
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
    *args, **kw
  File "/usr/lib/python3.7/ssl.py", line 1222, in wrap_socket
    suppress_ragged_eofs=suppress_ragged_eofs
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 426, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

[Regression Potential]
Low; codepaths are all Python 3.7 specific due to internal changes in the SSL implementation.

[Original bug report]
In an SSL enabled OpenStack Stein deploy, deployed on Disco, the cinder-scheduler service is crashing with the below stacktrace:

2019-06-27 18:35:07.593 5427 WARNING cinder.scheduler.host_manager [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] volume service is down. (host: juju-20eff5-mojo-0-7@LVM)
2019-06-27 18:35:07.594 5427 WARNING cinder.scheduler.host_manager [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] volume service is down. (host: cinder@cinder-ceph)
2019-06-27 18:35:07.610 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-volume objects version 1.37 as minimum service version.
2019-06-27 18:35:07.619 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-volume RPC version 3.16 as minimum service version.
2019-06-27 18:35:07.638 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-scheduler objects version 1.37 as minimum service version.
2019-06-27 18:35:07.647 5427 INFO cinder.rpc [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Automatically selected cinder-scheduler RPC version 3.11 as minimum service version.
2019-06-27 18:35:07.702 5427 WARNING oslo_config.cfg [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S release.). Its value may be silently ignored in the future.
2019-06-27 18:35:07.703 5427 WARNING oslo_config.cfg [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Deprecated: Option "signing_dir" from group "keystone_authtoken" is deprecated for removal (PKI token format is no longer supported.). Its value may be silently ignored in the future.
2019-06-27 18:35:07.705 5427 WARNING oslo_config.cfg [req-9c5d17cd-1414-482a-bfc9-a6bef2369637 - - - - -] Deprecated: Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken".
2019-06-27 18:35:07.726 5427 INFO cinder.service [-] Starting cinder-scheduler node (version 14.0.0)
2019-06-27 18:35:07.736 5427 INFO cinder.manager [req-e208c3e5-83e5-4c5b-85a3-1f5e09c86da9 - - - - -] Initiating service 4 cleanup
2019-06-27 18:35:07.745 5427 INFO cinder.manager [req-e208c3e5-83e5-4c5b-85a3-1f5e09c86da9 - - - - -] Service 4 cleanup completed.
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service [req-e208c3e5-83e5-4c5b-85a3-1f5e09c86da9 - - - - -] Error starting thread.: TypeError: wrap_socket() got an unexpected keyword argument '_context'
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service Traceback (most recent call last):
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service service.start()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/service.py", line 236, in start
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.rpcserver.start()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 270, in wrapper
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service log_after, timeout_timer)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 190, in run_once
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service post_fn = fn()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 269, in <lambda>
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service states[state].run_once(lambda: fn(self, *args, **kwargs),
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 414, in start
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.listener = self._create_listener()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 151, in _create_listener
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service return self.transport._listen(self._target, 1, None)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 143, in _listen
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service batch_timeout)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 652, in listen
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in _get_connection
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service purpose=purpose)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/common.py", line 407, in __init__
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.connection = connection_pool.create(purpose)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/pool.py", line 144, in create
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service return self.connection_cls(self.conf, self.url, purpose)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 531, in __init__
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.ensure_connection()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 646, in ensure_connection
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.connection.ensure_connection(errback=on_error)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/kombu/connection.py", line 405, in ensure_connection
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service callback)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/kombu/utils/functional.py", line 332, in retry_over_time
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service return fun(*args, **kwargs)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/kombu/connection.py", line 261, in connect
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service return self.connection
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/kombu/connection.py", line 802, in connection
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self._connection = self._establish_connection()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/kombu/connection.py", line 757, in _establish_connection
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service conn = self.transport.establish_connection()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service conn.connect()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/connection.py", line 307, in connect
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.transport.connect()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/transport.py", line 83, in connect
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.socket_settings, self.read_timeout, self.write_timeout,
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/transport.py", line 187, in _init_socket
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self._setup_transport()
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/transport.py", line 302, in _setup_transport
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service self.sock = self._wrap_socket(self.sock, **self.sslopts)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/transport.py", line 309, in _wrap_socket
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service return self._wrap_socket_sni(sock, **sslopts)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/amqp/transport.py", line 344, in _wrap_socket_sni
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service sock = ssl.wrap_socket(**opts)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 403, in wrap_socket
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service return GreenSSLSocket(sock, *a, **kw)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service *args, **kw
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3.7/ssl.py", line 1222, in wrap_socket
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service suppress_ragged_eofs=suppress_ragged_eofs
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 426, in wrap_socket
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service return GreenSSLSocket(sock, *a, _context=self, **kw)
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 80, in __init__
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service *args, **kw
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service TypeError: wrap_socket() got an unexpected keyword argument '_context'
2019-06-27 18:35:07.776 5427 ERROR oslo_service.service

Additionally, charm-neutron-gateway has the same stacktrace in the mentioned services.

summary: - Services not running that should be: cinder-scheduler
+ Services not running that should be: cinder-scheduler, neutron-
+ lbaasv2-agent, neutron-dhcp-agent, neutron-metering-agent
description: updated
description: updated
James Page (james-page) wrote :

Upstream eventlet bug report:

  https://github.com/eventlet/eventlet/issues/526

A further fix is in 0.25 (eoan development); I've cherry picked it back to disco/stein in:

  https://launchpad.net/~james-page/+archive/ubuntu/stein/+packages

I was not able to reproduce the stack trace using the minimal test case in the upstream bug report on eoan, but was able to on disco; testing packages once they have built.

James Page (james-page) wrote :

Simple reproducer:

import eventlet
eventlet.monkey_patch()

import socket
import ssl

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
wrappedSocket = ssl.wrap_socket(sock)

James Page (james-page) wrote :

Test package with my simple reproducer works OK so will upload to disco for SRU team review.

Changed in python-eventlet (Ubuntu Eoan):
status: New → Fix Released
importance: Undecided → High
Changed in python-eventlet (Ubuntu Disco):
status: New → In Progress
importance: Undecided → High
assignee: nobody → James Page (james-page)
Changed in charm-cinder:
status: New → Invalid
Changed in charm-neutron-gateway:
status: New → Invalid
James Page (james-page) on 2019-07-01
description: updated

Hello Chris, or anyone else affected,

Accepted python-eventlet into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-eventlet/0.24.1-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-eventlet (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco

All autopkgtests for the newly accepted python-eventlet (0.24.1-0ubuntu3.1) for disco have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#disco

Hello Chris, or anyone else affected,

Accepted python-eventlet into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
James Page (james-page) on 2019-07-03
tags: added: verification-done-disco
removed: verification-needed-disco verification-stein-needed
tags: added: verification-done
removed: verification-needed
Łukasz Zemczak (sil2100) wrote :

What kind of verification has been performed and on what package versions?

tags: added: verification-needed-disco
removed: verification-done-disco
tags: added: verification-needed
removed: verification-done
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.