Neutron service is failing on centos9 scenario012 master

[Arx Cruz]

It seems when neutron tries to make a request, it fails, and then start again, checking the logs it show neutron restart every time the error occurs.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario012-standalone-master/30d3c64/logs/undercloud/var/log/extra/podman/containers/neutron_api/log/neutron/server.log.txt.gz

2022-02-11 01:49:22.553 2 CRITICAL neutron [req-a4afab67-16dc-4005-8482-76a8e7b20609 - - - - -] Unhandled error: tooz.coordination.ToozConnectionError: Connection closed by server.
2022-02-11 01:49:22.553 2 ERROR neutron Traceback (most recent call last):
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/tooz/drivers/redis.py", line 52, in wrapper
2022-02-11 01:49:22.553 2 ERROR neutron return func(*args, **kwargs)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/tooz/drivers/redis.py", line 476, in _start
2022-02-11 01:49:22.553 2 ERROR neutron self._server_info = self._client.info()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/client.py", line 1304, in info
2022-02-11 01:49:22.553 2 ERROR neutron return self.execute_command('INFO')
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/client.py", line 898, in execute_command
2022-02-11 01:49:22.553 2 ERROR neutron conn = self.connection or pool.get_connection(command_name, **options)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/connection.py", line 1192, in get_connection
2022-02-11 01:49:22.553 2 ERROR neutron connection.connect()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/connection.py", line 567, in connect
2022-02-11 01:49:22.553 2 ERROR neutron self.on_connect()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/connection.py", line 643, in on_connect
2022-02-11 01:49:22.553 2 ERROR neutron auth_response = self.read_response()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
2022-02-11 01:49:22.553 2 ERROR neutron response = self._parser.read_response()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/connection.py", line 324, in read_response
2022-02-11 01:49:22.553 2 ERROR neutron raw = self._buffer.readline()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/connection.py", line 256, in readline
2022-02-11 01:49:22.553 2 ERROR neutron self._read_from_socket()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/redis/connection.py", line 201, in _read_from_socket
2022-02-11 01:49:22.553 2 ERROR neutron raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-02-11 01:49:22.553 2 ERROR neutron redis.exceptions.ConnectionError: Connection closed by server.
2022-02-11 01:49:22.553 2 ERROR neutron
2022-02-11 01:49:22.553 2 ERROR neutron The above exception was the direct cause of the following exception:
2022-02-11 01:49:22.553 2 ERROR neutron
2022-02-11 01:49:22.553 2 ERROR neutron Traceback (most recent call last):
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module>
2022-02-11 01:49:22.553 2 ERROR neutron sys.exit(main())
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main
2022-02-11 01:49:22.553 2 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/server/__init__.py", line 68, in boot_server
2022-02-11 01:49:22.553 2 ERROR neutron server_func()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/server/wsgi_eventlet.py", line 23, in eventlet_wsgi_server
2022-02-11 01:49:22.553 2 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/service.py", line 91, in serve_wsgi
2022-02-11 01:49:22.553 2 ERROR neutron LOG.exception('Unrecoverable error: please check log '
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-02-11 01:49:22.553 2 ERROR neutron self.force_reraise()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-02-11 01:49:22.553 2 ERROR neutron raise self.value
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/service.py", line 88, in serve_wsgi
2022-02-11 01:49:22.553 2 ERROR neutron service.start()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/service.py", line 64, in start
2022-02-11 01:49:22.553 2 ERROR neutron self.wsgi_app = _run_wsgi(self.app_name)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/service.py", line 321, in _run_wsgi
2022-02-11 01:49:22.553 2 ERROR neutron app = config.load_paste_app(app_name)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/common/config.py", line 134, in load_paste_app
2022-02-11 01:49:22.553 2 ERROR neutron app = loader.load_app(app_name)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/oslo_service/wsgi.py", line 351, in load_app
2022-02-11 01:49:22.553 2 ERROR neutron return deploy.loadapp("config:%s" % self.config_path, name=name)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 253, in loadapp
2022-02-11 01:49:22.553 2 ERROR neutron return loadobj(APP, uri, name=name, **kw)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 278, in loadobj
2022-02-11 01:49:22.553 2 ERROR neutron return context.create()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2022-02-11 01:49:22.553 2 ERROR neutron return self.object_type.invoke(self)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 148, in invoke
2022-02-11 01:49:22.553 2 ERROR neutron return fix_call(context.object,
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/util.py", line 55, in fix_call
2022-02-11 01:49:22.553 2 ERROR neutron val = callable(*args, **kw)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/urlmap.py", line 31, in urlmap_factory
2022-02-11 01:49:22.553 2 ERROR neutron app = loader.get_app(app_name, global_conf=global_conf)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 355, in get_app
2022-02-11 01:49:22.553 2 ERROR neutron return self.app_context(
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2022-02-11 01:49:22.553 2 ERROR neutron return self.object_type.invoke(self)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 148, in invoke
2022-02-11 01:49:22.553 2 ERROR neutron return fix_call(context.object,
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/util.py", line 55, in fix_call
2022-02-11 01:49:22.553 2 ERROR neutron val = callable(*args, **kw)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/auth.py", line 62, in pipeline_factory
2022-02-11 01:49:22.553 2 ERROR neutron app = loader.get_app(pipeline[-1])
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 355, in get_app
2022-02-11 01:49:22.553 2 ERROR neutron return self.app_context(
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 715, in create
2022-02-11 01:49:22.553 2 ERROR neutron return self.object_type.invoke(self)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 152, in invoke
2022-02-11 01:49:22.553 2 ERROR neutron return fix_call(context.object, context.global_conf, **context.local_conf)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/paste/deploy/util.py", line 55, in fix_call
2022-02-11 01:49:22.553 2 ERROR neutron val = callable(*args, **kw)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/api/v2/router.py", line 25, in _factory
2022-02-11 01:49:22.553 2 ERROR neutron return pecan_app.v2_factory(global_config, **local_config)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory
2022-02-11 01:49:22.553 2 ERROR neutron startup.initialize_all()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
2022-02-11 01:49:22.553 2 ERROR neutron manager.init()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 301, in init
2022-02-11 01:49:22.553 2 ERROR neutron NeutronManager.get_instance()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 252, in get_instance
2022-02-11 01:49:22.553 2 ERROR neutron cls._create_instance()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 391, in inner
2022-02-11 01:49:22.553 2 ERROR neutron return f(*args, **kwargs)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 238, in _create_instance
2022-02-11 01:49:22.553 2 ERROR neutron cls._instance = cls()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 126, in __init__
2022-02-11 01:49:22.553 2 ERROR neutron plugin = self._get_plugin_instance(CORE_PLUGINS_NAMESPACE,
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/manager.py", line 162, in _get_plugin_instance
2022-02-11 01:49:22.553 2 ERROR neutron plugin_inst = plugin_class()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/quota/resource_registry.py", line 124, in wrapper
2022-02-11 01:49:22.553 2 ERROR neutron return f(*args, **kwargs)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/plugin.py", line 276, in __init__
2022-02-11 01:49:22.553 2 ERROR neutron self.mechanism_manager.initialize()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/neutron/plugins/ml2/managers.py", line 448, in initialize
2022-02-11 01:49:22.553 2 ERROR neutron driver.obj.initialize()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/networking_ansible/ml2/mech_driver.py", line 83, in initialize
2022-02-11 01:49:22.553 2 ERROR neutron self.coordinator.start(start_heart=True)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/tooz/coordination.py", line 689, in start
2022-02-11 01:49:22.553 2 ERROR neutron super(CoordinationDriverWithExecutor, self).start(start_heart)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/tooz/coordination.py", line 426, in start
2022-02-11 01:49:22.553 2 ERROR neutron self._start()
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/tooz/drivers/redis.py", line 61, in wrapper
2022-02-11 01:49:22.553 2 ERROR neutron utils.raise_with_cause(
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/tooz/utils.py", line 224, in raise_with_cause
2022-02-11 01:49:22.553 2 ERROR neutron excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2022-02-11 01:49:22.553 2 ERROR neutron File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 142, in raise_with_cause
2022-02-11 01:49:22.553 2 ERROR neutron raise exc_cls(message, *args, **kwargs) from kwargs.get('cause')
2022-02-11 01:49:22.553 2 ERROR neutron tooz.coordination.ToozConnectionError: Connection closed by server.

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario012-standalone-master&pipeline=openstack-periodic-integration-main

the failure started on 02/10

[Arx Cruz (arxcruz)]

You can see this error in both undercloud and overcloud

[Ronelle Landy (rlandy)]

looking at possible impact patches: https://opendev.org/openstack/puppet-neutron/commits/branch/master

[Ronelle Landy (rlandy)]

Notes change in network-scripts: https://bugs.launchpad.net/tripleo/+bug/1960625 - notes change in network scripts

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Redis container/service is not running in the undercloud. E.g.:

Working job: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario012-standalone-master/3d6ee4d/logs/undercloud/var/log/containers/redis/redis.log.txt.gz

Non working: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario012-standalone-master/0e96f0d/logs/undercloud/var/log/containers/

I'll check what changed in tripleO that stops starting this service.

[Rabi Mishra (rabi)]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario012-standalone-master/0e96f0d/logs/undercloud/var/log/containers/stdouts/redis.log.txt.gz

2022-02-11T09:28:34.193483272+00:00 stderr F + . kolla_extend_start
2022-02-11T09:28:34.193727328+00:00 stdout F Running command: '/usr/bin/redis-server /etc/redis.conf'
2022-02-11T09:28:34.193746339+00:00 stderr F + echo 'Running command: '\''/usr/bin/redis-server /etc/redis.conf'\'''
2022-02-11T09:28:34.193746339+00:00 stderr F + exec /usr/bin/redis-server /etc/redis.conf
2022-02-11T09:28:34.202537710+00:00 stdout F 2:C 11 Feb 2022 09:28:34.202 # Fatal error, can't open config file '/etc/redis.conf': No such file or directory

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827949 needed after https://github.com/voxpupuli/puppet-redis/commit/9d2e3fc5b67c7246f7f575bcd66f2d6f3726ac7d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/827949
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/e93b454d68dd97e9471e3324c751b5436574e1fc
Submitter: "Zuul (22348)"
Branch: master

commit e93b454d68dd97e9471e3324c751b5436574e1fc
Author: Takashi Kajinami <email address hidden>
Date: Sat Feb 5 23:22:33 2022 +0900

    Redis: Hard-code paths of configuration files

    In current CentOS 9 Stream, configuration files of redis is placed in
    /etc/redis. Following that, the puppet-redis module has been updated to
    use the new path by [1].

    This change ensures that the old path is consistently used in both
    CentOS 8 and CentOS 9 as we require a hard-coded path to start up
    redis in standalone mode(without pacemaker). The path can be updated
    later, for example when we drop support for CentOS 8.

    [1] https://github.com/voxpupuli/puppet-redis/pull/434

    Closes-Bug: #1960620
    Change-Id: Ibf7c2f8d01ddf73c92d183e51c378a0770002e72

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 16.0.0 release.