neutron

neutron-fullstack-with-uwsgi job broken at resource setup

Bug #2065779 reported by yatin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Brian Haley

Bug Description

All tests fails as:-
ft1.1: neutron.tests.fullstack.test_agent_bandwidth_report.TestAgentBandwidthReport.test_agent_configurations(Open vSwitch agent)testtools.testresult.real._StringException: pythonlogging:'': {{{ERROR [stevedore.extension] Could not load 'oslo_cache.memcache_pool': No module named 'memcache'}}}
traceback-1: {{{
Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 743, in wait_until_true
    eventlet.sleep(sleep)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 38, in sleep
    hub.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
    self._setUp()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/process.py", line 175, in _setUp
    common_utils.wait_until_true(self.server_is_live)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 748, in wait_until_true
    raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 207, in setUp
    raise SetupError(details)
fixtures.fixture.SetupError: {}
}}}

traceback-2: {{{
Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 743, in wait_until_true
    eventlet.sleep(sleep)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 38, in sleep
    hub.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
    self._setUp()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/process.py", line 175, in _setUp
    common_utils.wait_until_true(self.server_is_live)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 748, in wait_until_true
    raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
    self._setUp()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/environment.py", line 423, in _setUp
    self.neutron_server = self.useFixture(
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 256, in useFixture
    fixture.setUp()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 211, in setUp
    raise MultipleExceptions(*errors)
testtools.runtest.MultipleExceptions: ((<class 'neutron.common.utils.WaitTimeout'>, WaitTimeout('Timed out after 60 seconds'), <traceback object at 0x7f849a0df280>), (<class 'fixtures.fixture.SetupError'>, SetupError({}), <traceback object at 0x7f849a3a9cc0>))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 207, in setUp
    raise SetupError(details)
fixtures.fixture.SetupError: {}
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 743, in wait_until_true
    eventlet.sleep(sleep)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 38, in sleep
    hub.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/fixtures/fixture.py", line 196, in setUp
    self._setUp()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/process.py", line 175, in _setUp
    common_utils.wait_until_true(self.server_is_live)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 748, in wait_until_true
    raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 60 seconds

neutron server fails as:-
2024-05-14 13:16:19.166 68768 ERROR neutron.service [None req-f40f8906-b901-4249-b984-b87293fbeb92 - - - - - -] Unrecoverable error: please check log for details.: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'openstack_citest'@'localhost' (using password: YES)")
(Background on this error at: https://sqlalche.me/e/20/e3q8)
2024-05-14 13:16:19.166 68768 ERROR neutron.service Traceback (most recent call last):
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 146, in __init__
2024-05-14 13:16:19.166 68768 ERROR neutron.service self._dbapi_connection = engine.raw_connection()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3304, in raw_connection
2024-05-14 13:16:19.166 68768 ERROR neutron.service return self.pool.connect()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 449, in connect
2024-05-14 13:16:19.166 68768 ERROR neutron.service return _ConnectionFairy._checkout(self)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1263, in _checkout
2024-05-14 13:16:19.166 68768 ERROR neutron.service fairy = _ConnectionRecord.checkout(pool)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 712, in checkout
2024-05-14 13:16:19.166 68768 ERROR neutron.service rec = pool._do_get()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 179, in _do_get
2024-05-14 13:16:19.166 68768 ERROR neutron.service with util.safe_reraise():
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
2024-05-14 13:16:19.166 68768 ERROR neutron.service raise exc_value.with_traceback(exc_tb)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 177, in _do_get
2024-05-14 13:16:19.166 68768 ERROR neutron.service return self._create_connection()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 390, in _create_connection
2024-05-14 13:16:19.166 68768 ERROR neutron.service return _ConnectionRecord(self)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 674, in __init__
2024-05-14 13:16:19.166 68768 ERROR neutron.service self.__connect()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 900, in __connect
2024-05-14 13:16:19.166 68768 ERROR neutron.service with util.safe_reraise():
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
2024-05-14 13:16:19.166 68768 ERROR neutron.service raise exc_value.with_traceback(exc_tb)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 896, in __connect
2024-05-14 13:16:19.166 68768 ERROR neutron.service self.dbapi_connection = connection = pool._invoke_creator(self)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 643, in connect
2024-05-14 13:16:19.166 68768 ERROR neutron.service return dialect.connect(*cargs, **cparams)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 617, in connect
2024-05-14 13:16:19.166 68768 ERROR neutron.service return self.loaded_dbapi.connect(*cargs, **cparams)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 358, in __init__
2024-05-14 13:16:19.166 68768 ERROR neutron.service self.connect()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 664, in connect
2024-05-14 13:16:19.166 68768 ERROR neutron.service self._request_authentication()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 976, in _request_authentication
2024-05-14 13:16:19.166 68768 ERROR neutron.service auth_packet = _auth.caching_sha2_password_auth(self, auth_packet)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/_auth.py", line 267, in caching_sha2_password_auth
2024-05-14 13:16:19.166 68768 ERROR neutron.service pkt = _roundtrip(conn, data)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/_auth.py", line 120, in _roundtrip
2024-05-14 13:16:19.166 68768 ERROR neutron.service pkt = conn._read_packet()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet
2024-05-14 13:16:19.166 68768 ERROR neutron.service packet.raise_for_error()
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
2024-05-14 13:16:19.166 68768 ERROR neutron.service err.raise_mysql_exception(self._data)
2024-05-14 13:16:19.166 68768 ERROR neutron.service File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
2024-05-14 13:16:19.166 68768 ERROR neutron.service raise errorclass(errno, errval)
2024-05-14 13:16:19.166 68768 ERROR neutron.service pymysql.err.OperationalError: (1045, "Access denied for user 'openstack_citest'@'localhost' (using password: YES)")
2024-05-14 13:16:19.166 68768 ERROR neutron.service

Example failure:- https://688891352003d7aa1ebf-eb9cf67f730d66b65f59031307d31060.ssl.cf1.rackcdn.com/919602/2/check/neutron-fullstack-with-uwsgi/94787d6/testr_results.html

Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-fullstack-with-uwsgi&project=openstack/neutron

See original description
yatin (yatinkarel)
Changed in neutron:
importance: Undecided → Critical
tags: added: fullstack gate-failure
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/919731

Revision history for this message
Mike Bayer (zzzeek) wrote :

hi -

the error ": (1045, "Access denied for user 'openstack_citest'@'localhost' (using password: YES)")" indicates a URL being used with an incorrect password. This particular problem would be caused by the issue that I have a fix for at https://review.opendev.org/c/openstack/neutron/+/913911 - as this code is part of the "fullstack" path this is likely the problem.

OpenStack Infra (hudson-openstack)
Changed in neutron:
status: New → In Progress
yatin (yatinkarel)
Changed in neutron:
assignee: nobody → Brian Haley (brian-haley)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/919594
Committed: https://opendev.org/openstack/neutron/commit/11255ede9715800e2945e0b318c93a5ac9f1e968
Submitter: "Zuul (22348)"
Branch: master

commit 11255ede9715800e2945e0b318c93a5ac9f1e968
Author: Takashi Kajinami <email address hidden>
Date: Wed May 15 00:12:22 2024 +0900

    Fix the broken neutron gate

    There are currently two issues affecting the neutron gate, one
    for pep8 job and the other for fullstack. We need to fix them
    both together since they cannot merge separately.

    Bump pylint to the latest version to fix pep8 job - it was
    pinned to an old version which is causing the job to hang.

    Do not use str(url) to stringify a URL for subsequent use
    to fix the fullstack job. The str(url) function in SQLAlchemy
    hides the password. For a URL string that is to be re-used, use
    render_as_string(hide_password=False).

    Co-authored-by: Mike Bayer <email address hidden>

    Closes-Bug: #2065779
    Change-Id: I8d04db91ddc2995fbff05b4c4c48baebcc418522

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by "<email address hidden> <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/913911
Reason: addressed in https://review.opendev.org/c/openstack/neutron/+/919594

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.