neutron

test_walk_versions tests fail with "IndexError: tuple index out of range" after timeout

Bug #1687027 reported by Ihar Hrachyshka
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Slawek Kaplonski

Bug Description

http://logs.openstack.org/99/460399/1/check/gate-neutron-dsvm-functional-ubuntu-xenial/25de43d/testr_results.html.gz

Traceback (most recent call last):
  File "neutron/tests/base.py", line 115, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/base.py", line 115, in func
    return f(self, *args, **kwargs)
  File "neutron/tests/functional/db/test_migrations.py", line 551, in test_walk_versions
    self._migrate_up(config, engine, dest, curr, with_data=True)
  File "neutron/tests/functional/db/test_migrations.py", line 537, in _migrate_up
    migration.do_alembic_command(config, 'upgrade', dest)
  File "neutron/db/migration/cli.py", line 109, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/command.py", line 254, in upgrade
    script.run_env()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/script/base.py", line 416, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/util/compat.py", line 75, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
    run_migrations_online()
  File "neutron/db/migration/alembic_migrations/env.py", line 114, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/runtime/environment.py", line 817, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/runtime/migration.py", line 323, in run_migrations
    step.migration_fn(**kw)
  File "/opt/stack/new/neutron/neutron/db/migration/alembic_migrations/versions/mitaka/expand/3894bccad37f_add_timestamp_to_base_resources.py", line 36, in upgrade
    sa.Column(column_name, sa.DateTime(), nullable=True)
  File "<string>", line 8, in add_column
  File "<string>", line 3, in add_column
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/ops.py", line 1551, in add_column
    return operations.invoke(op)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/base.py", line 318, in invoke
    return fn(self, operation)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 123, in add_column
    schema=schema
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 172, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1002, in _execute_ddl
    compiled
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1398, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
    result = self._query(query)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
    result.read()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
    first_packet = self.connection._read_packet()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet
    packet_header = self._read_bytes(4)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/connections.py", line 1021, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/pymysql/_socketio.py", line 59, in readinto
    return self._sock.recv_into(b)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 360, in recv_into
    return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 348, in _recv_loop
    self._read_trampoline()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 319, in _read_trampoline
    timeout_exc=socket.timeout("timed out"))
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 203, in _trampoline
    mark_as_closed=self._mark_as_closed)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline
    return hub.switch()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
    self.wait(sleep_time)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
    presult = self.do_poll(seconds)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
    return self.poll.poll(seconds)
  File "/opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
IndexError: tuple index out of range

It's interesting that ALL of them fail in the same job (something wrong with the db backend?) Counterparts for postgresql didn't fail.

The tests in question are quite long. (Tool psql test to pass in 55secs, while mysql failed after 200s+). We see that mysqld is the most intensive process during those 200secs (see dstat). Is it really just slow?

Several suggestions here:
- we should probably squash scripts, they span till Mitaka right now.
- there are no mysqld logs in functional job. Should we add those?
- there is no per-testcase log file for the tests, probably because the tests are special snowflakes that don't use common base test class that would set logdir.

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

In good runs, mysql flavours of the test work just fine, not taking much longer than postgresql.

Changed in neutron:
importance: Undecided → High
status: New → Confirmed
tags: added: functional-tests gate-failure
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Can https://bugs.launchpad.net/neutron/+bug/1693950 be related?

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

No hits in last 30 days for such error: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22IndexError%3A%20tuple%20index%20out%20of%20range%5C%22%20AND%20build_name%3Aneutron-functional

I will close it for now. If You think that it's still problem, please reopen this bug

Changed in neutron:
status: Confirmed → Incomplete
Revision history for this message
Jakub Libosvar (libosvar) wrote :

Got a fresh hit at http://logs.openstack.org/59/563159/4/check/neutron-functional/1dd8b0d/logs/testr_results.html.gz

Changed in neutron:
status: Incomplete → Confirmed
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

I think I just spotted similar issues at least twice:
http://logs.openstack.org/58/574058/10/check/neutron-functional/f38d685/logs/testr_results.html.gz
http://logs.openstack.org/61/567461/4/check/neutron-functional/81d69a4/logs/testr_results.html.gz

Miguel Lavalle (minsel)
Changed in neutron:
assignee: nobody → Miguel Lavalle (minsel)
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Another hit of this issue: http://logs.openstack.org/67/597567/21/check/neutron-functional/d28a948/logs/testr_results.html.gz

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/610003

Changed in neutron:
assignee: Miguel Lavalle (minsel) → Slawek Kaplonski (slaweq)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/610003
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c2c37272bf788aac63399127cddd487dc0f15c38
Submitter: Zuul
Branch: master

commit c2c37272bf788aac63399127cddd487dc0f15c38
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 12 12:35:10 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    In Neutron we hit quite often same issue as Manila, see [1] for
    details.
    It looks that solution for this problem may be increase timeout
    for test_walk_version functional tests.
    Higher timeout will be applied for tests for both pgsql and mysql
    backends but it is mostly needed for mysql because 'pymysql' works
    much slower on slow nodes than 'psycopg2'

    This patch adds also new decorator to set individual timeout for
    tests.

    [1] https://bugs.launchpad.net/manila/+bug/1501272

    Change-Id: I5f344af6dc3e5a6ee5f52c250b6c719e1b43e02d
    Closes-Bug: #1687027

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/611354

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/611357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/611358

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/611360

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/611354
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=026d911832256fa2f79ff808b99d578e43dd2c7f
Submitter: Zuul
Branch: stable/rocky

commit 026d911832256fa2f79ff808b99d578e43dd2c7f
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 12 12:35:10 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    In Neutron we hit quite often same issue as Manila, see [1] for
    details.
    It looks that solution for this problem may be increase timeout
    for test_walk_version functional tests.
    Higher timeout will be applied for tests for both pgsql and mysql
    backends but it is mostly needed for mysql because 'pymysql' works
    much slower on slow nodes than 'psycopg2'

    This patch adds also new decorator to set individual timeout for
    tests.

    [1] https://bugs.launchpad.net/manila/+bug/1501272

    Change-Id: I5f344af6dc3e5a6ee5f52c250b6c719e1b43e02d
    Closes-Bug: #1687027
    (cherry picked from commit c2c37272bf788aac63399127cddd487dc0f15c38)

tags: added: in-stable-rocky
tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/611357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=db07fe8698dc24b8a6b7f09a96492d1bb959cc74
Submitter: Zuul
Branch: stable/queens

commit db07fe8698dc24b8a6b7f09a96492d1bb959cc74
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 12 12:35:10 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    In Neutron we hit quite often same issue as Manila, see [1] for
    details.
    It looks that solution for this problem may be increase timeout
    for test_walk_version functional tests.
    Higher timeout will be applied for tests for both pgsql and mysql
    backends but it is mostly needed for mysql because 'pymysql' works
    much slower on slow nodes than 'psycopg2'

    This patch adds also new decorator to set individual timeout for
    tests.

    [1] https://bugs.launchpad.net/manila/+bug/1501272

    Change-Id: I5f344af6dc3e5a6ee5f52c250b6c719e1b43e02d
    Closes-Bug: #1687027
    (cherry picked from commit c2c37272bf788aac63399127cddd487dc0f15c38)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/611360
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1cb8901b4213648a7fec1978e98905c6eac8bcd9
Submitter: Zuul
Branch: stable/ocata

commit 1cb8901b4213648a7fec1978e98905c6eac8bcd9
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 12 12:35:10 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    In Neutron we hit quite often same issue as Manila, see [1] for
    details.
    It looks that solution for this problem may be increase timeout
    for test_walk_version functional tests.
    Higher timeout will be applied for tests for both pgsql and mysql
    backends but it is mostly needed for mysql because 'pymysql' works
    much slower on slow nodes than 'psycopg2'

    This patch adds also new decorator to set individual timeout for
    tests.

    [1] https://bugs.launchpad.net/manila/+bug/1501272

    Change-Id: I5f344af6dc3e5a6ee5f52c250b6c719e1b43e02d
    Closes-Bug: #1687027
    (cherry picked from commit c2c37272bf788aac63399127cddd487dc0f15c38)

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/611358
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f26eed6677965de51fe8d88f9750c1b6a5becce5
Submitter: Zuul
Branch: stable/pike

commit f26eed6677965de51fe8d88f9750c1b6a5becce5
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 12 12:35:10 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    In Neutron we hit quite often same issue as Manila, see [1] for
    details.
    It looks that solution for this problem may be increase timeout
    for test_walk_version functional tests.
    Higher timeout will be applied for tests for both pgsql and mysql
    backends but it is mostly needed for mysql because 'pymysql' works
    much slower on slow nodes than 'psycopg2'

    This patch adds also new decorator to set individual timeout for
    tests.

    [1] https://bugs.launchpad.net/manila/+bug/1501272

    Change-Id: I5f344af6dc3e5a6ee5f52c250b6c719e1b43e02d
    Closes-Bug: #1687027
    (cherry picked from commit c2c37272bf788aac63399127cddd487dc0f15c38)

tags: added: in-stable-pike
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

It looks that in some cases such test can take longer than 300 seconds and there are still failures there. See: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20test_walk_versions%5C%22%20AND%20filename%3A%5C%22job-output.txt%5C%22

Changed in neutron:
status: Fix Released → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/612505

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/612505
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2
Submitter: Zuul
Branch: master

commit c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 22 22:58:48 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    Patch [1] increased timeouts for test_walk_version functional tests
    for MySQL backend to 300 seconds to avoid failures due to timeouts.

    Unfortunately it looks that on nodes from some cloud providers used
    in the gate and with number of migration scripts which we have in
    Neutron those tests can take sometimes even around 400 seconds.

    So lets increase this to 600 seconds to avoid such failures of
    functional tests job.

    [1] https://review.openstack.org/#/c/610003/

    Change-Id: I9d129f0e90a072ec980aadabb2c6b812c08e1618
    Closes-Bug: #1687027

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/613230

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/613231

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/613232

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/613233

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/613232
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3eaaf0756cd1a422f03ebdfe2d8a55bb69e115e7
Submitter: Zuul
Branch: stable/pike

commit 3eaaf0756cd1a422f03ebdfe2d8a55bb69e115e7
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 22 22:58:48 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    Patch [1] increased timeouts for test_walk_version functional tests
    for MySQL backend to 300 seconds to avoid failures due to timeouts.

    Unfortunately it looks that on nodes from some cloud providers used
    in the gate and with number of migration scripts which we have in
    Neutron those tests can take sometimes even around 400 seconds.

    So lets increase this to 600 seconds to avoid such failures of
    functional tests job.

    [1] https://review.openstack.org/#/c/610003/

    Change-Id: I9d129f0e90a072ec980aadabb2c6b812c08e1618
    Closes-Bug: #1687027
    (cherry picked from commit c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/613231
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6a9450b2036808748668a3439fdcebb77f071c5f
Submitter: Zuul
Branch: stable/queens

commit 6a9450b2036808748668a3439fdcebb77f071c5f
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 22 22:58:48 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    Patch [1] increased timeouts for test_walk_version functional tests
    for MySQL backend to 300 seconds to avoid failures due to timeouts.

    Unfortunately it looks that on nodes from some cloud providers used
    in the gate and with number of migration scripts which we have in
    Neutron those tests can take sometimes even around 400 seconds.

    So lets increase this to 600 seconds to avoid such failures of
    functional tests job.

    [1] https://review.openstack.org/#/c/610003/

    Change-Id: I9d129f0e90a072ec980aadabb2c6b812c08e1618
    Closes-Bug: #1687027
    (cherry picked from commit c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/613233
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1a3cc6e810e02a5b0aaa3bf948aebb1f29cf8f80
Submitter: Zuul
Branch: stable/ocata

commit 1a3cc6e810e02a5b0aaa3bf948aebb1f29cf8f80
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 22 22:58:48 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    Patch [1] increased timeouts for test_walk_version functional tests
    for MySQL backend to 300 seconds to avoid failures due to timeouts.

    Unfortunately it looks that on nodes from some cloud providers used
    in the gate and with number of migration scripts which we have in
    Neutron those tests can take sometimes even around 400 seconds.

    So lets increase this to 600 seconds to avoid such failures of
    functional tests job.

    [1] https://review.openstack.org/#/c/610003/

    Change-Id: I9d129f0e90a072ec980aadabb2c6b812c08e1618
    Closes-Bug: #1687027
    (cherry picked from commit c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2)

Revision history for this message
Hongbin Lu (hongbin.lu) wrote :

Still happening: http://logs.openstack.org/88/555088/29/check/neutron-functional/0b00b31/job-output.txt.gz#_2018-10-25_22_18_28_107639

Changed in neutron:
status: Fix Released → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/613230
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=63ebee37bbcf8b780210b7d923c5ce4f1591c345
Submitter: Zuul
Branch: stable/rocky

commit 63ebee37bbcf8b780210b7d923c5ce4f1591c345
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 22 22:58:48 2018 +0200

    [Functional] Increase test_timeout for db migration tests

    Patch [1] increased timeouts for test_walk_version functional tests
    for MySQL backend to 300 seconds to avoid failures due to timeouts.

    Unfortunately it looks that on nodes from some cloud providers used
    in the gate and with number of migration scripts which we have in
    Neutron those tests can take sometimes even around 400 seconds.

    So lets increase this to 600 seconds to avoid such failures of
    functional tests job.

    [1] https://review.openstack.org/#/c/610003/

    Change-Id: I9d129f0e90a072ec980aadabb2c6b812c08e1618
    Closes-Bug: #1687027
    (cherry picked from commit c39afbd5fcdfff2ad5e7f6963768da9f3d8027d2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.2

This issue was fixed in the openstack/neutron 13.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.6

This issue was fixed in the openstack/neutron 11.0.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.0.5

This issue was fixed in the openstack/neutron 12.0.5 release.

Slawek Kaplonski (slaweq)
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/624685

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/624685
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b75cf8743cb4231c3a9bf9a34a58a083924ccf3b
Submitter: Zuul
Branch: master

commit b75cf8743cb4231c3a9bf9a34a58a083924ccf3b
Author: Slawek Kaplonski <email address hidden>
Date: Wed Dec 12 13:24:52 2018 +0100

    Mark mysql related functional tests as unstable

    Thos tests are failing quite often because of timeouts. Details
    are in related bug report.
    Lets make our life easier and mark them as unstable until we will
    figure out how to fix this issue.

    Change-Id: I47743e519c41795bba64e4da041a87bffd947fbd
    Related-Bug: #1687027

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/625556

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/625556
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0a4d2ee87721da4472099c1043c0146a9e02ee6d
Submitter: Zuul
Branch: master

commit 0a4d2ee87721da4472099c1043c0146a9e02ee6d
Author: Slawek Kaplonski <email address hidden>
Date: Mon Dec 17 12:20:48 2018 +0100

    Fix mysql functional migration tests

    In patch [1] I marked all MySQL related migration tests
    as unstable but I made mistake with calling super() method in
    tests from TestModelsMigrationsMysql class.
    This wasn't catched as tests were skipped instead of fail.

    This patch fixes that, so tests can be run properly and be skipped
    only if "real" issue with timeout will happen.

    [1] https://review.openstack.org/#/c/624685/

    Change-Id: Id42d6d8e24fdea5ac83f0fecc6975c19b933a501
    Related-Bug: #1687027

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.0.0b1

This issue was fixed in the openstack/neutron 14.0.0.0b1 development milestone.

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Today I checked one failed test, from http://logs.openstack.org/30/626830/9/check/neutron-functional/e66470c/logs/dsvm-functional-logs/neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql.test_check_mysql_engine.txt.gz and compared it with same test which passed: http://logs.openstack.org/85/624685/2/check/neutron-functional/fede00f/logs/dsvm-functional-logs/neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql.test_check_mysql_engine.txt.gz

There is no some single migration step which takes long time or hangs. It is just very slow all time in case of failed test. E.g. first step, called "Running upgrade -> kilo" took 6 minutes (!) in failed test and 2.5 second in good run.

Comparing dstat from those 2 runs I can see that in this failed test there was less than 100 IOPS for most of the time. In case of this passed test it was around 2k - 3k IOPS.
So I think that this issue is just because of some slow VMs which maybe are on some overloaded nodes.

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
tags: added: neutron-easy-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/629420

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/636892

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/629420
Reason: Let's try something different: https://review.openstack.org/636892

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/636892
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c0fec676723649a0516cf3d4af0dccc0fe832095
Submitter: Zuul
Branch: master

commit c0fec676723649a0516cf3d4af0dccc0fe832095
Author: Slawek Kaplonski <email address hidden>
Date: Thu Feb 14 11:08:19 2019 +0100

    Add new test decorator skip_if_timeout

    In some cases our db migration tests which run on MySQL are
    failing with timeout and it happens due to slow VMs on which
    job is running.
    Sometimes it may also happen that timeout exception is raised
    in the middle of some sqlalchemy operations and
    sqlalchemy.InterfaceError is raised as last one.
    Details about this exception can be found in [1].

    To avoid many rechecks because of this reason this patch
    introduces new decorator which is very similar to "unstable_test"
    but will skip test only if one of exceptions mentioned above will
    be raised.
    In all other cases it will fail test.

    That should be a bit more safe for us because we will not miss
    some other failures raised in those tests and will avoid rechecks
    because of this "well-known" reason described in related bug.

    [1] http://sqlalche.me/e/rvf5

    Change-Id: Ie291fda7d23a696aaa1160d126a3cf72b08c522f
    Related-Bug: #1687027

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/638116

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/638117

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/638120

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/638121

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/638120
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8b255a648c381bfcff0623ba97880855c7fa21eb
Submitter: Zuul
Branch: stable/pike

commit 8b255a648c381bfcff0623ba97880855c7fa21eb
Author: Slawek Kaplonski <email address hidden>
Date: Thu Feb 14 11:08:19 2019 +0100

    Add new test decorator skip_if_timeout

    In some cases our db migration tests which run on MySQL are
    failing with timeout and it happens due to slow VMs on which
    job is running.
    Sometimes it may also happen that timeout exception is raised
    in the middle of some sqlalchemy operations and
    sqlalchemy.InterfaceError is raised as last one.
    Details about this exception can be found in [1].

    To avoid many rechecks because of this reason this patch
    introduces new decorator which is very similar to "unstable_test"
    but will skip test only if one of exceptions mentioned above will
    be raised.
    In all other cases it will fail test.

    That should be a bit more safe for us because we will not miss
    some other failures raised in those tests and will avoid rechecks
    because of this "well-known" reason described in related bug.

    [1] http://sqlalche.me/e/rvf5

    Conflicts:
        neutron/tests/functional/db/test_migrations.py
        neutron/tests/base.py

    Change-Id: Ie291fda7d23a696aaa1160d126a3cf72b08c522f
    Related-Bug: #1687027
    (cherry picked from commit c0fec676723649a0516cf3d4af0dccc0fe832095)
    (cherry picked from commit e6f22ce81c0a1130d45d290b185b736501e6dd1e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/638117
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8bf3a905e7d05715a913587a9077615a733e5ec1
Submitter: Zuul
Branch: stable/queens

commit 8bf3a905e7d05715a913587a9077615a733e5ec1
Author: Slawek Kaplonski <email address hidden>
Date: Thu Feb 14 11:08:19 2019 +0100

    Add new test decorator skip_if_timeout

    In some cases our db migration tests which run on MySQL are
    failing with timeout and it happens due to slow VMs on which
    job is running.
    Sometimes it may also happen that timeout exception is raised
    in the middle of some sqlalchemy operations and
    sqlalchemy.InterfaceError is raised as last one.
    Details about this exception can be found in [1].

    To avoid many rechecks because of this reason this patch
    introduces new decorator which is very similar to "unstable_test"
    but will skip test only if one of exceptions mentioned above will
    be raised.
    In all other cases it will fail test.

    That should be a bit more safe for us because we will not miss
    some other failures raised in those tests and will avoid rechecks
    because of this "well-known" reason described in related bug.

    [1] http://sqlalche.me/e/rvf5

    Conflicts:
        neutron/tests/functional/db/test_migrations.py

    Change-Id: Ie291fda7d23a696aaa1160d126a3cf72b08c522f
    Related-Bug: #1687027
    (cherry picked from commit c0fec676723649a0516cf3d4af0dccc0fe832095)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/638116
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e6f22ce81c0a1130d45d290b185b736501e6dd1e
Submitter: Zuul
Branch: stable/rocky

commit e6f22ce81c0a1130d45d290b185b736501e6dd1e
Author: Slawek Kaplonski <email address hidden>
Date: Thu Feb 14 11:08:19 2019 +0100

    Add new test decorator skip_if_timeout

    In some cases our db migration tests which run on MySQL are
    failing with timeout and it happens due to slow VMs on which
    job is running.
    Sometimes it may also happen that timeout exception is raised
    in the middle of some sqlalchemy operations and
    sqlalchemy.InterfaceError is raised as last one.
    Details about this exception can be found in [1].

    To avoid many rechecks because of this reason this patch
    introduces new decorator which is very similar to "unstable_test"
    but will skip test only if one of exceptions mentioned above will
    be raised.
    In all other cases it will fail test.

    That should be a bit more safe for us because we will not miss
    some other failures raised in those tests and will avoid rechecks
    because of this "well-known" reason described in related bug.

    [1] http://sqlalche.me/e/rvf5

    Conflicts:
        neutron/tests/functional/db/test_migrations.py

    Change-Id: Ie291fda7d23a696aaa1160d126a3cf72b08c522f
    Related-Bug: #1687027
    (cherry picked from commit c0fec676723649a0516cf3d4af0dccc0fe832095)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/638121
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=55daaeca3e94aab0c27b82e59c79f182dd3ec1ea
Submitter: Zuul
Branch: stable/ocata

commit 55daaeca3e94aab0c27b82e59c79f182dd3ec1ea
Author: Slawek Kaplonski <email address hidden>
Date: Thu Feb 14 11:08:19 2019 +0100

    Add new test decorator skip_if_timeout

    In some cases our db migration tests which run on MySQL are
    failing with timeout and it happens due to slow VMs on which
    job is running.
    Sometimes it may also happen that timeout exception is raised
    in the middle of some sqlalchemy operations and
    sqlalchemy.InterfaceError is raised as last one.
    Details about this exception can be found in [1].

    To avoid many rechecks because of this reason this patch
    introduces new decorator which is very similar to "unstable_test"
    but will skip test only if one of exceptions mentioned above will
    be raised.
    In all other cases it will fail test.

    That should be a bit more safe for us because we will not miss
    some other failures raised in those tests and will avoid rechecks
    because of this "well-known" reason described in related bug.

    [1] http://sqlalche.me/e/rvf5

    Conflicts:
        neutron/tests/functional/db/test_migrations.py
        neutron/tests/base.py

    Change-Id: Ie291fda7d23a696aaa1160d126a3cf72b08c522f
    Related-Bug: #1687027
    (cherry picked from commit c0fec676723649a0516cf3d4af0dccc0fe832095)
    (cherry picked from commit e6f22ce81c0a1130d45d290b185b736501e6dd1e)

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

With last patch https://review.openstack.org/636892 I think we mitigate this issue as much as we could and we can now mark it as fixed.

Changed in neutron:
status: Confirmed → Fix Committed
status: Fix Committed → Fix Released
tags: removed: neutron-easy-proactive-backport-potential neutron-proactive-backport-potential
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Even with added recently skip_if_timeout decorator those tests are failing from time to time and it looks that it may happen that different type of exception is raised in such case.
For example: http://logs.openstack.org/73/636473/25/check/neutron-functional-python27/0ef9362/testr_results.html.gz

We will have to handle more exception types in this decorator.

Changed in neutron:
status: Fix Released → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/651371

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/651371
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b7458b615972dc6ff93a12abbd0a8f32e8da55eb
Submitter: Zuul
Branch: master

commit b7458b615972dc6ff93a12abbd0a8f32e8da55eb
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 22:04:36 2019 +0200

    Handle DBConnectionError in skip_if_timeout decorator

    In some cases it may happen that when db test will fail due
    to timeout oslo_db.exception.DBConnectionError will be raised
    instead of sqlalchemy_exc.InterfaceError.
    This patch adds handling such case in skip_if_timeout decorator.

    Change-Id: I7350d5c884784317c94ff42f28526065ff399b40
    Related-Bug: #1687027

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/651704

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/651705

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/651706

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/651707

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/stein)

Reviewed: https://review.openstack.org/651704
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=56974308424728637ab0303198036022c8568309
Submitter: Zuul
Branch: stable/stein

commit 56974308424728637ab0303198036022c8568309
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 22:04:36 2019 +0200

    Handle DBConnectionError in skip_if_timeout decorator

    In some cases it may happen that when db test will fail due
    to timeout oslo_db.exception.DBConnectionError will be raised
    instead of sqlalchemy_exc.InterfaceError.
    This patch adds handling such case in skip_if_timeout decorator.

    Change-Id: I7350d5c884784317c94ff42f28526065ff399b40
    Related-Bug: #1687027
    (cherry picked from commit b7458b615972dc6ff93a12abbd0a8f32e8da55eb)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/651705
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=14ed5515b9a3fbc01d03270dfcc210586e781bc4
Submitter: Zuul
Branch: stable/rocky

commit 14ed5515b9a3fbc01d03270dfcc210586e781bc4
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 22:04:36 2019 +0200

    Handle DBConnectionError in skip_if_timeout decorator

    In some cases it may happen that when db test will fail due
    to timeout oslo_db.exception.DBConnectionError will be raised
    instead of sqlalchemy_exc.InterfaceError.
    This patch adds handling such case in skip_if_timeout decorator.

    Change-Id: I7350d5c884784317c94ff42f28526065ff399b40
    Related-Bug: #1687027
    (cherry picked from commit b7458b615972dc6ff93a12abbd0a8f32e8da55eb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/651706
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3d434814ef52a1393f8eb0f33672ae51188f7178
Submitter: Zuul
Branch: stable/queens

commit 3d434814ef52a1393f8eb0f33672ae51188f7178
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 22:04:36 2019 +0200

    Handle DBConnectionError in skip_if_timeout decorator

    In some cases it may happen that when db test will fail due
    to timeout oslo_db.exception.DBConnectionError will be raised
    instead of sqlalchemy_exc.InterfaceError.
    This patch adds handling such case in skip_if_timeout decorator.

    Change-Id: I7350d5c884784317c94ff42f28526065ff399b40
    Related-Bug: #1687027
    (cherry picked from commit b7458b615972dc6ff93a12abbd0a8f32e8da55eb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/651707
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d6189d3bc32a130b4941ff3e1333d42d4e7f1b13
Submitter: Zuul
Branch: stable/pike

commit d6189d3bc32a130b4941ff3e1333d42d4e7f1b13
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 9 22:04:36 2019 +0200

    Handle DBConnectionError in skip_if_timeout decorator

    In some cases it may happen that when db test will fail due
    to timeout oslo_db.exception.DBConnectionError will be raised
    instead of sqlalchemy_exc.InterfaceError.
    This patch adds handling such case in skip_if_timeout decorator.

    Change-Id: I7350d5c884784317c94ff42f28526065ff399b40
    Related-Bug: #1687027
    (cherry picked from commit b7458b615972dc6ff93a12abbd0a8f32e8da55eb)

Slawek Kaplonski (slaweq)
Changed in neutron:
status: Confirmed → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/657813

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/657813
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d83bb4f160ed7b85a6be8ae490dba1d9ec8c5561
Submitter: Zuul
Branch: master

commit d83bb4f160ed7b85a6be8ae490dba1d9ec8c5561
Author: Slawek Kaplonski <email address hidden>
Date: Wed May 8 16:05:51 2019 +0200

    Add skip_if_timeout to additional MySQL migration test

    Test test_forbid_offline_migrations_starting_newton from
    neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql
    module can also fail because of db timeouts when are running on
    overloaded test node.
    To not fail job because of that, this patch adds skip_if_timeout
    decorator to this additional test.

    Change-Id: Ia255d331cbb24e1fdd12212580e85423da242eae
    Related-Bug: #1687027

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
tags: added: neutron-easy-proactive-backport-potential
Slawek Kaplonski (slaweq)
tags: removed: neutron-easy-proactive-backport-potential neutron-proactive-backport-potential
Slawek Kaplonski (slaweq)
Changed in neutron:
status: Fix Committed → Fix Released
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Related patch: https://review.opendev.org/c/openstack/neutron/+/784889 (backported up to Train)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/786026
Committed: https://opendev.org/openstack/neutron/commit/14eba92cee75d1b69360570009abb2a9fec66504
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 14eba92cee75d1b69360570009abb2a9fec66504
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 9 07:29:58 2021 +0000

    Group execution of SQL functional tests

    Functional SQL tests (MySQL and PostgreSQL) use the DB installed
    backend. To avoid overloading the DB engine, the test cases are
    grouped [1] together and executed by the same worker process. That
    will ensure those tests are not executed in parallel, interfering
    each other.

    Classes that are grouped:
    - TestModelsMigrationsMysql
    - TestModelsMigrationsPsql

    [1]https://stestr.readthedocs.io/en/latest/MANUAL.html#group-regex

    Related-Bug: #1687027

    Change-Id: If04e360c01cdb16e956d16357e2181db55445137
    (cherry picked from commit 7326fd622f2c3cb36681136d79e39e4efdda5411)

tags: added: in-stable-wallaby
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/786027
Committed: https://opendev.org/openstack/neutron/commit/cfeafa5ae7f473438a4d59408733203916d4cff8
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit cfeafa5ae7f473438a4d59408733203916d4cff8
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 9 07:29:58 2021 +0000

    Group execution of SQL functional tests

    Functional SQL tests (MySQL and PostgreSQL) use the DB installed
    backend. To avoid overloading the DB engine, the test cases are
    grouped [1] together and executed by the same worker process. That
    will ensure those tests are not executed in parallel, interfering
    each other.

    Classes that are grouped:
    - TestModelsMigrationsMysql
    - TestModelsMigrationsPsql

    [1]https://stestr.readthedocs.io/en/latest/MANUAL.html#group-regex

    Related-Bug: #1687027

    Change-Id: If04e360c01cdb16e956d16357e2181db55445137
    (cherry picked from commit 7326fd622f2c3cb36681136d79e39e4efdda5411)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/786028
Committed: https://opendev.org/openstack/neutron/commit/c5d24b4b446bf89cfef67f04af084691126c7f7e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit c5d24b4b446bf89cfef67f04af084691126c7f7e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 9 07:29:58 2021 +0000

    Group execution of SQL functional tests

    Functional SQL tests (MySQL and PostgreSQL) use the DB installed
    backend. To avoid overloading the DB engine, the test cases are
    grouped [1] together and executed by the same worker process. That
    will ensure those tests are not executed in parallel, interfering
    each other.

    Classes that are grouped:
    - TestModelsMigrationsMysql
    - TestModelsMigrationsPsql

    [1]https://stestr.readthedocs.io/en/latest/MANUAL.html#group-regex

    Related-Bug: #1687027

    Change-Id: If04e360c01cdb16e956d16357e2181db55445137
    (cherry picked from commit 7326fd622f2c3cb36681136d79e39e4efdda5411)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/786029
Committed: https://opendev.org/openstack/neutron/commit/6938fa1384489b329ef343228e18c784d2c0bdf8
Submitter: "Zuul (22348)"
Branch: stable/train

commit 6938fa1384489b329ef343228e18c784d2c0bdf8
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 9 07:29:58 2021 +0000

    Group execution of SQL functional tests

    Functional SQL tests (MySQL and PostgreSQL) use the DB installed
    backend. To avoid overloading the DB engine, the test cases are
    grouped [1] together and executed by the same worker process. That
    will ensure those tests are not executed in parallel, interfering
    each other.

    Classes that are grouped:
    - TestModelsMigrationsMysql
    - TestModelsMigrationsPsql

    [1]https://stestr.readthedocs.io/en/latest/MANUAL.html#group-regex

    Related-Bug: #1687027

    Change-Id: If04e360c01cdb16e956d16357e2181db55445137
    (cherry picked from commit 7326fd622f2c3cb36681136d79e39e4efdda5411)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron ocata-eol

This issue was fixed in the openstack/neutron ocata-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/792953

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795059

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/792953
Committed: https://opendev.org/openstack/neutron/commit/d0b16473d843bde923d74837d4e24911cba14642
Submitter: "Zuul (22348)"
Branch: master

commit d0b16473d843bde923d74837d4e24911cba14642
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue May 25 11:34:56 2021 +0000

    [FT] Remove skip guard from SQL tests

    Since [1], TestModelsMigrations* tests execution time was reduced
    significantly. The test skip guard in case of timeout is not needed
    anymore.

    Related-Bug: #1687027

    [1]https://review.opendev.org/c/openstack/neutron/+/786029

    Change-Id: I19a800a475692bc6e64f98d2eaaaae035e4fbfb6

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Bernard Cafarelli (bcafarel)
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/795059
Committed: https://opendev.org/openstack/neutron/commit/a1ca4980dada08fa9119ddefe439d616f4405fd1
Submitter: "Zuul (22348)"
Branch: master

commit a1ca4980dada08fa9119ddefe439d616f4405fd1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 7 08:24:52 2021 +0000

    Remove timeout skip guard for LP#1687027

    Since [1], SQL tests do not fail because of a timeout.

    Closes-Bug: #1687027

    [1]https://review.opendev.org/c/openstack/neutron/+/785568

    Change-Id: If8386dd40aa86a0a9a25d859cc66b7ac1fa03ba6

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Slawek Kaplonski (slaweq)
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.0.0.0rc1

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/831409

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/831409
Committed: https://opendev.org/openstack/neutron/commit/4a66adf94c4e01ca92dfe12cf8646d11acfc7176
Submitter: "Zuul (22348)"
Branch: master

commit 4a66adf94c4e01ca92dfe12cf8646d11acfc7176
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Feb 11 17:43:34 2022 +0000

    Group execution of SQL functional tests

    Functional SQL tests (MySQL and PostgreSQL) use the DB installed
    backend. To avoid overloading the DB engine, the test cases are
    grouped [1] together and executed by the same worker process.
    That will ensure those tests are not executed in parallel,
    interfering each other.

    All test classes ended in "MySQL" or "PostgreSQL" are now grouped.
    That means the tests implemented inside the same class are
    executed in the same worker, not in parallel. **HOWEVER**, that
    doesn't guarantee all test cases using the same database backend
    will run in the same worker serially. The FT job can still execute
    test cases from different classes, using the same DB backend,
    in two workers at the same time.

    This patch also fixes the ``TestReservation*`` classes to inherit
    from the parent classes in the correct order. That sets the
    "DRIVER" value to the expected one "mysql" or "postgresql".

    [1]https://review.opendev.org/c/openstack/neutron/+/785568

    Closes-Bug: #1962594
    Related-Bug: #1687027
    Change-Id: I6bca7daf4d3e46071686bb62d5988ae314af8e7e

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.