test_migrations.TestBaremetalMigrations.test_postgresql_opportunistically failure

[Hans Lindgren]

Unit test gate failed on the following tests:

nova.tests.test_migrations.TestBaremetalMigrations.test_postgresql_opportunistically
nova.tests.test_migrations.TestNovaMigrations.test_postgresql_opportunistically

Trace on both tests was:

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 179, in test_postgresql_opportunistically
    self._test_postgresql_opportunistically()
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 358, in _test_postgresql_opportunistically
    self._reset_databases()
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 305, in _reset_databases
    self._reset_pg(conn_pieces)
  File "/home/jenkins/workspace/gate-nova-python27/nova/openstack/common/lockutils.py", line 228, in inner
    retval = f(*args, **kwargs)
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 275, in _reset_pg
    self.execute_cmd(createtable)
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 255, in execute_cmd
    "Failed to run: %s\n%s" % (cmd, output))
MismatchError: 0 != 256: Failed to run: psql -w -U openstack_citest -h localhost -c 'create database openstack_citest;' -d template1
ERROR: source database "template1" is being accessed by other users
DETAIL: There are 1 other session(s) using the database.

See http://logs.openstack.org/24854/1/check/gate-nova-python27/19099/

[Boris Pavlovic (boris-42)]

Hm it should be fixed by this patch https://review.openstack.org/#/c/22983/

[Hans Lindgren (hanlind)]

Well, my change was based on a later commit (07694d4ffa2b64b7cb21d76af70b8f0df1166915) that surely has the referenced fix. Looks like that patch did not fully eliminate this condition.

[Hans Lindgren (hanlind)]

Same problem happening again here: http://logs.openstack.org/28503/2/gate/gate-nova-python27/22881/

Slightly different error output this time as it references 'database "openstack_citest"' compared to previously 'source database "template1"'.

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 240, in tearDown
    self._reset_databases()
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 297, in _reset_databases
    self._reset_pg(conn_pieces)
  File "/home/jenkins/workspace/gate-nova-python27/nova/openstack/common/lockutils.py", line 232, in inner
    retval = f(*args, **kwargs)
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 263, in _reset_pg
    self.execute_cmd(droptable)
  File "/home/jenkins/workspace/gate-nova-python27/nova/tests/test_migrations.py", line 247, in execute_cmd
    "Failed to run: %s\n%s" % (cmd, output))
MismatchError: 0 != 256: Failed to run: psql -w -U openstack_citest -h localhost -c 'drop database if exists openstack_citest;' -d template1
ERROR: database "openstack_citest" is being accessed by other users
DETAIL: There are 1 other session(s) using the database.

[Viktor Serhieiev (vsergeyev)]

Is it still actual? If yes, could you please provide steps to reproduce.

[Hans Lindgren (hanlind)]

As stated above, I have only seen this during gating (last time about two weeks ago).

I have no steps to reproduce.

[Attila Fazekas (afazekas)]

Logstash for similar type of issues:
message: "database" AND message: "is being accessed by other users" AND filename:"console.html"
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcImRhdGFiYXNlXCIgQU5EIG1lc3NhZ2U6IFwiaXMgYmVpbmcgYWNjZXNzZWQgYnkgb3RoZXIgdXNlcnNcIiBBTkQgZmlsZW5hbWU6XCJjb25zb2xlLmh0bWxcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMTQ0ODg5NTIyMiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

[Jeremy Stanley (fungi)]

http://logs.openstack.org/57/97757/1/gate/gate-nova-python26/663d4c3/console.html#_2014-06-07_05_17_25_339