Upgrade failed with 500 Server Error: Internal Server Error on checkers for container integration_postgres_nailgun_nginx

[Tatyanka]

Steps:
Revert snapshot with neutron gre ha env
Run upgrade on master
Check that upgrade was successful
Run patching and restart containers
Run network verification
Run OSTF
Create new ha cluster with 1 controller Neutron Vlan cluster
Deploy cluster
Run OSTF

Actual:
Upgrade failed with:
http://paste.openstack.org/show/406043/

that leads to different errors in upgrade log
2015-07-28 07:52:32 ERROR 24612 (utils) HTTP request ends with 500
Traceback (most recent call last):
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/utils.py", line 848, in _wrapped
    return fn(*args, **kwargs)
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/fuel_upgrade/clients/nailgun_client.py", line 51, in get_releases
    r.raise_for_status()
  File "/var/upgrade/.fuel-upgrade-venv/lib/python2.6/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: Internal Server Error
2015-07-28 07:52:34 ERROR 24612 (utils) HTTP request ends with 500

http://paste.openstack.org/show/406043/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "99"
  build_id: "2015-07-27_15-32-27"
  nailgun_sha: "63622e9df50dc8e534834001527787fee4e56833"
  python-fuelclient_sha: "c2f8b78686de9dceca8bb9c3f655a7fbbd1375a4"
  fuel-agent_sha: "2a65f11c10b0aeb5184247635a19740fc3edde21"
  astute_sha: "34e0493afa22999c4a07d3198ceb945116ab7932"
  fuel-library_sha: "0c29bbc15cf981a4d7d2c6ef7d9c25ba9f828873"
  fuel-ostf_sha: "94a483c8aba639be3b96616c1396ef290dcc00cd"
  fuelmain_sha: "921918a3bd3d278431f35ad917989e46b0c24100"

[Ihor Kalnytskyi (ikalnytskyi)]

Assasind in Nailgun couldn't launch due to PSQL's error:

2015-07-28 10:32:55,382 DEBG 'assassind' stdout output:
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2412, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2427, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column nodes.hostname does not exist
LINE 1: ...ta, nodes.mac AS nodes_mac, nodes.ip AS nodes_ip, nodes.host...
                                                             ^
 'SELECT nodes.id AS nodes_id, nodes.uuid AS nodes_uuid, nodes.cluster_id AS nodes_cluster_id, nodes.group_id AS nodes_group_id, nodes.name AS nodes_name, nodes.status AS nodes_status, nodes.meta AS nodes_meta, nodes.mac AS nodes_mac, nodes.ip AS nodes_ip, nodes.hostname AS nodes_hostname, nodes.manufacturer AS nodes_manufacturer, nodes.platform_name AS nodes_platform_name, nodes.kernel_params AS nodes_kernel_params, nodes.progress AS nodes_progress, nodes.os_platform AS nodes_os_platform, nodes.pending_addition AS nodes_pending_addition, nodes.pending_deletion AS nodes_pending_deletion, nodes.error_type AS nodes_error_type, nodes.error_msg AS nodes_error_msg, nodes.timestamp AS nodes_timestamp, nodes.online AS nodes_online, nodes.labels AS nodes_labels, nodes.roles AS nodes_roles, nodes.pending_roles AS nodes_pending_roles, nodes.primary_roles AS nodes_primary_roles, nodes.agent_checksum AS nodes_agent_checksum, nodes.replaced_deployment_info AS nodes_replaced_deployment_info, nodes.replaced_provisioning_info AS nodes_replaced_provisioning_info, nodes.network_template AS nodes_network_template, nodes.extensions AS nodes_extensions \nFROM nodes \nWHERE nodes.status != %(status_1)s AND nodes.timestamp + %(timestamp_1)s < %(param_1)s AND nodes.online = true' {'param_1': datetime.datetime(2015, 7, 28, 10, 32, 55, 358025), 'status_1': 'provisioning', 'timestamp_1': datetime.timedelta(0, 180)}

[Ihor Kalnytskyi (ikalnytskyi)]

The error above is occured due to failure in fuel_7_0 migration:

Notice: /Stage[main]/Nailgun::Venv/Exec[nailgun_syncdb]/returns: sqlalchemy.exc.IntegrityError: (IntegrityError) could not create unique index "_hostname_cluster_uc"

2015-07-28 07:43:30,944 DEBG 'docker-nailgun' stdout output:
Notice: /Stage[main]/Nailgun::Venv/Exec[nailgun_syncdb]/returns: DETAIL: Key (cluster_id, hostname)=(1, ) is duplicated.
Notice: /Stage[main]/Nailgun::Venv/Exec[nailgun_syncdb]/returns: 'ALTER TABLE nodes ADD CONSTRAINT _hostname_cluster_uc UNIQUE (cluster_id, hostname)' {}

We should create constraint index after filling hostname.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/206971

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/206971
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=8e0e645f67c657d098343d7cbf335bc49965710d
Submitter: Jenkins
Branch: master

commit 8e0e645f67c657d098343d7cbf335bc49965710d
Author: Igor Kalnitsky <email address hidden>
Date: Wed Jul 29 13:08:28 2015 +0300

    Fix hostname unique constraint in 7.0 migration

    We have to create a unique constraint for (hostname, cluster) only when
    hostname column will be filled with proper value; otherwise we will fail
    to create it because two nodes in one cluster will fall under conflict.

    Closes-Bug: #1478892

    Change-Id: I73753314adf12374552201cb2ef26188cc8fdca9
    Signed-off-by: Igor Kalnitsky <email address hidden>

[Artem Hrechanychenko (agrechanichenko)]

Verified on fuel-7.0-upgrade-107-2015-07-29_16-13-31.tar.lrz