OpenStack-Ansible

os_nova cell_v2 discover failure

Bug #1752540 reported by git-harry on 2018-03-01

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack-Ansible	Fix Released	Undecided	git-harry

Bug Description

There appears to be an intermittent race condition with the use of nova-manage that exists when tasks are delegated to the same conductor host. This was discovered in stable/pike but the same delegation code appears to exist in master.

https://github.com/openstack/openstack-ansible-os_nova/blob/c11f8230514d5c2ae577a6af033253dd7099d097/tasks/main.yml#L128-L143

https://github.com/openstack/openstack-ansible-os_nova/blob/c11f8230514d5c2ae577a6af033253dd7099d097/tasks/nova_db_post_setup.yml

Successful run:
The included task file is delegated to different conductor hosts for each compute host and the set of tasks is run in serial.
-----------------------------------------------------
TASK [os_nova : include] *******************************************************
Friday 23 February 2018 12:08:35 -0600 (0:00:00.116) 2:08:27.021 *******
included: /etc/ansible/roles/os_nova/tasks/nova_db_post_setup.yml for compute1
included: /etc/ansible/roles/os_nova/tasks/nova_db_post_setup.yml for compute2

TASK [os_nova : Set the delegated task facts] **********************************
Friday 23 February 2018 12:08:35 -0600 (0:00:00.368) 2:08:27.390 *******
ok: [compute1 -> 10.0.238.221]

TASK [os_nova : Perform a cell_v2 discover] ************************************
Friday 23 February 2018 12:08:37 -0600 (0:00:01.184) 2:08:28.574 *******
ok: [compute1 -> 10.0.238.221]

TASK [os_nova : Get UUID of new Nova Cell] *************************************
Friday 23 February 2018 12:08:43 -0600 (0:00:06.405) 2:08:34.980 *******
ok: [compute1 -> 10.0.238.221]

TASK [os_nova : Map instances to new Cell1] ************************************
Friday 23 February 2018 12:08:49 -0600 (0:00:05.900) 2:08:40.881 *******
ok: [compute1 -> 10.0.238.221]

TASK [os_nova : Set the delegated task facts] **********************************
Friday 23 February 2018 12:08:55 -0600 (0:00:06.039) 2:08:46.920 *******
ok: [compute2 -> 10.0.237.64]

TASK [os_nova : Perform a cell_v2 discover] ************************************
Friday 23 February 2018 12:08:56 -0600 (0:00:01.145) 2:08:48.065 *******
ok: [compute2 -> 10.0.237.64]
-----------------------------------------------------

Failed run:
The included task file is delegated to the same conductor host for the two compute hosts and the tasks are run in parallel.
-----------------------------------------------------
TASK [os_nova : include] *******************************************************
Thursday 22 February 2018 12:09:43 -0600 (0:00:00.142) 2:11:19.853 *****
included: /etc/ansible/roles/os_nova/tasks/nova_db_post_setup.yml for compute1, compute2

TASK [os_nova : Set the delegated task facts] **********************************
Thursday 22 February 2018 12:09:43 -0600 (0:00:00.320) 2:11:20.174 *****
ok: [compute1 -> 10.0.237.173]
ok: [compute2 -> 10.0.237.173]

TASK [os_nova : Perform a cell_v2 discover] ************************************
Thursday 22 February 2018 12:09:44 -0600 (0:00:01.295) 2:11:21.470 *****
fatal: [compute1 -> 10.0.237.173]: FAILED! => {"changed": false, "cmd": ["/openstack/venvs/nova-r16.1.0/bin/nova-manage", "cell_v2", "discover_hosts"], "delta": "0:00:05.240976", "end": "2018-02-22 12:09:51.292287", "failed": true, "rc": 1, "start": "2018-02-22 12:09:46.051311", "stderr": "", "stderr_lines": [], "stdout": "An error has occurred:\nTraceback (most recent call last):\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/cmd/manage.py\", line 1858, in main\n ret = fn(*fn_args, **fn_kwargs)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/cmd/manage.py\", line 1583, in discover_hosts\n hosts = host_mapping_obj.discover_hosts(ctxt, cell_uuid, status_fn)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 229, in discover_hosts\n status_fn)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 185, in _check_and_create_host_mappings\n host_mapping.create()\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/oslo_versionedobjects/base.py\", line 226, in wrapper\n return fn(self, *args, **kwargs)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 114, in create\n db_mapping = self._create_in_db(self._context, changes)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 979, in wrapper\n return fn(*args, **kwargs)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 107, in _create_in_db\n return _apply_updates(context, db_mapping, updates)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 33, in _apply_updates\n db_mapping.save(context.session)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py\", line 50, in save\n session.flush()\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2171, in flush\n self._flush(objects)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2291, in _flush\n transaction.rollback(_capture_exception=True)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py\", line 66, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2255, in _flush\n flush_context.execute()\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 389, in execute\n rec.execute(self)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 548, in execute\n uow\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 181, in save_obj\n mapper, table, insert)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 835, in _emit_insert_statements\n execute(statement, params)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 945, in execute\n return meth(self, multiparams, params)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/sql/elements.py\", line 263, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1053, in _execute_clauseelement\n compiled_sql, distilled_params\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1189, in _execute_context\n context)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1398, in _handle_dbapi_exception\n util.raise_from_cause(newraise, exc_info)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1182, in _execute_context\n context)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/default.py\", line 470, in do_execute\n cursor.execute(statement, parameters)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/cursors.py\", line 166, in execute\n result = self._query(query)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/cursors.py\", line 322, in _query\n conn.query(q)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 856, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 1057, in _read_query_result\n result.read()\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 1340, in read\n first_packet = self.connection._read_packet()\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 1014, in _read_packet\n packet.check_error()\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 393, in check_error\n err.raise_mysql_exception(self._data)\n File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/err.py\", line 107, in raise_mysql_exception\n raise errorclass(errno, errval)\nDBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u\"Duplicate entry 'compute2' for key 'uniq_host_mappings0host'\") [SQL: u'INSERT INTO host_mappings (created_at, updated_at, cell_id, host) VALUES (%(created_at)s, %(updated_at)s, %(cell_id)s, %(host)s)'] [parameters: {'created_at': datetime.datetime(2018, 2, 22, 18, 9, 51, 170519), 'cell_id': 7, 'host': u'compute2', 'updated_at': None}]", "stdout_lines": ["An error has occurred:", "Traceback (most recent call last):", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/cmd/manage.py\", line 1858, in main", " ret = fn(*fn_args, **fn_kwargs)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/cmd/manage.py\", line 1583, in discover_hosts", " hosts = host_mapping_obj.discover_hosts(ctxt, cell_uuid, status_fn)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 229, in discover_hosts", " status_fn)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 185, in _check_and_create_host_mappings", " host_mapping.create()", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/oslo_versionedobjects/base.py\", line 226, in wrapper", " return fn(self, *args, **kwargs)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 114, in create", " db_mapping = self._create_in_db(self._context, changes)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py\", line 979, in wrapper", " return fn(*args, **kwargs)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 107, in _create_in_db", " return _apply_updates(context, db_mapping, updates)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/nova/objects/host_mapping.py\", line 33, in _apply_updates", " db_mapping.save(context.session)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py\", line 50, in save", " session.flush()", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2171, in flush", " self._flush(objects)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2291, in _flush", " transaction.rollback(_capture_exception=True)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py\", line 66, in __exit__", " compat.reraise(exc_type, exc_value, exc_tb)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/session.py\", line 2255, in _flush", " flush_context.execute()", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 389, in execute", " rec.execute(self)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py\", line 548, in execute", " uow", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 181, in save_obj", " mapper, table, insert)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py\", line 835, in _emit_insert_statements", " execute(statement, params)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 945, in execute", " return meth(self, multiparams, params)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/sql/elements.py\", line 263, in _execute_on_connection", " return connection._execute_clauseelement(self, multiparams, params)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1053, in _execute_clauseelement", " compiled_sql, distilled_params", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1189, in _execute_context", " context)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1398, in _handle_dbapi_exception", " util.raise_from_cause(newraise, exc_info)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/util/compat.py\", line 203, in raise_from_cause", " reraise(type(exception), exception, tb=exc_tb, cause=cause)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/base.py\", line 1182, in _execute_context", " context)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/sqlalchemy/engine/default.py\", line 470, in do_execute", " cursor.execute(statement, parameters)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/cursors.py\", line 166, in execute", " result = self._query(query)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/cursors.py\", line 322, in _query", " conn.query(q)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 856, in query", " self._affected_rows = self._read_query_result(unbuffered=unbuffered)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 1057, in _read_query_result", " result.read()", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 1340, in read", " first_packet = self.connection._read_packet()", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 1014, in _read_packet", " packet.check_error()", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/connections.py\", line 393, in check_error", " err.raise_mysql_exception(self._data)", " File \"/openstack/venvs/nova-r16.1.0/lib/python2.7/site-packages/pymysql/err.py\", line 107, in raise_mysql_exception", " raise errorclass(errno, errval)", "DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u\"Duplicate entry 'compute2' for key 'uniq_host_mappings0host'\") [SQL: u'INSERT INTO host_mappings (created_at, updated_at, cell_id, host) VALUES (%(created_at)s, %(updated_at)s, %(cell_id)s, %(host)s)'] [parameters: {'created_at': datetime.datetime(2018, 2, 22, 18, 9, 51, 170519), 'cell_id': 7, 'host': u'compute2', 'updated_at': None}]"]}
ok: [compute2 -> 10.0.237.173]

TASK [os_nova : Get UUID of new Nova Cell] *************************************
Thursday 22 February 2018 12:09:51 -0600 (0:00:06.734) 2:11:28.204 *****
ok: [compute2 -> 10.0.237.173]

TASK [os_nova : Map instances to new Cell1] ************************************
Thursday 22 February 2018 12:09:57 -0600 (0:00:05.971) 2:11:34.176 *****
ok: [compute2 -> 10.0.237.173]

TASK [openstack_openrc : Create openrc file] ***********************************
Thursday 22 February 2018 12:10:03 -0600 (0:00:05.916) 2:11:40.092 *****
ok: [compute2]
-----------------------------------------------------

Tags:

git-harry (git-harry) on 2018-03-01

Changed in openstack-ansible:
assignee:	nobody → git-harry (git-harry)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-01: Fix proposed to openstack-ansible-os_nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/548921

Changed in openstack-ansible:
status:	New → In Progress

Revision history for this message

Jesse Pretorius (jesse-pretorius) wrote on 2018-03-05:

I think this issue is already being worked on in https://review.openstack.org/547072 ?

Revision history for this message

git-harry (git-harry) wrote on 2018-03-05:

@Jesse, it appears that review addresses this issue as a side effect of the changes being made but is not intentionally trying to address this bug. I'm happy to abandon 548921 in favour of that change as long as its larger scope isn't likely to significantly slow its progress. This bug is a common source of failure when deploying multiple compute nodes.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-05: Change abandoned on openstack-ansible-os_nova (master)

Change abandoned by git-harry (<email address hidden>) on branch: master
Review: https://review.openstack.org/548921
Reason: Currently blocked by https://bugs.launchpad.net/openstack-ansible/+bug/1753443

Abandoning in favour of https://review.openstack.org/#/c/547072/ which will address this bug as a side effect of the changes being made.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-14: Related fix merged to openstack-ansible-os_nova (master)

Reviewed: https://review.openstack.org/547072
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=6321cf90304023c20dc6abb87e2fa20107b7051d
Submitter: Zuul
Branch: master

commit 6321cf90304023c20dc6abb87e2fa20107b7051d
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

    After a new compute host is built, it will need to mapped to a cell
    through the 'discover_hosts' command of nova-manage. This command will
    still need to be run against a host that has direct database access, but
    the command is able to handle mapping of all compute hosts that do not
    currently belong to the cell so it only needs to be run once per play.

    There can be a short delay in the time between a compute service
    starting and becoming available to be discovered and mapped but, for
    most deployments, compute host discovery will be handled automatically
    as a periodic task.

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-14: Related fix proposed to openstack-ansible-os_nova (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/552898

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-14: Related fix proposed to openstack-ansible-os_nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/552899

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-14: Related fix proposed to openstack-ansible-os_nova (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/552900

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-14: Related fix merged to openstack-ansible-os_nova (stable/pike)

Reviewed: https://review.openstack.org/552899
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=c581c2af7b2bfdf60fb4467d7d5b17bed32fc85f
Submitter: Zuul
Branch: stable/pike

commit c581c2af7b2bfdf60fb4467d7d5b17bed32fc85f
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

Conflicts:
>------tasks/main.yml

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540
    (cherry picked from commit 6321cf90304023c20dc6abb87e2fa20107b7051d)

tags:

added: in-stable-pike

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-14: Related fix merged to openstack-ansible-os_nova (stable/ocata)

#10

Reviewed: https://review.openstack.org/552900
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=707398789403293aaedc1d78ddc4a164611366c7
Submitter: Zuul
Branch: stable/ocata

commit 707398789403293aaedc1d78ddc4a164611366c7
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

Conflicts:
>------tasks/main.yml

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540
    (cherry picked from commit 6321cf90304023c20dc6abb87e2fa20107b7051d)

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-15: Related fix merged to openstack-ansible-os_nova (stable/queens)

#11

Reviewed: https://review.openstack.org/552898
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=d451edc2b83128e182e8885adf01f925ea2f98b6
Submitter: Zuul
Branch: stable/queens

commit d451edc2b83128e182e8885adf01f925ea2f98b6
Author: Jimmy McCrory <email address hidden>
Date: Thu Feb 22 08:55:06 2018 -0800

Rearrange cell mapping tasks

    Existing instances can be mapped to a newly created cell immediately
    after the cell is created and by the same conductor host that created
    the cell.

Conflicts:
>------tasks/main.yml

    Change-Id: I4f7328e1aff79eb94f278999b97fbad8f44c6a3e
    Closes-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540
    (cherry picked from commit 6321cf90304023c20dc6abb87e2fa20107b7051d)

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-28: Related fix merged to openstack-ansible (stable/pike)

#12

Reviewed: https://review.openstack.org/554675
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=5ca262641f5a8182d0f87c413f919c47b43b27ac
Submitter: Zuul
Branch: stable/pike

commit 5ca262641f5a8182d0f87c413f919c47b43b27ac
Author: d34dh0r53 <email address hidden>
Date: Tue Mar 20 14:44:51 2018 -0500

Update os_nova SHA

This updates the os_nova SHA to pull in the Nova API downtime fixes for
bug 1749037.

    Related-Bug: 1749037
    Related-Bug: 1729661
    Related-Bug: 1752540

Depends-On: https://review.openstack.org/552899

Change-Id: I3e21a4b741c88a72294175786bf66df8ab19e390

Revision history for this message

Kevin Lefevre (archifleks) wrote on 2018-04-10:

#13

Hi,

I add to revert this commit it the latest 16.0.11 release,

each time os_nova is failing on cell_v2 playbook :

TASK [os_nova : Perform a cell_v2 discover] *****************************************************************************************│·
********************************************* │·
2018-04-05 09:43:46,909 p=179270 u=root | Thursday 05 April 2018 09:43:46 +0000 (0:00:01.202) 1:10:21.644 ******** │·
2018-04-05 09:44:13,853 p=179270 u=root | fatal: [os-rone-kvm-01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hos│·
tname os-rone-cp-01_nova_conductor_container-dbee0504: Temporary failure in name resolution\r\n", "unreachable": true} │·
2018-04-05 09:44:13,857 p=179270 u=root | NO MORE HOSTS LEFT ******************************************************************************************************************│·

I don't know how the delegation is working, from the deployer machine, /etc/hosts is not populated but I assume the name to resolve from inventory.

On a controller node, I can resolve and SSH into os-rone-cp-01_nova_conductor_container-dbee0504.

From a compute node I can resolve but not SSH into os-rone-cp-01_nova_conductor_container-dbee0504

Revision history for this message

Ondrej Vasko (lirt) wrote on 2018-05-03:

#14

So since this task is delegated, compute node (os-rone-kvm-01 in Kevin's case) should have SSH key to access containers. I tried to add the ssh key to compute node to user root manually and after that I tried to connect to container correctly. But anyway after running playbooks I receive this error:

Failed to connect to the host via ssh: Could not resolve hostname ...: No address associated with hostname\r\n", "unreachable": true

I have LXC host added in /etc/hosts

So now I don't understand what is issue here...

Revision history for this message

Ondrej Vasko (lirt) wrote on 2018-05-04:

#15

So deployment host is delegating task to nova_conductor container, but it uses its container name as hostname for SSH eg. controller1_nova_conductor_container-c4707efe and deployment host doesn't have hosts file prepared as target nodes.

So the fix may be using IP address of `random_conductor` in delegation instead of container name.

Revision history for this message

Ondrej Vasko (lirt) wrote on 2018-05-04:

#16

As hotfix can be copying hosts file from any target host to deployment host.

Dmitriy Rabotyagov (noonedeadpunk) on 2024-02-14

Changed in openstack-ansible:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.