Undercloud upgrade from newton to ocata fails on mistralclient

[Jan Provaznik]

After deploying tripleo with: "./tripleo-quickstart/quickstart.sh -R newton -T all -t all -N tripleo-quickstart/config/nodes/1ctlr_1comp_1ceph.ym", when I upgrade undercloud to ocata (by following steps https://docs.openstack.org/developer/tripleo-docs/post_deployment/upgrade.html), then "openstack undercloud upgrade" fails with:

2017-06-13 19:59:32,451 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 99-refresh-completed completed
2017-06-13 19:59:32,452 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 ----------------------- PROFILING -----------------------
2017-06-13 19:59:32,453 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017
2017-06-13 19:59:32,455 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 Target: post-configure.d
2017-06-13 19:59:32,456 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017
2017-06-13 19:59:32,457 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 Script Seconds
2017-06-13 19:59:32,458 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 --------------------------------------- ----------
2017-06-13 19:59:32,459 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017
2017-06-13 19:59:32,468 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 10-iptables 0.004
2017-06-13 19:59:32,473 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 80-seedstack-masquerade 0.022
2017-06-13 19:59:32,478 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 98-undercloud-setup 156.676
2017-06-13 19:59:32,483 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 99-refresh-completed 1.228
2017-06-13 19:59:32,485 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017
2017-06-13 19:59:32,486 INFO: dib-run-parts Tue Jun 13 19:59:32 UTC 2017 --------------------- END PROFILING ---------------------
2017-06-13 19:59:32,487 INFO: [2017-06-13 19:59:32,486] (os-refresh-config) [INFO] Completed phase post-configure
2017-06-13 19:59:32,495 INFO: os-refresh-config completed successfully
2017-06-13 19:59:34,351 INFO: Not creating flavor "baremetal" because it already exists.
2017-06-13 19:59:34,377 INFO: Not creating flavor "control" because it already exists.
2017-06-13 19:59:34,404 INFO: Not creating flavor "compute" because it already exists.
2017-06-13 19:59:34,432 INFO: Not creating flavor "ceph-storage" because it already exists.
2017-06-13 19:59:34,462 INFO: Not creating flavor "block-storage" because it already exists.
2017-06-13 19:59:34,488 INFO: Not creating flavor "swift-storage" because it already exists.
2017-06-13 19:59:34,488 DEBUG: An exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1587, in install
    _post_config(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1499, in _post_config
    session=sess)
  File "/usr/lib/python2.7/site-packages/mistralclient/api/client.py", line 19, in client
    return client_v2.Client(auth_type=auth_type, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistralclient/api/v2/client.py", line 42, in __init__
    req = copy.deepcopy(kwargs)
  File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.7/copy.py", line 329, in _reconstruct
    y = callable(*args)
  File "/usr/lib64/python2.7/copy_reg.py", line 93, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
2017-06-13 19:59:34,490 ERROR:
#############################################################################
Undercloud upgrade failed.

Reason: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

See the previous output for details about what went wrong. The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1587, in install
    _post_config(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1499, in _post_config
    session=sess)
  File "/usr/lib/python2.7/site-packages/mistralclient/api/client.py", line 19, in client
    return client_v2.Client(auth_type=auth_type, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistralclient/api/v2/client.py", line 42, in __init__
    req = copy.deepcopy(kwargs)
  File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib64/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib64/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib64/python2.7/copy.py", line 329, in _reconstruct
    y = callable(*args)
  File "/usr/lib64/python2.7/copy_reg.py", line 93, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
Command 'instack-upgrade-undercloud' returned non-zero exit status 1

[Jan Provaznik (jan-provaznik)]

there were actually following changes in the oooq config file (-N parameter):
# cat config/nodes/3ctlr_1comp_ceph.yml
# This config file is used to deploy a dedicated node in the overcloud for ceph

# Define a controller node, a compute node, and a ceph node.
overcloud_nodes:
  - name: control_0
    flavor: control
    virtualbmc_port: 6230

  - name: control_1
    flavor: control
    virtualbmc_port: 6231

  - name: control_2
    flavor: control
    virtualbmc_port: 6232

  - name: compute_0
    flavor: compute
    virtualbmc_port: 6233

  - name: ceph_0
    flavor: ceph
    virtualbmc_port: 6234

node_count: 5

# Tell tripleo which nodes to deploy.
topology: >-
  --ceph-storage-scale 1
  --control-scale 3
  -e {{overcloud_templates_path}}/environments/storage-environment.yaml
  -e {{overcloud_templates_path}}/environments/enable-swap.yaml
  -e {{overcloud_templates_path}}/environments/puppet-pacemaker.yaml
  -e {{overcloud_templates_path}}/environments/manila-cephfsnative-config.yaml

network_isolation: true
network_isolation_type: 'single-nic-vlans'

[Emilien Macchi (emilienm)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (FUTURE, PIKE, QUEENS, ROCKY, STEIN).
  Valid example: CONFIRMED FOR: FUTURE